Changes in robots.txt file
Webmasters want their site to get indexed by the search engines frequently. To achieve this sitemaps.org an independent body has requested site administrators to add an extra line in the robots.txt file. By default robots.txt is the first file that is accessed by any search engine. If the sitemap URL is provided in the robots.txt file itself then the chances of the website getting indexed becomes easier. For example the robots.txt for meevy will be
Sitemap: http://www.meevy.com/sitemap.xml
User-agent: *
Disallow:
The first line provides the sitemap link which will in turn help identify the various pages of a site. Thus new contents could easily get crawled by search engines. The second line (user agent = any) means that the robots.txt file can be accessed by any search engine. The third line (disallow = none) indicates that no search engine would be blocked.
via Yahoo Search Blog, Google Webmaster Central Blog , Quick Online Tips