Hundreds of web robots crawl the Internet and build search engine databases, but they generally follow the instructions in a site’s robots.txt.
Web robots and spiders have been crawling the Internet since it’s inception. With the development of the web and the growth of search engines through the late 1990’s, the number of automated robots wandering the web grew quickly, so that it was not unusual to find several dozen visiting in the course of a day.
Recognizing the need, to help control these automatic visitors, Martijn Koster designed the file “robots.txt” for use by a web site to tell search engines which directories and files to exclude from their robot scans, either for functional or privacy reasons. Robots are not obliged to follow the instructions in a robots.txt file, but most do. For example, a web site might exclude one or more directories from one or more robots simply because it cannot afford too much bandwidth expenditure, or because it contains very dynamic information, like topical news data, and the pages would either be missing or changed if retrieved later through a search engine.
You can read the robots.txt file for a site by opening the site name appended with “robots.txt”, such in the examples below:
Resources. The following sites provide more information about web robots: