Search engines use spiders or bots for indexing Web sites. A spider is an automated program that is managed by the search engine system. Spider visits to a Web site, read the content on the site, sites and meta tags also follow links that the site logs. The spider then returns all of this information to a central depository, where data are indexed. He visited every link you on your website and indexes these sites.

Some spiders no indication of a number of pages on your site. The spider will return periodically on the sites to verify any information that has changed. The frequency with which this happens is determined by the moderators of the search engine. Put simply: a spider is almost like a book that it contains a table of contents, the actual content and links and references to all the sites it finds in the course of his research, and it can index up to a million pages per day.

Ex: Excite,Google, Lycos, AltaVista.