Spiders, Crawlers, and
Other Search Engine Robots...
The technology of search
engine is automating the indexing through the use of
search engine "spider" or "crawler".
Search engine "spider" or "crawler"
go out to take a "picture" of your site, and then puts that "picture" in its index. When a user goes to the
search engine looking for information and types in whatever he or she is looking for, the search engine refers to its index to find the best matches. Your goal is to make sure that the
search engine considers YOUR site to be the best match for the keyword phrases that are important to your site.
But what are search engine spiders, crawlers, or robots?
They are merely computer programs which capture certain information from your Web page and carry it back to the engine from which it came. There, the information gets indexed and afterward, it is available to be found within the search results. Whether your page is easily found or not will depend upon how well your page is optimized.
Search engine robots, also called spiders, crawlers, ants, wanderers, harvesters and other names..... often travel the Web on their own schedule and nearly behave as though they have a life of their own.
The search engine robot's job is to:
