Spiders may be used to search for keywords or phrases on a website or collection of websites. This could be done by examining the text on the page in question, but this can be a tedious task, especially if the target site is large. A spider can search multiple websites at once by storing information it finds and returning it later.
Spiders are commonly used to monitor websites, find links, check how users interact with a website and so on.
Spiders usually have 3 main functions:
The “follow the links” function searches for any links within the current page’s page elements and provides the location of each one to the user.
The “keyword search” function uses the information it finds within a page to find matching keywords or key phrases and reports back to the user.
The “click analysis” function monitors user interaction with a website to build up a profile of that website’s visitors.
Spiders can be automated using a server-side or a client-side solution.
Some types of spiders that search for links on the web:
List Viewers. This type of spider finds the keywords or phrases that are visible on web pages and automatically generates a list of links to that page.
Watchers. Watchers usually spider a site looking for information or new content. This type of spider is known as a web crawler, and it is a highly intelligent spider capable of finding and retrieving web content that is hidden from the original web page.
Spiders like Google, Yandex, Bing, etc. These spiders are optimized to find, search and store a large amount of links. Many of these spiders can also crawl private, internal and locked pages.
List viewers
List Viewers or List-o-Spiders will simply generate a list of the links within a page for the user to view. The information may be presented in a simple HTML format or in a special format known as RSS. Sometimes list viewers will also be able to generate a list of images, flash movies and sounds, but it is not always true that this functionality is present.
List viewers can be found on a wide variety of programs or software including the many browsers such as Netscape and Internet Explorer. They are also present on a wide variety of websites. Some of these websites include:
www.findlinks.com
www.linklist.com
www.spider
Related links: