What are Spiders and what are they doing?

[Lenny]

[quote]Reload this Page 4 members and 6 guests
Most users ever online was 108, 07-29-2004 at 02:32 AM.
Last Activity User Name Location Instant Messaging
11:01 PM Bryan Tuvell 33ZX Viewing Thread
Miami, 500 pictures, browse on his dime!
11:02 PM Lenny Viewing User Control Panel
10:49 PM onesubdrvr Viewing Index
The Donzi Registry
11:01 PM Walt. H. Viewing Thread
Another what is it!
11:00 PM Altavista Spider Viewing User Profile
10:56 PM Guest Viewing Index
The Donzi Registry
10:59 PM Guest Viewing Thread
DONZI Customer Service
10:49 PM Guest Viewing Index
The Donzi Registry
10:58 PM Google Spider Viewing User Profile
11:02 PM Guest Viewing Thread
Donzi Decals


...curious.. Wes? Scot Van A?

That's 2:AM your time in most parts in the East. What's up?

[Formula Jr]

Spider, Spyder That part of a search engine which surfs the web, storing the URLs and indexing the keywords and text of each page it finds. Please refer to the Search Engine Watch SpiderSpotting Chart (http://searchenginewatch.com/webmasters/spiderchart.html) for details of individual spiders. See also Robot (http://www.cadenza.org/search_engine_terms/srchor.htm#set_robot). Spidering The process of surfing the web, storing URLs and indexing keywords, links and text. Typically, even the largest search engines cannot spider all of the pages on the net. This is due to the huge amount of data available, the speed at which the new data appears, the use of politeness windows (http://www.cadenza.org/search_engine_terms/srchor.htm#set_politeness_window) and practical limits on the number of pages that can be visited in a given time . The search engines have to make compromises in order to visit as many sites as possible, and they do this in different ways. For example, some only index the home pages of each site, some only visit sites they're explicitly told about, and some make judgements about the importance of sites (from number and quality of inbound links) before "digging deeper" into the subpages of a site.

[txtaz]

Lenny, JR is correct. I'm curious where or how you got that list. Spiders started out 15 years ago as small programs that sat at a node (a computer connected to a network) and collected info and then routed it to a specific address. They were used from anything like search engines, web tracking to viruses. I wrote one once to track a cheating girlfriends web activity when she would go to the library to use the web. Tooo bad for her, she missed the Donzi Daze.
Spiders are only used for ligitamate uses these days. They collect the data, sends back to host and stored in a database. They call it metadata (data about data). It is to easy to track the destination of the program for virus/spamware writers to use them. They know they have liability and someone eventually would come after them. Hymmmm, have I mentioned my email program that can send as many "quit spamming me" messages as you want. It works great on broadband. :rolleyes: I'd be happy to share it but there's not much use for it anymore.
Hope this helps.
Wes
PS, I noticed the time is out of sequence. That seams unusual to me. Things happen sequentially or indexed based on time.

[mattyboy]

they're big ugly orange furry things that live in your PC or under a gas tank in Texas :eek::tongue:

[ToonaFish]

they're big ugly orange furry things that live in your PC or under a gas tank in Texas :eek::tongue:

Well, sort of...