Ruby web crawler


Category:
Scripts
Ruby
Requirements:
Seller:
Price:
-
System:
Rating:
4.7
License:

Description - Ruby web crawler



This Ruby script reads in a list of links from links.dat, it then picks out the ones it can easily spider and gets a list of URLs from each PaGE listed in links.dat. Every new URL it finds will be added to newlinks.dat for later spidering by another bot running along side this one.



More in Ruby-Ruby web crawler

Urls From Each