The Web Robots Pages (robot exclusion standard)
info.webcrawler.com/mak/projects/robots/robots.html

CPAN
www.perl.com/CPAN/

gnuplot
ftp://prep.ai.mit.edu/pub/gnu/

MOMSpider
www.ics.uci.edu/WebSoft/MOMspider/

Teleport Pro
www.tenmax.com

w3mir
www.math.uio.no/~janl/w3mir/