The Robots Page
info.webcrawler.com/mak/projects/robots/robots.html

Perl LWP Library
www.perl.com/CPAN/modules/by-module/LWP/