Figure 3


Basic statements for robots.txt.

# robots.txt for http://www.genome.wi.mit.edu 
User-agent: *   # match any robot name
Disallow: /cgi-bin/   # donšt allow robots into cgi-bin