Figure 4


Using multiple Disallow statements.

User-agent: *        # match any robot name
Disallow: /cgi-bin/   # don't allow robots into cgi-bin
Disallow: /tmp/       # no need to index temporary files
Disallow: /private/   # don't index the private directory