Figure 5


Granting access privileges to specific robots.

# robots.txt for multiple robots
User-agent: NorthStar   # Northstar Jumpstation
User-agent: RBSE-Spider # RBSE project
Disallow: /private/     # don't index the private directory

User-agent: *          # everyone else
Disallow: /            # don't allow any access