# /robots.txt file for http://www.rivetek.com/ User-agent: * Disallow: /support # robots.txt how to information below: # # References: http://www.robotstxt.org/wc/faq.html # References: http://www.robotstxt.org/wc/active.html # # Example: # User-agent: webcrawler # Disallow: # # User-agent: lycra # Disallow: / # # User-agent: * # Disallow: /tmp # Disallow: /logs # # The first paragraph specifies that the robot called # 'webcrawler' has nothing disallowed: it may go anywhere. # # The second paragraph indicates that the robot called 'lycra' # has all relative URLs starting with '/' disallowed. Because # all relative URL's on a server start with '/', this means # the entire site is closed off. # # The third paragraph indicates that all other robots should # not visit URLs starting with /tmp or /log. Note the '*' is a # special token, meaning "any other User-agent"; you cannot # use wildcard patterns or regular expressions in either # User-agent or Disallow lines. # # Two common errors: # # Wildcards are _not_ supported: instead of 'Disallow: /tmp/*' # just say 'Disallow: /tmp/'. # # You shouldn't put more than one path on a Disallow line # (this may change in a future version of the spec)