You can view my robots.txt file for an example...
http://d3xt3r.net/robots.txtI suggest not going to files that are hidden. My security system may ban you.
But, you can learn more about robots.txt at
http://robotstxt.orgHere is a example file that would ban all bots.. (That listened, anyhow.)
User-Agent: *
Disallow: /
Or to only allow Google...
User-Agent: *
Disallow: /
User-Agent: googlebot (I think this is the useragent, might wanna check on googles site)
Allow: /
Avoid using * within actual paths. Cause some don't abide by it. I found that out the hard way...