Webmaster Tutorials

A Guide to Robots.txt

This guide looks at setting up your own robots.txt file so that you can control spiders from search engines as well as more malicious sources.

Read tutorial