A robots.txt is text file that resides on the root directory of the web server.  The purpose of this file is it to indicate to visiting robots and crawlers which parts of the underlying web site should be visited for the purpose of inclusion in a search engine.

More information can be obtained on the following site: http://www.robotstxt.org/

In most cases you will want the search engines to access the site and crawl all parts of the site.  An example Robots.txt file is below which indicates to ALL crawlers and robots (indicated by ‘User-agent: *’) to index the whole site apart from content under the folder ‘mySecretFolder’

User-agent: *
Disallow: /mySecretFolder/