1. How to allow all search engine spiders to index all files

Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site:

User-agent: *
Disallow:

2. How to disallow all spiders to index any file

If you don't want search engines to index any file of your Web site, use the following:

User-agent: *
Disallow: /