Robots.txt File


A Robots.txt file is a means of communicating with search engines that most webmasters will never use. The intent is to tell search engines which pages to index and which ones to not index, used primarily on the ones to not index.

This is done by noting either a folder or a specific page for the search engine crawlers (robots) to never crawl. Search engines typically abide by these rules, but they don’t have to.

Example: Apple’s robots.txt file:

Screen Shot 2015-01-31 at 10.14.50 AM

Although many webmasters will never touch their robots.txt file, I bring it up because it can have extreme effects on indexation in search.

For example, if you include these two lines within your robots.txt file, search engines will not index any part of your site:

User-agent: *
Disallow: /
To learn more about Robots.txt File (how to properly and improperly implement, as well as how to find it on your site), signup for SEO Pro here.