Robots.txt and Robots Meta Tag

Question: What is a "robots file", and do I need to use one?

Answer: Robots.txt is an optional text file that lives in the root folder of your web site. Using the file, you can specify areas of your web site that you do not want the search engine spiders to crawl or index. Examples could include client pages, admin areas, or other sensitive information. A poorly-formatted robots.txt file could prevent all or some of your optimized pages from being indexed. You can read a lot more about robots.txt here:
http://en.wikipedia.org/wiki/Robots.txt

Similarly, you can employ a Robots Meta Tag in the header of individual pages to keep that page from being indexed by spiders. The following Meta Tag tells spiders to not index this page or follow any of the links on the page:
< name="ROBOTS" content="noindex,nofollow">

Both of these methods are optional, and you do not need to use them unless you are trying to keep specific pages from being indexed. Misuse of these methods can have unwanted consequences. So, if you find a page isn’t showing up in the search engines, check the Robots.txt file and look for an errant robots Meta tag.

Need more help? Let me know