Why you should use the Robots.txt carefully...
The robots.txt file has been around since 1994! Its purpose is to tell search engines such as Google what website areas can be crawled by web robots (also known as Web Wanderers, Crawlers, or Spiders) to harvest information (or index) to store for searching.
More information about the robots.txt fi...Read more...