Hi Guys
For those making use of robot.txt files to prevent search engines from crawling certain content, here is important news confirmed by John Mueller of Google:
http://www.eworldpost.com/search-engine-crawlers-cache-your-robots-txt/
Search engines cache robot.txt files. You could end up getting that content indexed at first and later removed from listings.
You should first update your robot.txt file at least 24 - 48 hours before adding the content.
Hope it helps.
For those making use of robot.txt files to prevent search engines from crawling certain content, here is important news confirmed by John Mueller of Google:
http://www.eworldpost.com/search-engine-crawlers-cache-your-robots-txt/
Search engines cache robot.txt files. You could end up getting that content indexed at first and later removed from listings.
You should first update your robot.txt file at least 24 - 48 hours before adding the content.
Hope it helps.
