Unreachable Robots.txt – Google Couldn’t Crawl Your Site Because We Were Unable To Access The Robots.txt File

Recently Google Webmaster Tools tells me that it is unable to crawl my site because my robots.txt file is unreachable. Having a robots.txt file registered in Google Webmaster Tools that cannot be accessed is worse than not having a robots.txt file at all. A robots.txt file tells search engines whether they are allowed to crawl […]
Continue reading…