My stupidest seo mistake (till date)

Many regular readers of Binaryday will be aware that I have been working on a used bikes website for quite sometime. Around 1 month back I moved binaryday and the used bikes site to a new server on bluehost. While doing some routine seo checkup on the site, I realized that google has not indexed the site since September 15 i.e almost 1 month time!!! A quick check on other sites showed that all sites that had been moved to bluehost, were showing the same issue. I was really scared at this point as that meant google might have flagged off something about my site.

I logged into google webmaster tools and went straight to the crawl stats page. Here is a screenshot of the crawl stat page that confirmed my fear that google has stopped crawling my pages. From an average of 200 pages per day, Google had stopped crawling my pages all together!!!

My next stop was the sitemaps section of google webmaster tool. The sitemap for used mobiles site was showing some stupid error i.e robots.txt unreachable. I never put up a robots.txt file on my website out of my love for GoogleBot. It has always worked fine. What is this new issue??

A quick search on google led me to this file. This is a google webmaster help file.I found the problem at the bottom of the page. It is OK to not have a robots.txt on your site if an attempt to access robots.txt results in a 404 error. However if you do not have a robots.txt file and the server is giving some other error status then google does not crawl your site at all !!!

I have just added a new robots.txt to the site and google has started to access the files. Hope everything will get back to normal soon. This has been one of the stupidest mistakes that I have done in the last 5-6 years. I am confident of going past it in future though.

Tagged as: