Last updated on June 26th, 2022 at 11:41 am
It is good to have Googlebot crawl your site as often as possible, but not always. What if your site is already well-indexed?
Sometimes Googlebot thinks that it is worth crawling the site even more frequently. And this is where the issues start, especially if you are on a shared hosting plan. You simply need to limit your crawl rate if Google is slowing down your server.
I recently noticed some really high crawl rate resulting in a very high load on my shared hosting server. For some reason, Googlebot decided to start crawling my site more and more frequently reaching more than 85,000 pages crawled per day.
So, in my case, I needed to find a way to decrease the crawl rate in Google Webmaster Tools. Luckily, there’s still an option to do this, in the old Search Console interface.
Decrease the crawl rate in of Googlebot from Google Webmaster Tools
- Go to https://www.google.com/webmasters/tools/home
- Select your property.
- Click on Site Settings
- Click on Limit Google’s maximum crawl rate to limit (decrease in my case!) Googlebot’s crawl rate
