Replies: 1
I recently started using the rate limiting feature to stop bots from overly crawling my sites. I was having a problem specifically with Amazonbot. My sites were getting thousands of requests from Amazonbot which caused performance problems for my sites. I don’t sell anything on Amazon. Is there any point to having Amazonbot crawl my sites? I already blocked it in robots.txt. It appears that Amazonbot follows this rule.
This is currently my settings for rate limiting. I’m just a beginner when it comes to setting this up as I never used it before. Are these settings too strict? Or is it optimal? I followed the Wordfence guide on setting up rate limiting. It seems 240 per minute is the recommended value but what exactly does it mean? Compared to say 120?
How should we treat Google’s crawlers – Verified Google crawlers will not be rate-limited
If anyone’s requests exceed – 240 per minute, then throttle it
If a crawler’s page views exceed – 120 per minute, the throttle it
If a crawler’s pages not found (404s) exceed – 60 per minute, then throttle it
If a human’s page views exceed – 240 per minute, then throttle it
If a human’s pages not found (404s) exceed – 60 per minute, then throttle it
How long is an IP address blocked when it breaks a rule – 30 minutes