New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Google bot requests are not being cached
- In recent weeks, we have been experiencing a significant influx of requests from GoogleBot, resulting in frequent downtime for our system
I created the rule this way but it doesn't work
- If it can't be cached, is there any way for googlebot to send requests to my website without filling up the cpu?
- If it can't be cached, is there any way for googlebot to send requests to my website without filling up the cpu?
this is the amount of requests the bot sends, i have to create a js rule to challenge so the website can continue to work
this is my mysql setting, is there any way to set it up so it can handle the requests from bots
i am using vps contabo 4 vCPU Cores 6 GB RAM or 400 GB SSD
someone please help me
Comments
curl -I -H "User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/path
Check CF header cache status
When your page(s) is not yet cached by CF, it still load your Origin (use proper index and redis/memcached caching to reduce load of your DB).
Move out from contabo?
Serious answer:
investigate why your mysql use 400% cpu. Maybe missing index? Investigate slow log? IO bottleneck?
Increase innodb buffer pool size about additional 1 GB since your memory still have about 2.5 GB free memory?
I found out the cause was because googlebot requested too many requests, as in the article, I turned on the challenge rule with the bot and my website ran normally again, and had 200 visitors.
the first problem is contabo
https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate
You can return HTTP status 429 to a percentage of the requests, so that Googlebot will reduce its crawl rate.
Besides this, is there any other way?
Cloudflare has a rate limiter in security settings area. You can set a limit for requests to prevent overload.
I had to do that last year when googlebot got stuck in an infinite loop due to a bad plugin setup (someone used a shortcode posts display plugin inside a tabs plugin, infinite loop chaos unlocked). Googlebot was hitting my server with 20 million requests per day!!! DOS'ed by Google lol
If so, Googlebot receives a block signal, will it affect SEO?