New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Is BingBot so Horrid for everyone?
So I have a wonderfully large stats website and bing likes to "attack" it while scraping its contents for its search engine. I don't want to block it though. I would rather have it be like google, where is just searches at a nice content rate, and I was wondering if anyone here has a good idea on how to do that?
user-agent:bingbot
crawl-delay:1
For robots.txt, but it doesn't change the rate of the lookup, just makes it happen at a very very very slow rate. Any ideas?
Comments
I don't understand - you make a condtradictory statement; it doesn't change the rate, but yet it does?
Distributed bots can take a few days to pick up on robots.txt changes, hopefully will calm down on its own... In the mean time you could ban the UA from less important parts of your site with .htaccess?
You should feel privileged. I would love to see bingbot a bit more often. Currently it represents 2.1% of the googlebot traffic.
You can create webmaster account at bing and set the preferences
I have created an account with bing webmaster, but it still comes in waves, and these waves do 100 queries in less then 1 minute. It is rather intensive of a processes. I have set it up with a very slow crawl now, yet it still happens.
Sorry about my wording before, what I meant was it still comes in massive waves, or doesn't happen at all. So yes it is changing, but to an undesirably result.