New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How to block un-necessary traffic wit HubSpot Webcrawler in it?
Hi ,
I have seen a spike in the web traffic on my wordpress site and by checking the access logs , i see the following type:
54.226.254.1 - - [10/Aug/2014:15:04:15 +0200] "GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.0" 404 1182 "xxxxx" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler"
54.226.254.1 - - [10/Aug/2014:15:04:15 +0200] "GET /wp-content/plugins/crayon-syntax-highlighter/js/min/crayon.min.js?ver=2.6.5 HTTP/1.0" 404 1182 "xxxx" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler"
All with the: HubSpot Webcrawler , Is there anyway , i can block traffic coming with HubSpot Webcrawler?
i am using vestaCP with centos
Comments
Okay if someone else is looking for the solution , I tried this with .htaccess and it seems to be working cause i dont see anymore bot like that. Will keep mosttly unwanted bots away