Weird crawlers?

JamesFitz

Hey every one,
I have my site all set up and stuff, and i see crawlers all over my site...
/tags/%25e6%2597%25a5%25e5%25a0%25b1 HTTP/1.1" 404 116308 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.xxx - - [27/Jun/2015:06:36:44 -0400] "GET /wordpress/plugins/webcijfers-seo-scan HTTP/1.1" 404 116308 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.xxx - - [27/Jun/2015:06:37:01 -0400] "GET /wordpress/plugins/captain-slider HTTP/1.1" 404 116308 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
82.103.128.xx - - [27/Jun/2015:06:37:10 -0400] "GET / HTTP/1.1" 200 816 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com)"
66.249.65.xxx - - [27/Jun/2015:06:37:18 -0400] "GET /wordpress/plugins/social-contact-display HTTP/1.1" 404 116308 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.xxx - - [27/Jun/2015:06:37:35 -0400] "GET /wordpress/plugins/weather-slider HTTP/1.1" 404 116308 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.xx - - [27/Jun/2015:06:37:36 -0400] "GET /tags/curiyo HTTP/1.1" 404 116308 "-"

I don't have wordpress on my site, just static HTML that i made.
Thanks...

Dillybob

Googlebot is a not weird crawler. They are the crawlers and they want to index the shit out of your site. Nothing to worry about.

JamesFitz

@Dillybob said:
Googlebot is a not weird crawler. They are the crawlers and they want to index the shit out of your site. Nothing to worry about.

Alright, thanks.

hostnoob

The previous owner of the domain probably had Wordpress set up

BeardyUnixGuy

Browser agents can easily be faked so, before you write it off as GoogleBot, maybe just take a second to verify it yourself using a reverse DNS lookup.

I checked out an IP address on that subnet and it appeared to be GoogleBot.

Traffic

@BeardyUnixGuy said:
Browser agents can easily be faked so, before you write it off as GoogleBot, maybe just take a second to verify it yourself using a reverse DNS lookup.

I checked out an IP address on that subnet and it appeared to be GoogleBot.

And remember to do the opposite check as well or you won't prove anything at all (anyone can set their rDNS to what they like, but not make them match the IP).

https://es.wikipedia.org/wiki/Forward_confirmed_reverse_DNS

lyytwenty

It's google bot, Dont worry about that!

Clouvider

I would extend it to not worry about any crawler as long as they are not impacting the performance or security of your website.