New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Looking for a host/package/offer that can crawl 3 million pages a day
jeffreylroberts
Member
I was able to accomplish this with an AWS r4.large instance, I am looking for a similar offer/package. Ideally something with a 10Gbps uplink that can handle 100 concurrent connections, for the right service, I can go as high as $40/month
Comments
you forgot to add price?
I can go as high as $40/month for the right service
@jeffreylroberts , i am not a provider just wanted you to add budget providers always ask! IMHO you should go for big players with unlimited traffic etc ... why don't you check with OVH, hetzner ?
I just tried OVH via Kimsufi, and they struck out, I cant seem to get the crawler to crawl at a decent page per minute rate, so far AWS is the only contender and they charge to much for storage and bandwidth IMO for this project
What? Doesn't your cablemodem stack up?
@wss lol, nah it doesn't
Go with hetzner. gigabit link, unlimited incoming bw.
@teamacc checking them out now, thanks
You might want to try Hetzner or Worldstream.