New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Multiple VPS with multiple IPs
I am working on a distributed crawler where I would need multiple servers with multiple IPs with cheapest possible price. There are two distinct type of VPSes that I would need.
- Workers with decent amount of CPU and RAM
- Routers that will used to route web requests to crawl data.
I am more interested in the second type since there are plenty of VPSes available for workers. Any suggestions or offers?
Comments
what are the exact specs you are looking for them. i can make a custom deal for you. and we can also offer you a vps for second option, since unmetered bandwidth is going to help in that case
The routers can be very lightweight. A 128mb VPS with good network should be fit for the job.
You can try here for $2 per month http://lowendbox.com/blog/host1plus-com-2month-256mb-openvz-in-south-africa-brazil-germany-or-the-usa/
Why does this need multiple IPs? Do these crawlers not respect robots.txt and get blocked?
@rds100 It is not a wholesale spider like 80legs. We have specific list of URLs to fetch and monitor.
@tija i think it would be best to use something like Amazon or the other cloud providers, where incoming traffic is free. That is if you have a lot of incoming traffic of course.
Do you need IPv4 or will v6 do?
@wych Will definitely need IPv4 since there are many servers that needs IPv4. That being said for some servers we could try out IPv6 we get a good deal.
Do you have a price in mind? We have a decent 256mb RAM plan that would handle your router for crawling.
http://slicify.com/
Add me on skype vivek.murai
Please open a sales ticket at https://beastvm.com/client/submitticket.php?step=2&deptid=1 or contact me on skype: BeastVM_Floris so we can work out a deal.