All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hosted Machine or Purchase Hardware
Hey guys, I am on a project including GeoCoding. The first step is importing full data (the entire planet!) from OpenStreetMaps (OSM) into PostgreSQL. The second step is GeoCoding our DB of 20K locations, including full building-perimeter polygons. This second step is essentially a one-time job.
On an ongoing basis, we will need to refresh the full OSM database, and new locations will be added, but infrequently. In theory, we could run these new GeoCoding tasks once a week and keep a VPS off the rest of the time.
I tried importing just North America on my PC (16 GB RAM, 4 CPU cores) and it took over 36 hours. GeoCoding locations on the same machine takes a few minutes each.
So, it seems we need a machine with roughly 12–16 CPU cores, 64 GB RAM, and a 2 TB SSD to run efficiently. North America alone is a few hundred GIGs of disk space.
The question is whether we should purchase such a machine or rent a VPS or a dedi server. If we purchase, we could perform the initial 20K import locally and then expose the PostgreSQL database to the live server via SSH port forwarding, for ongoing GeoCoding.
Short-term, renting is cheaper, but long-term, owning the hardware likely makes more sense. There may be other considerations I haven’t thought of however.
Comments
Rent the machine or RTO.
Renting would be cheaper!
Big boys like @PureVoltage @crunchbits @fiberstate can provide that spec for very good price, even if you buy that server and colo with them, difference would be like $10-$15/mo
If you're going to host at home, then buying would be best, roughly around $200-$350 if you look at the right place.
Renting is always cheaper, until you hit a 'critical mass' of equipment. But as @kuroit said - hosting at home can also be affordable and you don't deal with the logistics of shipping, remote hands, networking configs, firewalling and etc. which usually drive overall costs up.
Renting is a turnkey service, where you get the machine as a service and you don't deal with anything else, other than your actual work.
Should you need a dedicated machine, hit me up. We can probably get these specs up for you for 70-80EUR per month @spolenta.
Alex
@kuroit Yes, I would host it at home.
Dude, $350 for such a machine? If you can find me that, I will buy it right now!
But one issue is that I am in Tel Aviv, so unless the ship overseas, a US deal won't help too much...
Why just not use any of the already existing services for 20k locations? It will be cheaper.
Do you expect locations to shift IRL that you gonna run it every week? o.O
@JabJab Can you suggest one? I tried one and it it didn't work.
Technically speaking, tectonic plates are constantly moving due to convection currents within the Earth's mantle.
Regards
What did you try? This is a dedicated server teritory for sure, not a VPS.
Also, if you're importing hardware be careful with import taxes and customs fees. I would generally buy from EU.
I tried several API methods:
By storing OSM building polygons locally (PostGIS/PostgreSQL) and matching them to addresses, I reliably get accurate polygons.
Have you tried https://developers.google.com/maps/documentation/geocoding/building-attributes?hl=he ?
(&extra_computations=BUILDING_AND_ENTRANCES)
Not sure why you'd do all that for 20k locations instead of just use an exiting API... Geocode.Farm (biased lol), Google, Mapbox, etc all have solutions.
20k is an extremely low amount to run a full osm node for.
You can even use the osm nominatim API for that low of an amount...
Running a whole server to rarely use seems very very redundant and not at all cost or time effective.
First of all, thanks everyone who responded, and thanks to those who private messaged me. This is a great forum!
@OhJohn I was not aware of this Google API, but it doesn't actually help. I tested several addresses using this API. The results are just rectangular bounding boxes around the address -- each is just a 300×250 meter rectangle. That's what they mean when they say there:
Google’s method is fine for visualization, but not for precise geofencing, which is what I need.
Geocode.Farm only returns lat/lon points, not building shapes.
Mapbox has building footprints in its tiles, but no API to fetch polygons, as far as I can see.
Also Nominatim only gives you points (lat/lon) and bounding boxes, not building polygons.
Hm, did you not get sth. like:
.
.
.
with a lot of coordinates for the polygon in the result? Rectangular bounding boxes do not really sound like the polygon (except when the building is itself a rectangle) you should get.
But yes, that (new) extra seems to be in beta state, so: ymmv depending on where on earth you try that.
Yes, I tested Google Maps building outlines. For all our addresses, it only returned rectangular bounding boxes, not the detailed building polygons you might expect. Even though the API mentions
"building_outlines", in practice it’s just a rectangle per address.After visualization of the results, I had an AI bot confirm for me that each rectangle is 300x250, and I tested on a few buildings in USA, in normal (not rural) areas.
[
Hm, they (g geoc w/ buildings_and_entrances extra) have some building polygons, this one example is "3151 Mt Pleasant St NW, Washington, D.C., District of Columbia, USA" but they seem to lack the ones you need then.
(Don't use the bounding box from the result, that should be a rectangular, but the display_polygon. That is actually the same with OSM, e.g. Nominatim: if the address matches a building mapped as a polygon and not just a node or street, you could get a polygon).
Umm, dude, @OhJohn, can I buy you a beer?
I was using it wrong, and now that I got it right, you are correct, it works!
You rock, buddy. Thanks a lot!