Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


yammdb - just another .mmdb
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

yammdb - just another .mmdb

NeoonNeoon Community Contributor, Veteran

Hey,

Today I wanted to present you, yammdb.
Which is another, different, geodatabase, based on real measurements.

You can find it here: https://github.com/Ne00n/yammdb
The Database is build weekly, on fridays.

As long the buildserver doesn't blow up.
The primary use case for me is to compare existing geo data, maybe someone of you will find it useful.

If you have any ideas and feedback, please lemme know.

Enjoy.

«1

Comments

  • LeviLevi Member

    What's the practical purprose of this database?

    Thanked by 1xms
  • NeoonNeoon Community Contributor, Veteran

    @LTniger said:
    What's the practical purprose of this database?

    The reason why I have created it and was, to check existing geo data how accurate it is.
    Its not a drop in database replacement, the coverage for that would be to low.

  • Can you tell something about the methodology?

  • BasToTheMaxBasToTheMax Member, Host Rep

    @Neoon said:
    Hey,

    Today I wanted to present you, yammdb.
    Which is another, different, geodatabase, based on real measurements.

    You can find it here: https://github.com/Ne00n/yammdb
    The Database is build weekly, on fridays.

    As long the buildserver doesn't blow up.
    The primary use case for me is to compare existing geo data, maybe someone of you will find it useful.

    If you have any ideas and feedback, please lemme know.

    Enjoy.

    Where do you get all your data from?

  • NeoonNeoon Community Contributor, Veteran

    @rick2610 said:
    Can you tell something about the methodology?

    Huh?

    @BasToTheMax said:
    Where do you get all your data from?

    Just simply by pinging everything in the internet.

  • @Neoon said: Just simply by pinging everything in the internet.

    Have you received any "abuse" reports for it(or reports on abuseipdb)?
    It would've been pretty funny to see people complain about legitimate ICMP, and it would not surprise me at all.

  • NeoonNeoon Community Contributor, Veteran

    @treesmokah said:

    @Neoon said: Just simply by pinging everything in the internet.

    Have you received any "abuse" reports for it(or reports on abuseipdb)?
    It would've been pretty funny to see people complain about legitimate ICMP, and it would not surprise me at all.

    No, I never received any.

    Thanked by 1treesmokah
  • This can be classified as "internet scanning" and this is illegal in USA.

  • treesmokahtreesmokah Member
    edited May 2023

    @Hotmarer said:
    This can be classified as "internet scanning" and this is illegal in USA.

    He is not scanning anything, he is just pinging random IP addresses.
    Also scanning is NOT illegal in the USA, just look at Shodan and Censys(both american companies).

    Thanked by 1mrTom
  • NeoonNeoon Community Contributor, Veteran

    @Hotmarer said:
    This can be classified as "internet scanning" and this is illegal in USA.

    Can you point me out the law, that says that?
    Especially icmp, no port scanning or anything related.

    Thanked by 1raindog308
  • GanonkGanonk Member

    💯

  • bgerardbgerard Member
    edited May 2023

    @Hotmarer said:
    This can be classified as "internet scanning" and this is illegal in USA.

    Crossing the road not at a crossing is illegal in the USA...

    On topic: @Neoon could you provide some more details on how you build this db please? I'd love to play around myself.

  • NeoonNeoon Community Contributor, Veteran

    @bgerard said:

    @Hotmarer said:
    This can be classified as "internet scanning" and this is illegal in USA.

    Crossing the road not at a crossing is illegal in the USA...

    On topic: @Neoon could you provide some more details on how you build this db please? I'd love to play around myself.

    I wrote something on LES about creating your own .mmdb a while back.
    https://lowendspirit.com/discussion/5791/how-to-create-your-own-mmdb-for-gdnsd-or-any-other-nameserver

  • @Hotmarer said:
    This can be classified as "internet scanning" and this is illegal in USA.

    Shodan does this and theyre based out of Seattle...

  • MannDudeMannDude Host Rep, Veteran
    edited May 2023

    So, is this basically GeoIP but based on actual routes taken to the IP? Because I hate standard GeoIP DBs since they're man made and based on data that isn't confirmed. Additionally, too many DB's out there used for different services and it's annoying when (for example) these three popular GeoIP lookup tools say your server is in XYZ, but this other one is never updated and still shows ZYX. Currently relying on a geofeed in ARIN comments which is picked up by most, but not all.

    /rant

    Let me know if you want a server in the US or NL and what is required, happy to sponsor one.

    Thanked by 1BasToTheMax
  • NeoonNeoon Community Contributor, Veteran

    @MannDude said:
    So, is this basically GeoIP but based on actual routes taken to the IP?

    Yes, however the hit rate, is quite lower than compared to standard geo databases.
    Usually they have a 99%+ hit rate, this build has a 64% hit rate, but relies on actual data no guesses.

    After fixing a few bugs its up to 74%, I prob. need better data and need to make more optimizations.

    Let me know if you want a server in the US or NL and what is required, happy to sponsor one.

    Neat, I will.

  • BasToTheMaxBasToTheMax Member, Host Rep
    edited May 2023

    @Neoon said: After fixing a few bugs its up to 74%, I prob. need better data and need to make more optimizations.

    I have a few questions:

    • How long will it take to ping every subnet?
    • How many subnets/ips do you ping per second?
  • NeoonNeoon Community Contributor, Veteran
    edited May 2023

    @BasToTheMax said:

    @Neoon said: After fixing a few bugs its up to 74%, I prob. need better data and need to make more optimizations.

    I have a few questions:

    • How long will it take to ping every subnet?

    No idea about the exact number.
    Its done in less than a day.

    • How many subnets/ips do you ping per second?

    I didn't measure it.
    20% of the time is wasted just for waiting, which is necessary but still.

  • @Hotmarer said:
    This can be classified as "internet scanning" and this is illegal in USA.

    I don't think this qualifies as "unauthorized access".

    It's probably the equivalent of knocking on someone's door and walking away.

  • It should be "yet another mmdb" or change it to jammdb for "just another mmdb".

  • NeoonNeoon Community Contributor, Veteran

    @TimboJones said:
    It should be "yet another mmdb" or change it to jammdb for "just another mmdb".

    Yea but I leave it.

  • NeoonNeoon Community Contributor, Veteran

    Thanks to https://virtury.com/ we got a new Probe in Pakistan!

  • NeoonNeoon Community Contributor, Veteran
    edited May 2023

    I took a bit longer than expected, however the software is now mostly optimized for more probes.
    Expect more probes in the next weeks.

    Daily test builds, not guaranteed, will be available under https://yammdb.serv.app/test.mmdb
    Weekly build will happen as usual.

    Also, thanks to https://ginernet.com for a new probe in Madrid.

    Thanked by 1mrTom
  • BasToTheMaxBasToTheMax Member, Host Rep

    @Neoon said: Daily test builds, not guaranteed, will be available under https://yammdb.serv.app/test.mmdb

    Weekly build will happen as usual.

    It gives me a 404 error page.

  • NeoonNeoon Community Contributor, Veteran

    @BasToTheMax said:

    @Neoon said: Daily test builds, not guaranteed, will be available under https://yammdb.serv.app/test.mmdb

    Weekly build will happen as usual.

    It gives me a 404 error page.

    The build started a few hours ago, will take some time.

  • NeoonNeoon Community Contributor, Veteran

    @Neoon said:
    I took a bit longer than expected, however the software is now mostly optimized for more probes.
    Expect more probes in the next weeks.

    Daily test builds, not guaranteed, will be available under https://yammdb.serv.app/test.mmdb
    Weekly build will happen as usual.

    Also, thanks to https://ginernet.com for a new probe in Madrid.

    It was supposedly to be already done, however I did a fuck up.
    One function had the build hang for hours over hours.

    This is fixed, thanks to GPT4 once again.
    It should finish in a bit, once this is done, I will make a second test build, including a bunch of new locations.

  • NeoonNeoon Community Contributor, Veteran

    Got a bunch more locations, for this Friday build.
    Gonna be interesting.

  • BasToTheMaxBasToTheMax Member, Host Rep

    Hi Neoon, maybe you could use GlobalPing. They have servers in 66 different countries.

  • NeoonNeoon Community Contributor, Veteran

    @BasToTheMax said:
    Hi Neoon, maybe you could use GlobalPing. They have servers in 66 different countries.

    Technically yes, Practically no.
    I don't think that @jimaek is going to be happy when I would send him a few million icmp requests.

    Yes we could spread them out as we do now, however, I still don't think this is a good idea.
    Same goes with Ripe Atlas.

  • kaitkait Member

    icmp flood the world :) Interesting project hope it goes well, will you do a comparison against maxmind and other ip2location services?

This discussion has been closed.