Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


mapping the internet (routing table collection)
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

mapping the internet (routing table collection)

cloudmappercloudmapper Member
edited July 2017 in General

Hi everyone - I apologize in advance for my lack of knowledge on this subject. I'm interested in monitoring as many complete routing tables as I can possibly get access to in an effort to gain a comprehensive view into peering relationships. I'm aware of RouteViews and similar projects that aggregate public routing tables but I'm interested in getting access to new routing tables beyond what these projects offer.

I also understand there are public route servers I can connect to but I'm not clear on how I would go about downloading the full bgp routing table from them (my understanding is the table is huge and would crush the server's CPU etc.).

Lastly, I've been reading about buying a /24, registering an ASN, and using BIRD / Quagga / exabgp as a BGP daemon on a VPS.

Would this BGP daemon have access to a complete routing table? If I spin up a bunch of VPSes around the world running Quagga would this give me access to many routing tables?

I found this post which seems to indicate this is possible but my BGP daemon would need to peer with another ASN? How would I pull that off? Is that the challenging part in all this? Finding peers?

Thanks for schooling me on the topic and pointing me in the right direction :)

Comments

  • classyclassy Member

    my understanding is the table is huge and would crush the server's CPU etc.

    Not really. You need a few gigs of RAM but most Cisco's ASA are just Intel Atoms (the better performing ones though) and they need to handle routing real time, all the time.

    Thanked by 1vimalware
  • rskrsk Member, Patron Provider

    You need to learn it before you go ahead. Trust me, this will be the best things ever.

    https://dn42.eu/

    Simulate it there, run your own AS, announce your IPs, peer with IX or just some other random guys and learn.

    Thanked by 1vimalware
  • @rsk said:
    You need to learn it before you go ahead. Trust me, this will be the best things ever.

    https://dn42.eu/

    Simulate it there, run your own AS, announce your IPs, peer with IX or just some other random guys and learn.

    and realize in the real world, many ISPs are dicks about peering. "buy my transit instead hahahaha"

    Thanked by 1vimalware
  • cloudmappercloudmapper Member
    edited July 2017

    Ok super helpful! dns24.eu looks awesome, will definitely practice there.

    @doghouch said:
    realize in the real world, many ISPs are dicks about peering. "buy my transit instead hahahaha"

    So the real world is all I care about. It seems the real challenge is finding peers? (please correct me if that's not the case). A few questions on this front:

    1. What does the AS have to do to peer with me? Do I have to engage with a human on their end to make it happen or is it an automated process?
    2. What are transit costs like? Could I buy one month and cancel if all I wanted was routing information?
    3. What do these networks value? If I am peered with many networks, would each network value seeing the routes of the others? I spoke to Neustar last year and they mentioned wanting access to new BGP routes (they called them "feeds") so perhaps others might want this as well?

    Lastly, if I spin up a VPS w/ Vultr or DigitalOcean, how would I identify the list of possible peers available to me?

    Thanks for your help!

  • cubedatacubedata Member, Patron Provider
    edited July 2017

    just why do you need to map all bgp tables now? what is with everyone wanting to use bgp all of a sudden to "map" everything...just why do they need "map" anything that other services provide better?

    nevermind super suspicious the person signed up just to ask this today:

  • cloudmappercloudmapper Member
    edited July 2017

    Well, I did specifically register here to ask this question ;)

    I've been interested in this topic for quite some time but never had the resources (time/money) to actually attempt it.

    Are there others pursuing a similar direction that are considered "bad actors"? Thanks for educating me so I can avoid heading down a similar path.

    And to answer your question re: why. I find the relationship information in routing tables to be very interesting and at times indicative of how markets change (say Netflix begins peering w/ Apple or something to that affect). I have a data science background and I'm interested in performing analysis on these relationships in an attempt to understand what's happening in the "real world".

    Hope this helps!

  • cubedatacubedata Member, Patron Provider

    @cloudmapper said:
    snip

    you should know registering today and this being your first discussion/comment is very suspicious as this has been seen in recent times too often someone wants to "map" bgp tables which in itself is very suspicious when someone first joins and this is the first you ask about.

    hopefully you understand it now since the important parts are now bold and some italicized.

  • WSSWSS Member

    Yow! Zippy wants th' BIG BGP BURGER, but is afraid th' OSPF might go right back out!

  • lol! :) What is suspicious about aggregating / mapping bgp tables? More bold and italics please!!

  • WSSWSS Member

    That depends on your intentions.

  • I'm just trying to learn here. Can you give an example of bad intentions?

  • WSSWSS Member

    I'll defer to @William. I don't like to masturbate over routing, because I don't get paid for it- and he knows more than I've forgotten.

  • Thanks @clouvider! Yes, I'm familar w/ RouteViews. I'd like to add to their dataset (create something more comprehensive in nature).

  • @cloudmapper said:
    Thanks @clouvider! Yes, I'm familar w/ RouteViews. I'd like to add to their dataset (create something more comprehensive in nature).

    I was nodding along with your posts until I read this.

    Oh well, good luck.

  • Thanks @vimalware. Could you explain why? :)

  • rskrsk Member, Patron Provider
    edited July 2017

    I recall, not 100% positive, that with Vultr you can either bring in your own AS or use a private one from them, and you can announce your own subnet. AFAIK this has to be done in a ticket, and there is some human touch involved; it is not automated.

    Edit: https://www.vultr.com/docs/configuring-bgp-on-vultr

  • TomTom Member

    @cubedata said:

    @cloudmapper said:
    snip

    you should know registering today and this being your first discussion/comment is very suspicious as this has been seen in recent times too often someone wants to "map" bgp tables which in itself is very suspicious when someone first joins and this is the first you ask about.

    hopefully you understand it now since the important parts are now bold and some italicized.

    To be honest I don't really see the problem, seems like a cool project to do. Best of luck @cloudmapper!

  • ClouviderClouvider Member, Patron Provider
    edited July 2017

    I don't see what can you possibly add to their dataset heaving a handful of full tables from a few ISPs?

  • r0xzr0xz Member

    @cubedata said:

    @cloudmapper said:
    snip

    you should know registering today and this being your first discussion/comment is very suspicious as this has been seen in recent times too often someone wants to "map" bgp tables which in itself is very suspicious when someone first joins and this is the first you ask about.

    hopefully you understand it now since the important parts are now bold and some italicized.

    what's wrong with asking a question??? not like he's promoting some shit here. chill out!

  • edited July 2017

    @cloudmapper, I think you are doing a really neat project to learn and expand your horizons. Don't give a rats ass what @cubedata says, he has no idea wtf he is talking about just spoutting a bunch of dumb stuff his words don't mean shit. Frankly speaking, all he is saying is "suspicious", "suspicious", "your first day" without furthering articulating how are your intentions "suspicious"

    He lacks ambitions to do cool projects like this one you are doing, and doesn't have the voracious appetite to learn like you. So he labels everything above his level of understanding(or what he considered normal learning of understanding) "very suspicious", because it took him quite a while to understand BGP routing.

  • r0t3nr0t3n Member

    @cloudmapper I currently do this sort of thing with Vultr, and even run my own modified Bird daemon to get data out programmatically easier.

    The big issue you have is you will not see all routes from your direct BGP peers, you can see the full routing table but this is default/best routes, you will not see every route held without using BGP extensions like bgp add path.

    Feel free to send me a PM maybe we can pull some resources together.

  • cloudmapper said: lol! :) What is suspicious about aggregating / mapping bgp tables? More bold and italics please!!

    No idea, it is fairly worthless data in malicious ways, and even then it can be replaced by something as simple as traceroute (which will give the ASN... sometimes...).

    cloudmapper said: and at times indicative of how markets change (say Netflix begins peering w/ Apple or something to that affect).

    If we stay with that example, this likely means nothing: Apple is not a content provider for Netflix (and even then it would be mirrored only once and is then in netflix), neither reverse, and none does end-user connectivity that would matter as transit/transit-peer.

    Anyway from the tech side, you generally do not see what you have no path to - unless you have access to something via Apple or Netflix and record the route to the other side you have, simply, no data.

    cloudmapper said: It seems the real challenge is finding peers

    Peer does ultimately not mean free. Many gladly sell you transit via BGP, without IP blocks or customers to sell said transit to (you do not need IPs for that, generally) this is... fairly useless though.

    cloudmapper said: If I am peered with many networks, would each network value seeing the routes of the others

    No, as generally you only use a free (thus exchange or direct) peer for what they send you - you send only your and your customer routes; why would you send your private peering to ISP Z to IX A users for free usage. Would be dumb.

    A transit provider will generally have a path to nearly any prefix and send you all (in case of a full table) which you can use with another ISP that does this and your IX/private peerings to use the shortest (or other regex like known reliability of path). BGP 1x1.

    cloudmapper said: What do these networks value?

    Well, what do you offer?

    • I am on a local IX as i do not pay the crossconnect and port is cheap (few cent/Mbit), any traffic i can get there off is very cheap to me, nearly free
    • I have BGP downstream users with default route (for simple anycast) and full table that get all routes for, well, anycast

    etc etc

    cloudmapper said: What are transit costs like? Could I buy one month and cancel if all I wanted was routing information?

    Depends where. Depends on definition of "transit". Full transit from basically 10c to 150$ depending on location (eg. low = Amsterdam, LA or New York, med = Tokyo or Hong Kong, high = Brazil or South Africa + China direct at the top).

    EU peering only (so a blend of multiple exchanges and backhaul through the ISP network basically, but no transit routes they pay for) can be had for few cent per Mbit.

    No, month to month is rare for anything requiring a crossconnect to colo, VPS/dedicated with BGP for just a month yields the provider likely hating you also (setting up session for just 1mo is damn annoyance especially if done free as value service for customers).

    Transit contracts range from 6mo to 10yr with clauses from very, very simple (eg. Cogent) to complex as hell (transit eg. Telmex and Claro, peering UPC and Comcast).

    Thanked by 1Aidan
Sign In or Register to comment.