Tips for running a (hobby) Autonomous System?

andreasd

Hi,

I run my own AS as a hobby.

I started playing around with DN42 last year, and got interested enough that I also wanted to participate in the "real internet". Currently, I only run my website in my IP prefix (and I also set up the DNS servers for the reverse zone). Possibly I'll also play around with Geolocation in the future (https://xkcd.com/713/) ("eyeballing"?).

My AS currently consists of three VPS (Hetzner in Nuremberg, FirstRoot in Düsseldorf, and VirtuaCloud in Paris), All running Debian as OS with Bird as routing daemon. BGP peerings are done directly with the providers (FirstRoot, Virtua) or via a tunnel (iFog BGPTunnel, FreeTransit) [4]. The VPS are connected via Wireguard tunnels and I run Babel as IGP over it [1].

The VirtuaCloud VPS is the newest, I was initially impressed with how well they are connected (their upstreams). However, now I realize that this leads to a "problem" with the latency: The AS path to VirtuaCloud is the best, but "all my content" resides on the German servers, so all the traffic needs to get shuffled through the Wireguard tunnel from Paris to Germany [2]. I'm not sure if I "actually gain" anything by virtua's better upstream connection, or if the inter-server traffic kinda "destroys" this gain [3].

I should mention, I have no "formal education" in this field (networking). I'm self-taught in this regard. This makes me now question my choice of the setup. I found enough resources for setting up bird, but haven't found any for "properly running" an AS. Can you offer me some pointers (links, resources…) what I could look into? I know Anycast is a thing (haven't looked into it); In theory I know you can do "traffic engineering" with the proper bird filters, I know what path prepending is… I kinda ran into a brick wall now and don't know where to go from here.

It's still only a hobby, I'm not really willing to spend any more money on it, these 3 VPS with max. 5 € / month in the most I'm willing to spend, i.e. I'm going with e.g. iFogs IXP VMs. And yes, I am aware of the upcoming RIPE AS fees, which in itself seem still fine for me for continuing this as a hobby.

[1] Also, iBGP between Hetzner and FirstRoot. I think I set this up as a "fallback", in case the tunnel BGP session on the Hetzner server goes down, as I can't peer with Hetzner with only a VPS.
[2] Which shouldn't matter as it's "only a static website".
[3] traceroute fluctuates as well, which shouldn't surprise me. BGP updates happen all the time.
[4] Personally, I found route64 and tunnelbroker.ch unreliable.

Maounique

@andreasd said: I should mention, I have no "formal education" in this field (networking). I'm self-taught in this regard.

You are doing well.

@andreasd said: The AS path to VirtuaCloud is the best, but "all my content" resides on the German servers, so all the traffic needs to get shuffled through the Wireguard tunnel from Paris to Germany

Does it have to be a tunnel? Also, can't the content be replicated in the VM?
I would setup this more like a star topology, not a circle, i.e. everything connected with each other for YOU, while leaving the other ends connect over the rest of the net with each other, taking the best possible routes.

If you would like to have your own, fully contained, LAN over the internet, then there are much better setups out there.

jasonxu

@andreasd said: [4] Personally, I found route64 and tunnelbroker.ch unreliable.

Isn't route64 died?I can not creat any tunnel or bgp session on it
I think tunnelbroker.ch works well for me

SGraf

If you have a total budget of around 20 Euro per month(equivalent to your 3 vm each at 5 euro), you could probably colo a routerboard with MyRootPW in amsterdam. We can connect you to a few IXP's (ie: 100m to fog-ix, 1G to eraIX, 1G to frysIX) and give you a bgp session for transit.

That however would have a one-time fee for some mikrotik hardware.

If you want to continue the software-router route, then maybe a low budget dedi. For those we can also do connections to IXP's

regards

kjartan

@andreasd said:
In theory I know you can do "traffic engineering" with the proper bird filters, I know what path prepending is… I kinda ran into a brick wall now and don't know where to go from here.

I have a few pointers, although nothing comprehensive:

• Pretending is the simplest form of traffic engineering, but it turns out to be surprisingly effective due to how BGP selects the best path. In a real world example, my own anycast, I attempt to keep the path length observed by ISPs via transit roughly the same in every single PoP, so that traffic is likelier to hit the actual closest PoP and not one transited by a network with really good peering.
• If your transits have action BGP communities for local preference and prepends it may sometimes be desirable to use those for traffic engineering as you gain partial control over the behaviour of a router that's not yours. A very common example is prepending n times to peers on an IX your provider is connected to, or even to a specific peer on that IX [1].
• I highly recommend using Babel (or OSPF) link metric, if you haven't already, to influence best path selection based on latency, since you are doing an "overlay "backbone"" of sorts. I have used average latency divided by two as my OSPF metric but this is kind of a shot in the dark that seemed to work alright. But like Maounique has mentioned there are less fragile/inefficient ways of doing this kind of experimentation using overlay networks.

Hope I was of any help!

[1] Since AS6939 Hurricane Electric provides transit over IXPs problems happen when people local-pref the IX port and with it the AS6939 routes. In this scenario it can be desirable to have your provider specifically prepend your routes to AS6939 so they aren't preferred by AS6939 peers.

andreasd

@Maounique said:
Does it have to be a tunnel? Also, can't the content be replicated in the VM?

What alternatives are there to a tunnel, I guess you mean this outside of the different tunnel protocols / technologies?
I am not sure of it, but were I to put the website on all 3 VPS, give all 3 VPS the "web server IP" (www. in DNS), would this already be anycast? I read that anycast with TCP has its own set of problems (BGP path changes during the TCP stream? Then again, it's "only loading a small website", not a video stream or anything? But then again, Murphy is sure to strike).
Next up on the list would probably be "DNS anycast" (Returning the "closest" IP address, what, to my knowledge, Cloudflare is doing. More or less: all servers (or two) running a DNS server, and each server returning their respective IP for www.)?

@Maounique said:
I would setup this more like a star topology, not a circle, i.e. everything connected with each other for YOU, while leaving the other ends connect over the rest of the net with each other, taking the best possible routes.

(emphasis mine)

For the rest, I don't think I can quite follow. I'm not getting what you mean by star topology. With only three servers currently connected to each other, "cutting" one connection would make it a star topology? Also, what ends should connect to where?

@Maounique said:
If you would like to have your own, fully contained, LAN over the internet, then there are much better setups out there.

Hm. So I'm not really "supposed" to distribute my AS over multiple VPS on different providers?
Or is it just the choice of technology (Wireguard + Bird) that's not optimal?
I mean, I know and used Tinc, for example, which "automatically finds it connections" given the node has at least one peer.

@jasonxu said:
I think tunnelbroker.ch works well for me

The servers in Germany never worked for me, and I never got a reply from support.

@kjartan said:
I highly recommend using Babel (or OSPF) link metric [...]

Thanks for the pointers.
I guess that would that be the rxcost option in bird, and/or the rtt cost one?

@kjartan said:
Since AS6939 Hurricane Electric provides transit over IXPs problems happen when people local-pref the IX port and with it the AS6939 routes.

Interestingly, I would've assumed that it's actually preferable to have the HE upstream. I guess that could lead to a worse latency in that case?

Maounique

@andreasd said: I am not sure of it, but were I to put the website on all 3 VPS

There are a few ways, from replication to reverse proxies.

@andreasd said: give all 3 VPS the "web server IP" (www. in DNS), would this already be anycast

well, that would be a way too, but is then the best way to serve the same content from the 3 VMs?

I understand you are trying to learn and anycast is probably the next step, but I have a different method of learning, when I need to do something, then I study the problem and find the best (to my knowledge and ability) way to solve it, with the fewest SPoFs as possible. I am not learning just for learning's sake, I would definitely forget everything in a few years AND the technology would likely change, so, while you could choose any method you would like just to learn it, and I can understand some people do like to learn new things, but it is not my way.

For example, if I were you, I would be going at the bleeding edge of the development, since I have to learn new things anyway, and I would design a kind of layer 8, a mesh over the current Internet routing, with self-healing, autorouting, maybe distributed storage and self-contained services, like webservers, as well as gateways to the rest of the net, where everyone contributes and shares, a kind of Tor/i2p/Kademlia/Freenet whatever, all-in-one-wonder-pro with complete anonymity and resistance to censorship. A kind of NI, Netizen's Internet, own routing, own storage, own encryption based on some "currency" or token, score, etc, for storage, traffic shared, routing etc.

You would learn a lot in the process, from the design up.

yoursunny

@andreasd said:
It's still only a hobby, I'm not really willing to spend any more money on it, these 3 VPS with max. 5 € / month in the most I'm willing to spend

€50/year RIPE invoice incoming.

kait

@yoursunny said: €50/year RIPE invoice incoming.

You mean at least 60

andreasd

@Maounique seems like a great way of learning indeed. I agree hands-on / practical projects and doing it yourself is a great method to learn things. That's how I learned developing for microcontrollers. Yeah, I started with some Arduino libraries, but after I while I went with actual register accesses (for a while, until I understood things). Alas, my day job, doing microcontroller stuffs, currently leaves me with no motivation to do much programming in my free time. I hope this will change in the future.

@yoursunny / @kait yeah, I mean, I would go with a 70-/80-ish AS + IPv6 prefix bundle. That's still "only" about 6 € / month. I'd probably drop one of the VMs for that and still be within my (probably laughably tiny, depending on who you ask) budget

ScreenReader

@kait said:

@yoursunny said: €50/year RIPE invoice incoming.

You mean at least 60

i don't get why €60/year is such a big deal for AS (especially if it's for your hobby, did you not have fun with that?). I'm sure most of LET poster make €250k/year

Thundas

@ScreenReader said:

@kait said:

@yoursunny said: €50/year RIPE invoice incoming.

You mean at least 60

i don't get why €60/year is such a big deal for AS (especially if it's for your hobby, did you not have fun with that?). I'm sure most of LET poster make €250k/year

After deducting rent, car & woman, I'm left with nothing.

totally_not_banned

The most important part is finding the right building to squat. Also since there might be people trying to stop you i recommend bringing a knife and a big dog.

yoursunny

@ScreenReader said:
I'm sure most of LET poster make €250k/year

In Maryland we are paid $15/hour.
A full time worker can make $600/week or $31200/year.

jar

I didn't even want to do it professionally and here you are doing it for fun. How about you run mine and I can never think about it again. I joke, but I wish I didn't 🤣

kait

@ScreenReader said:

@kait said:

@yoursunny said: €50/year RIPE invoice incoming.

You mean at least 60

i don't get why €60/year is such a big deal for AS (especially if it's for your hobby, did you not have fun with that?). I'm sure most of LET poster make €250k/year

Because it was a onetime cheap price at first.

emgh

@jar said:
I didn't even want to do it professionally and here you are doing it for fun. How about you run mine and I can never think about it again. I joke, but I wish I didn't 🤣

Right

I feel this everytime someone buys servers for fun

andreasd

@ScreenReader said: I'm sure most of LET poster make €250k/year

I mean, I can be quite content with my 48k€ (before taxes).