All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Tips for running a (hobby) Autonomous System?
Hi,
I run my own AS as a hobby.
I started playing around with DN42 last year, and got interested enough that I also wanted to participate in the "real internet". Currently, I only run my website in my IP prefix (and I also set up the DNS servers for the reverse zone). Possibly I'll also play around with Geolocation in the future (https://xkcd.com/713/) ("eyeballing"?).
My AS currently consists of three VPS (Hetzner in Nuremberg, FirstRoot in Düsseldorf, and VirtuaCloud in Paris), All running Debian as OS with Bird as routing daemon. BGP peerings are done directly with the providers (FirstRoot, Virtua) or via a tunnel (iFog BGPTunnel, FreeTransit) [4]. The VPS are connected via Wireguard tunnels and I run Babel as IGP over it [1].
The VirtuaCloud VPS is the newest, I was initially impressed with how well they are connected (their upstreams). However, now I realize that this leads to a "problem" with the latency: The AS path to VirtuaCloud is the best, but "all my content" resides on the German servers, so all the traffic needs to get shuffled through the Wireguard tunnel from Paris to Germany [2]. I'm not sure if I "actually gain" anything by virtua's better upstream connection, or if the inter-server traffic kinda "destroys" this gain [3].
I should mention, I have no "formal education" in this field (networking). I'm self-taught in this regard. This makes me now question my choice of the setup. I found enough resources for setting up bird, but haven't found any for "properly running" an AS. Can you offer me some pointers (links, resources…) what I could look into? I know Anycast is a thing (haven't looked into it); In theory I know you can do "traffic engineering" with the proper bird filters, I know what path prepending is… I kinda ran into a brick wall now and don't know where to go from here.
It's still only a hobby, I'm not really willing to spend any more money on it, these 3 VPS with max. 5 € / month in the most I'm willing to spend, i.e. I'm going with e.g. iFogs IXP VMs. And yes, I am aware of the upcoming RIPE AS fees, which in itself seem still fine for me for continuing this as a hobby.
[1] Also, iBGP between Hetzner and FirstRoot. I think I set this up as a "fallback", in case the tunnel BGP session on the Hetzner server goes down, as I can't peer with Hetzner with only a VPS.
[2] Which shouldn't matter as it's "only a static website".
[3] traceroute fluctuates as well, which shouldn't surprise me. BGP updates happen all the time.
[4] Personally, I found route64 and tunnelbroker.ch unreliable.
Comments
You are doing well.
Does it have to be a tunnel? Also, can't the content be replicated in the VM?
I would setup this more like a star topology, not a circle, i.e. everything connected with each other for YOU, while leaving the other ends connect over the rest of the net with each other, taking the best possible routes.
If you would like to have your own, fully contained, LAN over the internet, then there are much better setups out there.
Isn't route64 died?I can not creat any tunnel or bgp session on it
I think tunnelbroker.ch works well for me
If you have a total budget of around 20 Euro per month(equivalent to your 3 vm each at 5 euro), you could probably colo a routerboard with MyRootPW in amsterdam. We can connect you to a few IXP's (ie: 100m to fog-ix, 1G to eraIX, 1G to frysIX) and give you a bgp session for transit.
That however would have a one-time fee for some mikrotik hardware.
If you want to continue the software-router route, then maybe a low budget dedi. For those we can also do connections to IXP's
regards
I have a few pointers, although nothing comprehensive:
Hope I was of any help!
[1] Since AS6939 Hurricane Electric provides transit over IXPs problems happen when people local-pref the IX port and with it the AS6939 routes. In this scenario it can be desirable to have your provider specifically prepend your routes to AS6939 so they aren't preferred by AS6939 peers.
What alternatives are there to a tunnel, I guess you mean this outside of the different tunnel protocols / technologies?
I am not sure of it, but were I to put the website on all 3 VPS, give all 3 VPS the "web server IP" (www. in DNS), would this already be anycast? I read that anycast with TCP has its own set of problems (BGP path changes during the TCP stream? Then again, it's "only loading a small website", not a video stream or anything? But then again, Murphy is sure to strike).
Next up on the list would probably be "DNS anycast" (Returning the "closest" IP address, what, to my knowledge, Cloudflare is doing. More or less: all servers (or two) running a DNS server, and each server returning their respective IP for www.)?
(emphasis mine)
For the rest, I don't think I can quite follow. I'm not getting what you mean by star topology. With only three servers currently connected to each other, "cutting" one connection would make it a star topology? Also, what ends should connect to where?
Hm. So I'm not really "supposed" to distribute my AS over multiple VPS on different providers?
Or is it just the choice of technology (Wireguard + Bird) that's not optimal?
I mean, I know and used Tinc, for example, which "automatically finds it connections" given the node has at least one peer.
The servers in Germany never worked for me, and I never got a reply from support.
Thanks for the pointers.
I guess that would that be the
rxcostoption in bird, and/or thertt costone?Interestingly, I would've assumed that it's actually preferable to have the HE upstream. I guess that could lead to a worse latency in that case?
There are a few ways, from replication to reverse proxies.
well, that would be a way too, but is then the best way to serve the same content from the 3 VMs?
I understand you are trying to learn and anycast is probably the next step, but I have a different method of learning, when I need to do something, then I study the problem and find the best (to my knowledge and ability) way to solve it, with the fewest SPoFs as possible. I am not learning just for learning's sake, I would definitely forget everything in a few years AND the technology would likely change, so, while you could choose any method you would like just to learn it, and I can understand some people do like to learn new things, but it is not my way.
For example, if I were you, I would be going at the bleeding edge of the development, since I have to learn new things anyway, and I would design a kind of layer 8, a mesh over the current Internet routing, with self-healing, autorouting, maybe distributed storage and self-contained services, like webservers, as well as gateways to the rest of the net, where everyone contributes and shares, a kind of Tor/i2p/Kademlia/Freenet whatever, all-in-one-wonder-pro with complete anonymity and resistance to censorship. A kind of NI, Netizen's Internet, own routing, own storage, own encryption based on some "currency" or token, score, etc, for storage, traffic shared, routing etc.
You would learn a lot in the process, from the design up.
€50/year RIPE invoice incoming.
You mean at least 60
@Maounique seems like a great way of learning indeed. I agree hands-on / practical projects and doing it yourself is a great method to learn things. That's how I learned developing for microcontrollers. Yeah, I started with some Arduino libraries, but after I while I went with actual register accesses (for a while, until I understood things). Alas, my day job, doing microcontroller stuffs, currently leaves me with no motivation to do much programming in my free time. I hope this will change in the future.
@yoursunny / @kait yeah, I mean, I would go with a 70-/80-ish AS + IPv6 prefix bundle. That's still "only" about 6 € / month. I'd probably drop one of the VMs for that and still be within my (probably laughably tiny, depending on who you ask) budget
i don't get why €60/year is such a big deal for AS (especially if it's for your hobby, did you not have fun with that?). I'm sure most of LET poster make €250k/year
The most important part is finding the right building to squat. Also since there might be people trying to stop you i recommend bringing a knife and a big dog.
In Maryland we are paid $15/hour.
A full time worker can make $600/week or $31200/year.
I didn't even want to do it professionally and here you are doing it for fun. How about you run mine and I can never think about it again. I joke, but I wish I didn't 🤣
Because it was a onetime cheap price at first.
Right
I feel this everytime someone buys servers for fun
I mean, I can be quite content with my 48k€ (before taxes).