One Global Mesh VPN Network for My LEBs: Nebula, or...?
raindog308
Administrator
My LowEndEmpire spans many providers: Hetzner, IncogNET, BuyVM, HyperExpert, RackNerd, InterServer, Vultr, TNAHosting, and whatever that other one is that keeps renewing on PayPal, plus I have home systems.
I was thinking to setup a global VPN network so every host has a 10.x (or whatever) IP and they can talk to each other on this VPN network. I have no actual need for this but it sounds fun.
Considering I barely understand IPv4 this will be a learning exercise. As a bonus I'm planning to do it all via IPv6, which I have never used. (So not a 10.x address actually...) My VPN experience to date has been limited to @Nyr's road warrior script.
So my questions:
(1) I believe what I want is a mesh VPN? Slack's Nebula VPN seems like just the ticket. This would require having highly-available lighthouse servers (because my hobbies are mission critical!) to direct traffic...which of course means getting more LEBs. Drat. Or since the list of nodes changes so infrequently, perhaps I could distribute static maps?
Or should I look at something other than Nebula?
(2) I'm assuming there is a way I could tunnel out of my home so some of my home systems are on this global network? I have the typical consumer NAT setup.
(3) Right now I do backups over rsync/ssh, plus there's some other rsyncs and moving files around and what-not. Would there be a network penalty of going over this fabled VPN network versus public IP? I'm guessing minimal.
(4) I'm assuming I can just use the IPv6 private address range and don't need to go all @Otus9051 and get an ASN.
Comments
Wireguard?
As for the IPv6, many ppl give you /64 which you can distribute to your network OR get one from some tunnel broker. You will not need anything else.
I have been thinking of something similar as well as a new paradigm regarding a layer 8, an encrypted internet on top of the existing infrastructure, completely encrypted and accessible from anywhere with a client of sorts, something like freenet but also for computing and with some kind of beancounting where everyone contributes to a huge supercomputer and data storage.
This.
@raindog308 - Also, just to add, if I recall correctly, @Neoon's built something that you might be interested in.
https://yggdrasil-network.github.io/
That looks interesting, I remember browsing something like it a while ago, but concluded it was too immature yet. I hope I will have some time to test it.
I've tried setting things here yesterday with wireguard, but due to having dynamic DNS at home and (I would guess) shitty router on IP change some (yeah, some...) tunnels are dropped and even after proper refresh ( https://github.com/WireGuard/wireguard-tools/blob/master/contrib/reresolve-dns/reresolve-dns.sh + ddclient running every 5 minutes to update DNS) they can't connect - just spam "no response from endpoint" - from both sides. I've stopped wireguard for 2 hours and one of those reconnected... for 5 minutes and then died again. Fuck it - I can't restart router every IP change.
Went with tailscale testing today, seems to be working nice, website is woah amazing. Yes, I know this goes via 3rd party servers... and this is exactly why I picked this - I assume (hope?) if my router decides to block/choke/shititselfagain the client (that is based on
wireguard) will switch to theirs other endpoint with different IP and re-establish connection ascloudflaredtunnel is working fine, for monthsFrom what I've seen tailscale is not the fastest one, but I don't think I will be pushing > 300Mbit/s here.
also because 20 devices is not enough for any idler: https://www.reddit.com/r/selfhosted/comments/z0vg77/psa_you_can_get_tailscale_with_increased_maximum/
But tailscale systemd is meh - seems it returns faster than interface is actually created and for things that I wanted to bind strictly to tailscale interace/IP I had to run
ExecStartPre=/bin/sleep 5becauseRequiredandAfterwasn't enoughTried to configure my wireguard earlier with
wg-meshconf, but it's broken by design. AllowedIPs have wrong netmask; PersKeepAlive is applied on wrong end... totally not readyGoing through a third party is a deal breaker, I am sure.
Yeah, frequent disconnects and IP changes are a PITA for any complex VPN, I know as at some point I had very bad power dropping multiple times a day, but IIRC there was some setting for aggressive reconnect or renegotiation or something. It is too late to look it up now, I am going to bed, but, if your connection does not drop multiple times a day, wireguard will do just fine.
Wireguard is perfect for this.
What about Netmaker?
https://www.netmaker.org/getting-started
I am using Netmaker (Kernel wireguard) & HeadScale (Self hosted Tailscale control plane/Userspace WireGuard)
There are some other WG based soulutions like Netbird, Innernet. Non WG based ZeroTier.
This all is self-hosted solutions. There is hosted TailScale, ZeroTier available too.
I came to recommend Yggdrasil.
https://github.com/Ne00n/wg-mesh obviously.
@JabJab I use Wireguard (granted its the one in pfSense) in a site-to-site setup with one of the sites in a DHCP setup. Using No-IP and their DUC I haven't had any complaints from the client for quite a few years and before using Wireguard was using OpenVPN.
@raindog308 an overall mesh I'm not familiar too much with but what about maybe a site-to-multisite VPN setup? (Yes I know this is for pfSense but gives an idea)
https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2ms.html
I use Zerotier; it's very good. The only thing that is annoying is when you reboot a server and it decides not to detect it being up, so a quick ping and away you go.
i highly recommend tailscale, but if you prefer to self-host the control server use headscale.
If using public IPs would something no UptimeRobot work on keeping it alive?
Tinc in switch mode (or even plain VXLAN) if you need L2 (e.g. for VRRP/CARP), not as performant as the previously mentioned L3 tunnels though.
Tailscale with Headscale as the control plane is nice if your clients are behind hostile NAT setups.
It's the internal VPN IPs, it doesn't always do it, but sometimes it won't just come back, so a quick restart of zerotier-one or ping it and it starts working.
+1 for Tailscale, it will save you a lot of hassle. Setup is really quick, install and auth, that's it, little to no configuration is needed.
Its MagicDNS is magic! But I don't like the subdomain they provide, you'll get xxxxx.ts.net, although you can re-roll it. So I bought a domain for this purpose alone last month, now I can access my servers through servername.mydomain.com instead.
You can also override DNS configuration of all your servers, something that had been frustrating me recently because a DNS provider I used blocked/returned a wrong IP of a domain despite using an unfiltered option. Updating the DNS of all my servers wasn't hard, but it was annoying. So now I'm using NextDNS with override local DNS on on Tailscale, you can use your own pihole too, but I like anycasted DNS resolver. Because of this too I discovered a bug of one of installed apps that made over 300k DNS request in a day according to the log.
Yeah, I was running wireguard to 2 different VPSes without issues for months (or years, idr) but when I bumped the connection number to 20+ seems like it did a nasty thing on my router. And I am 99.95% sure it's a router issues because one of those VPS-tunnels that died was Oracle that I've was able to change (ephemeral) public IP "on the fly" - without rebooting/restarting anything and shit reconnected instantly.
So I am not blaming wg here (worked perfectly), but my router, but I am not really in position to change it ( y0 @crunchbits have any soho router/ap+switch in the giveaway? (-: ) as buying new one and putting it for the bridge mode to just get moar wg seems like an expensive thing to do.
Not heard of Nebula before but it sounds quite interesting.
If you're prepared to do a bit of manual configuration and/or automating config files, I'd second everyone's comments about wireguard. I do almost exactly this, and as you can make it properly point-to-point, you don't have to worry about any particular nodes being more highly available than the others. It also supports one-sided connections so if a few nodes are behind a NAT, they can still form the network if they initiate a connection.
As others have said too, @Neoon has stuff built on top of that that can help optimise your routes on top. But if you don't want to try it, you can achieve the most of the results using plain wireguard with point-to-point and manual routes for the ones that can't connect directly.
Yea, you can for sure setup p2p wireguard connections manually.
But depending on the size, its going to suck.
Plus, you have nothing that would make it a routable giant network.
The thing I tried to solve with wg-mesh.
Its not magic, just a wrapper around wireguard and bird.
So far I am using it on 30+ machines, most of the issues have been fixed.
Stability wise, you can't beat wireguard and bird, it just works.
Red flags everywhere. Reminder to make a date night. You're going to be like that @vitobotta dude.
Sounds like yeast infection medication.
TAILSCALE is exactly what you need.
You can even ping your home printer from this mesh network, because you can utilize one of your home computer as a "subnet router", to reach devices that cannot installed the VPN agent. It is based on wireguard. I tried Zerotier also, but Tailscale is my favorite.
@vitobotta is way out of my league. Have you see all the stuff he self-hosts, plus he's releasing his blogging platform project... @vitobotta is the hotness. I am pretty tepid.
Well I have a lot to read up on. I completely side-stepped Wireguard after seeing that mesh was in their TODO.
So what is bird doing here? I'm envisioning 20-ish systems on one global "LAN". If b00b::1 wants to talk to b00b::2 on this global VPN mesh, does it need a routing daemon?
That's why all the wg-mesh projects on github do exist.
You can use tinc if you don't care about throughput, which by design supports mesh.
If you care about throughput, use wireguard.
Also latency is a concern on tinc, every hop adds like 1.5ms.
Wireguard adds nearly nothing, sure if you measure it, you can, but its marginal.
The decentral part boils down to, every node has its own ID, hence its own Subnet.
To avoid any IP collisions since the Node only can give out ip addresses within its own range.
Except you give out the ID twice, which initially will be checked against the routing table by the other endpoint and it will refuse setting up a wireguard link.
Hence, you need to share with others, who has what subnet and what is reachable via which link, which is the shortest path etc.
So yes, it does need a routing daemon, but its auto configured in wg-mesh anyway.
Plus, you can modify it and add your own stuff.
IPv6 is not added yet, I added it with the route-bender but it was unstable for me, so I left it out for now.
@raindog308 I think this is a good start if you're looking for options https://tailscale.com/compare/
"SSL VPN" What?
Been using tinc for a while in switch mode, just did an iperf3 between two nodes and hit 250Mbit/s (max line speed) without any issues between a server hosted in Vultr and at home over tinc. I'll try to do some tests with faster connections as well.
Also, if I have two tinc hosts on a local lan, but they connect only to the Vultr server, the first few pings are 2-3ms, then after that they go down to 0.1-0.2ms when they have established a direct connection.
Also, tinc is great because I can use any IP-range, including public IPs, whereas tailscale is limited to 100.64.0.0/10.
So what about
What limits have you faced?
Sorry, when I recall correctly, it was OpenVPN, which adds roughly 1.5ms latency on every hop.
Tinc does provide a redundant link, but its not latency optimized at all.
If you just browse, that's likely fine, but anything latency critical, nah.
So I ended up dropping it.
Throughput wise it wasn't as terrible as OpenVPN however not as fast as wireguard.
The Subnet Range limitation sounds more like a software limitation.
Tailscale could for sure change that anytime if they really want.