New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
If you have more than 5 VPSes, how do you manage them?
Recently there was a discussion in community on how many VPS servers do they have.
It's like the median for LET community is 10! (which made me feel like this).
But, I was wondering, how do you even manage this many instances?! I think after 10, you'd need a SAAS or DB for mere remembering the IPs!
So you just put the IP and user in ~/.ssh/config and call it a day?
How do you manage secrets and ssh keys? (using third-party services like 1pass?)
Do you, like, set aside one VPS as a monitoring instance for others? (grafana,...)
Thanked by 1greentea
Comments
Termius is a very simple SSH client that can serve as a nice little dashboard.
Hetrix Tools is free for a few server monitors.
I will admit though, I only manage 2 VPS personally.
4 more at work, but I have removed them from my brain because I just resigned.
termius, ansible, prometheus, grafana, zabbix.
Notepad
Winscp, uptimerobot, hetrixtool
some book keeping
https://github.com/cp6/my-idlers
The honest answer.
Yes.
I have same SSH private key on every local computer.
Each VPS has a folder of text files that contain its passwords and setup notes.
These folders are stored in Seafile, installed on one VPS.
No, I only use UptimeRobot.
dns/ansible
ssh keys are on Yubikeys (PIV), app/service secrets are in some kind of vault (Hashicorp or Azure)
one vps has Uptime-Kuma/healthchecks.io for service/jobs monitoring, telegraf/influxdb/grafana for server monitoring
provisioning and updates mostly through ansible
I add all my VPSes to the MTPuTTY + this thing support scripts, so I don't need to type every command manually everytime.
MTPuTTY (Multi-Tabbed PuTTY) is a small utility enabling you to wrap unlimited number of PuTTY applications in one tabbed GUI interface.
wow, nice! what's the credentials for the demo site?
DNS.
My servers are only accessible from a certain jumphost. This jumphost is heavily locked down and requires 2FA and VPN. From there its ssh keys, which are distributed and revoked with a semi-automatic system of scripts.
Coming from BSD I do not allow root login via ssh, so the root password is still typed in manually. Every vps has a unique root password that can be calculated based on the hostname if you just know how.
Yes. Or actually, several vps's. The bundles of nat vps's from Webhorizon or MrVM are excellent for monitoring. I use Icinga, Grafana and Prometheus together with a bunch of shell and python scripts to gather metrics, and I can actually look at almost any metric from whatever part of the world I want.
keyboard
open telnet because I keep forgetting my passwords.
Most interesting solution so far. So your single point of failure is that VPN.
Don't you find the 2FA being too much of a hassle? Unless you work in cybersecurity field...
My main stack is;
I have been using hostnames like [distro][mainusage][location].domain.tld and linked to the IPv6 (or IPv4 in the very few cases where not available). For example, debtortx for debian tor texas or debtorssdfr for debian tor on the ssd dedi in france.
I am not usually installing monitoring tools unless I have a reason for it.
And then you
ssh debtorx.domain.tldfor example? How does it resolve to your server IP?Yes.
What do you mean? With standard DNSs, of course. I have AAAA records and I am using that method with dynamic IPv6 at my houses too through freedns.afraid.org
No need to bypass NAT or anything.
Keys plural?, just wondering what your use case is for having multiple SSH keys rather than using one everywhere. I can sort of imagine a tenuous argument for that if they were file based keys, but juggling or carrying multiple hardware tokens would seem a needless hassle for no extra security.
With my penis
The VPN has multiple entrypoints and the jumphost is located in an ip range that I have full control over, so replacing it with a new vm and give it the same ip would not be to much hassle. The ip range in question is spread over multiple dc's and have BGP with several major providers, so you would basically have to take out my part of the world to stop me from using it. But yes, it is kind of a single point of failure. But then again, this is just my personal playground so if I cant access it, I just play with something else.
The 2FA doesn't really bother me, I usually authenticate once in the morning and then my session is open for the rest of the day so it's not really a problem. Setting up sshd to use 2FA is a 2 minute job so I have zero sympathy for people that get hacked when they allow root login from anywhere and use a simple password.
And while security is not exactly what my title says right now, it is a very big part of everything I do. As it should be for everyone in IT.
Actually, my main concern security wise is usually the control panels of the providers. Every provider out there provides some kind of console access to your vm's accessible via their webportal, but few provides a way to really secure access to that webinterface. It does not really matter how much I harden and lock down access to my vm's if an attacker still can get console access via a webpage protected with a simple password. It kind of bugs me that this is not more of an issue in the hosting business.
A bit off topic for this thread, but I seriously think this is something that should be discussed.
MobaXterm with folders for organizing all of the VPS's. MobaXTerm also has a "Mutli-Exec" feature for executing mass commands all at once, but using something proper like Ansible is still way better.
UptimeRobot for notifications/alerts, it's been really solid and is very quick to report any issues.
Smokeping on a few of the instances that are from seperate providers/data centers then the rest. Helps in tracking packet loss / latency over time.
A "IP Mapping" text file with entries like
buyvm-ma-1
FQDN: ma-1.ft.mydomain.tld
Location: Miami
IP: Unprotected IP
Private IP: Private Wireguard IP
For stuff I actually care about, I have a Wireguard mesh with one node running Grafana + VictoriaMetrics (Prometheus but much more efficient in all ways, with unlimited retention). Node-exporter and other popular exporters have Ansible roles that make them really easy to deploy.
For actually deploying/setting stuff up, I have a mix of Ansible playbooks and text files with scripts to run in them (lol). Not all of the providers I use support cloud init or some sort of automated setup, so I just stick to the latest Debian and run the same Start up script to set everything up.
I'm using multiple keys (1 ssh key/yubikey, multiple yubikeys) as backup/in case they get lost or stolen. Should that happen, I just remove the lost one from authorized_keys and keep using the other ones.
Anytime I get a VPS, I put on Debian 11 and then install Proxmox and add it to my PVE cluster.
Then management is a piece of cake, all in one browser window.
Even if someone is at that console, need to provide password to access actual system. Unless you have not left it logged as I have seen quite a few times by other admins or password is too simple, no one can help. Further you can password protect bios if you think someone can reset your password but it will be hassle to reboot server for simple task.
Dafuq are you doing here? This is LET, people still use 256 MB or even 128 MB VMs, how dare you!
Uhm, yes, but preventing login with just a password is the whole idea behind jumphosts, VPN and 2FA. We kind of left the whole a password is enough way of thinking like 15 years ago.
Hmm... I am dangerous enough to try proxmox on a 512MB server. I'll post after I try that out one of these days.
in 512 MB I ran 2008 server with an 128 mb debian vm in virtualbox. That is beefy machine, bro!
I have a bunch of CMD scripts that just ssh's into the VPS.