New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How do you upgrade prod servers?
I have a few servers that runs production on vps, with some active daily users, that earns some money. Typically I use debian or ubuntu. Some were setup as far back 2017.
Sometimes, once in 5 years or so, I need to install something. But my OS is so old that the repo is not active anymore and I can't install what I need. Nice to have updated OS from time to time.
My question is, how do you safely upgrade your debian/ubuntu servers without breaking anything? The only solution I can think of right now, is setup a new server, and migrate everything there. Point DNS there. Then decommission the old server.
Comments
Hmm.. firstly you have to take backups, anytime!
Secondly, I'm updating the servers on a monthly basis. Testing updates are disabled. Reboot every 3-6 months.
You can do that to upgrade to a newer OS via ssh.
Create backups.
Packages installed will be upgraded to newer version.
Maybe you will run into compatibility issues, since setup since 2017. I am not sure though.
I have used this method to go from Debian 11 to 12, but I have never upgraded to skip so many versions.
The command should still work on Debian from back then though.
Also, you will need to add other repo, not the one that isn't working. Can be found on google.
Just run apt upgrade like a man, no backup.
Just move around over time
Brudda you can upgrade causing 20 sec of downtime two times a year it’s fine
New server, lots of testing and repointing DNS when it works is the usual approach for upgrading ancient systems.
Don't do "apt full-upgrade" because you've skipped too many versions and things will almost certainly break.
I would say you're on the right track. If you can do it then do a blue green deployment.
Give out notice / emails that there will be a maintenance at a specific time slot and stuff so that the users know.
jcaleb, it's been a while hadn't see you here
For me, some of my server still use Debian 6/9 (because some app would failed due to openssl version/cant work with the new one), so when the application needed an upgrade, I try to use newer OS, such Debian 11/12, try some config, rsync the code, then change the dns.
Mentally strong people run sudo apt full-upgrade -y, no backup and full homo.
You can use our script that checks if new updates/upgrades are available and auto-apply them.
It will abort the update/upgrade and send an email if potential errors are detected and/or if any package is being removed as result of the upgrade.
By default it is set to keep old configs (eg. /etc/apache2/*.conf; /etc/php/(apache2|cli)/php.ini ... etc.). But you can change this in the script.
https://github.com/ENGINYRING/Linux-Auto-Update-Script
Just set it to run once per day and it will keep your servers updated.
PS: Feel free to rate it on Github if you found this (or other scripts of ours) helpful.
Upgrading can be a bit of a dance, but your idea of spinning up a new server and migrating is a smart move, it gives you a clean slate.
You can have monthly scheduled maintenance window for updates. Like holiday or a non critical time based on your users timezone.
You need to spend time to modernize your setup. That's an incredible security risk.
if it works don't touch it lol
/jk
True!!
I did 3 push-ups today.
Or skip upgrade and go on a monthly trip to Thailand
IIRC, debian and ubuntu both keep their old repos online, you just may need to edit the URI in the source file. Though I would suggest setting up a new server because not only would it be cleaner, it will let you test things to see if anything breaks (you said it's been years so my guess is at least one thing will break), and with all the CVEs/Vulns over the years, who knows if there's a backdoor/rootkit/etc on the server.
P.S. Going forward, you really need to update more often. It doesn't take long to apply updates and to reboot a server. I'm not saying you need to do it daily but something like weekly, monthly, or even quarterly is better than not doing it at all. I highly doubt it's being used 24x7 or that criticial where you can't find a couple of minutes that will minimize impact to customers.
I did 20 last week was in pain for 2 days
I'm trying to reach 5 daily
Ehm,
Don't think about backup/snapshot; run
sudo apt full-upgrade -y && reboot; go to buy/make couple cups of coffee for preparing bunch of support-ticketsAt that point I'd just build new boxes and throw the existing ones away. Upgrading would be too risky and very unlikely work with that many versions in between.
Even less frequently if you just use LivePatch, KernelCare or something to keep the kernel patched and just restart upgraded services every now and then. Even less if everything is containerized
Who needs backups when server's got nine lives?
At this point, I would migrate the content to shiny server (if i'm sure the content is compatible)
That on its own won't dist upgrade any distributions that I know about.
just do what you are doing now. this is how i do.
+1 to this
i dont consider anything prod ready until it's been running untouched without an update for atleast 3 years
In our case, we have staging servers for every important production services that we run, which are basically replicates of the production servers, with smaller scale. We'd upgrade the staging server first, and troubleshoot any problems that occurred during the upgrade. Once we get everything sorted out, we then upgrade the real production server in the same way.
For less critical services, we do similar as what you did - just spin up another node every 3 - 4 years and migrate the data.
Thanks for all suggestions.
The thing is, although these servers makes money, it's not big enough money. So I really don't want to monitor it much and keep updated all the way. And it ain't broken, so I was lazy to keep things updated. Everything pretty stable and smooth for years.
I just needed to install new software, and saw some issues. I updated the sources.list a bit so I can install what I want. Then couple of commands later, I broke the entire OS. Can't start a lot of services, can't ssh (except the current connection I had), lots of commands not working.
And then since I am a champ of not monitoring the servers, I found out my backup system was not working for a very long time. Not synching to the remote backup server.
Luckily we subscribed to weekly backup and was able to restore.
I will definitely check this https://github.com/ENGINYRING/Linux-Auto-Update-Script
Will also check staging if budget permits
Hello @ErawanArifNugroho how you doing
https://wiki.debian.org/DontBreakDebian#Don.27t_make_a_FrankenDebian
I got confused haha, I was trying to update/upgrade, forgot it did not work, then installed a software that broke