How often do you reboot because of kernel security patches? (both providers & customers)

crunchbits · August 8

@Mainfrezzer said:

@crunchbits said:

Because the time to "live migrate" 10-30 VMs, hope you didn't blow a VM/data up, and then reboot is insanely complicated versus a 1-3m reboot. It's also very high risk. If you need something better, as a customer, pay for it. Otherwise deal with a host rebooting a machine for critical CVE's.

Prior to AI-assists, CVEs like this that affect such a wide breadth of deployments were incredibly rare. I can think of maybe 1-2 in ~6 years?

@HostBilby said:

Thanks for the back up Dad, love you 😘

How dare you 2, that you 2 would hypothetically inconvenience me for 3 minutes?! OUTRAGEOUS

Just trying to convince wallets to open up, lambo insurance renewal is higher than expected this year

dedigod · August 8

@crunchbits said:

@dedigod said:

@HostBilby said:

[@dedigod said]

So you take clients offline for all patches? How often are you bringing clients VMs offline?

If you don't have enough capacity to migrate for patches what do you do when there's a hardware failure like main board on a server?

No. Routine maintenance and updates don’t generally require a reboot. So they happen seamlessly to the client.

We have cold spares ready to swap in. Like most providers.

Like I said, there aren’t many providers with true HOT capacity ready to migrate an entire node of clients just for maintenance. Even the large cloud providers are the same.

If you need that level of uptime, you’re in a different snack bracket cost wise, or with a provider large enough to justify the resource sharing - shoutout @crunchbits (Synteq HPC’s new cloud, it’s awesome!)

why cant you turn on a cold spare, patch it then live migrate to them then keep doing the other servers and make the last one a spare? all enterprise systems do this and auto turn on based on capacity

Because the time to "live migrate" 10-30 VMs, hope you didn't blow a VM/data up, and then reboot is insanely complicated versus a 1-3m reboot. It's also very high risk. If you need something better, as a customer, pay for it. Otherwise deal with a host rebooting a machine for critical CVE's.

Prior to AI-assists, CVEs like this that affect such a wide breadth of deployments were incredibly rare. I can think of maybe 1-2 in ~6 years?

So you can't do it either? What hypervisor are you using and do you have shared storage?

It isn't complicated at all its automated and very normal to load balance and live migrate. There's no risk for live migration. Explain the risk here.

Are you saying you've only rebooted your servers twice in past 6 years?

drivex · August 8

@dedigod said:

@crunchbits said:

@dedigod said:

@HostBilby said:

[@dedigod said]

So you take clients offline for all patches? How often are you bringing clients VMs offline?

If you don't have enough capacity to migrate for patches what do you do when there's a hardware failure like main board on a server?

No. Routine maintenance and updates don’t generally require a reboot. So they happen seamlessly to the client.

We have cold spares ready to swap in. Like most providers.

Like I said, there aren’t many providers with true HOT capacity ready to migrate an entire node of clients just for maintenance. Even the large cloud providers are the same.

If you need that level of uptime, you’re in a different snack bracket cost wise, or with a provider large enough to justify the resource sharing - shoutout @crunchbits (Synteq HPC’s new cloud, it’s awesome!)

why cant you turn on a cold spare, patch it then live migrate to them then keep doing the other servers and make the last one a spare? all enterprise systems do this and auto turn on based on capacity

Because the time to "live migrate" 10-30 VMs, hope you didn't blow a VM/data up, and then reboot is insanely complicated versus a 1-3m reboot. It's also very high risk. If you need something better, as a customer, pay for it. Otherwise deal with a host rebooting a machine for critical CVE's.

Prior to AI-assists, CVEs like this that affect such a wide breadth of deployments were incredibly rare. I can think of maybe 1-2 in ~6 years?

So you can't do it either? What hypervisor are you using and do you have shared storage?

It isn't complicated at all its automated and very normal to load balance and live migrate. There's no risk for live migration. Explain the risk here.

Are you saying you've only rebooted your servers twice in past 6 years?

When you want HA with Shared Storage (CEPH or Flash-Storage) you have to pay.

HostBilby · August 8

@crunchbits said:

@Mainfrezzer said:

@crunchbits said:

Because the time to "live migrate" 10-30 VMs, hope you didn't blow a VM/data up, and then reboot is insanely complicated versus a 1-3m reboot. It's also very high risk. If you need something better, as a customer, pay for it. Otherwise deal with a host rebooting a machine for critical CVE's.

Prior to AI-assists, CVEs like this that affect such a wide breadth of deployments were incredibly rare. I can think of maybe 1-2 in ~6 years?

@HostBilby said:

Thanks for the back up Dad, love you 😘

How dare you 2, that you 2 would hypothetically inconvenience me for 3 minutes?! OUTRAGEOUS

Just trying to convince wallets to open up, lambo insurance renewal is higher than expected this year

Don’t forget gas prices!! It’s getting tough to survive on -1% margin…

OhJohn · August 8

So back in 2024 I probably had to do like 3 to 6 reboots per machine per year.

Now I had one per machine yesterday, the last one per machine before 7 days ago.

So now I'm doing an extra weekend shift deploying an automated reboot routine for each machine to be done with that sh*t in the future.

AlteredParadox · August 8

annnnd @dedigod is gone!

luckypenguin · August 8

This question is vague for so many reasons. I reboot when I need the kernel patches
to be applied based on severity, if I run a few low-end instances that do monitoring,
I don't care about local privilege escalation, etc. So it is such a bad survey like "what
time do you go to sleep?" When I feel I need to.

crunchbits · August 8

@dedigod said:

@crunchbits said:

@dedigod said:

@HostBilby said:

[@dedigod said]

So you take clients offline for all patches? How often are you bringing clients VMs offline?

If you don't have enough capacity to migrate for patches what do you do when there's a hardware failure like main board on a server?

No. Routine maintenance and updates don’t generally require a reboot. So they happen seamlessly to the client.

We have cold spares ready to swap in. Like most providers.

Like I said, there aren’t many providers with true HOT capacity ready to migrate an entire node of clients just for maintenance. Even the large cloud providers are the same.

If you need that level of uptime, you’re in a different snack bracket cost wise, or with a provider large enough to justify the resource sharing - shoutout @crunchbits (Synteq HPC’s new cloud, it’s awesome!)

why cant you turn on a cold spare, patch it then live migrate to them then keep doing the other servers and make the last one a spare? all enterprise systems do this and auto turn on based on capacity

Because the time to "live migrate" 10-30 VMs, hope you didn't blow a VM/data up, and then reboot is insanely complicated versus a 1-3m reboot. It's also very high risk. If you need something better, as a customer, pay for it. Otherwise deal with a host rebooting a machine for critical CVE's.

Prior to AI-assists, CVEs like this that affect such a wide breadth of deployments were incredibly rare. I can think of maybe 1-2 in ~6 years?

So you can't do it either? What hypervisor are you using and do you have shared storage?

It isn't complicated at all its automated and very normal to load balance and live migrate. There's no risk for live migration. Explain the risk here.

Are you saying you've only rebooted your servers twice in past 6 years?

No, our new platform is HA for VPS. It's just not in the same price category as non-HA. Need to brush up on that comprehension and stop attempting to straw man everyone's statements.

Edit: RIP, looks like already gone.

rpqu · August 8

I think the questions that emerged from this thread is supposed to be pre-sales question.
I’d rather this were a joke, to expect lowend hosts on paper thin margin performing live migration. That's unrealistic demand to host with technical challenges that went unsolved for months after users reported it

buggedout · August 8

@AlteredParadox said:
annnnd @dedigod is gone!

Nicee.... I was feeling nauseatic !!

dbadude · August 8

The small vps-es reboot every few weeks. The big dedis every few months.

host_c · August 8

@miniopt

" How often do you reboot because of kernel security patches? (both providers & customers) "

According to some customers, too often....

If this drags on too much, heck, I will switch back to Hyper-V, at this point, not even MSFT has weekly almost daily patches on Windows Server Editions

TimboJones · August 9

A common problem is that shared libraries get updated and the services are not restarted because there's no mechanism to detect if a reboot is needed, so many people thought for years Windows was garbage for forcing unnecessary reboots to ensure running files get updated and Linux was intelligent when it just lacked any such capability.

forest · August 9

For my servers, whenever my system wants to:

root@forest-pfcloud-1-si:~# cat /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,codename=${distro_codename},label=Debian";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
        "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
        "site=deb.torproject.org";
        "site=repo.i2pd.xyz";
        "site=apt.syncthing.net";
};

For my home computer, whenever there's a new vanilla kernel release.

tentor · August 9

@Frobsy said:
I’m rich. I use kernelcare.

I wish... cries in daily reboot

FYI some fixes are hard to be implemented as live patches, so it is not really a silver bullet either. Reboot is the king in the end.

tentor · August 9

@forest said:
For my servers, whenever my system wants to:

root@forest-pfcloud-1-si:~# cat /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,codename=${distro_codename},label=Debian";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
        "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
        "site=deb.torproject.org";
        "site=repo.i2pd.xyz";
        "site=apt.syncthing.net";
};

For my home computer, whenever there's a new vanilla kernel release.

label=Debian? Did not you face this issue?

forest · August 9

@tentor said: label=Debian? Did not you face this issue?

It only does automatic upgrades for security releases.

Oh never mind I have it set to do nearly all upgrades. Either way, I haven't run into trouble.

iamnobody · August 9

never

Frobsy · August 10

@tentor said:

@Frobsy said:
I’m rich. I use kernelcare.

I wish... cries in daily reboot

FYI some fixes are hard to be implemented as live patches, so it is not really a silver bullet either. Reboot is the king in the end.

Thank you. Gave me more reasons not to pay for it 👌

MacawHost · August 10

With KernelCare on the node, most security-related kernel patches shouldn’t require a reboot. We use KernelCare on our nodes for this reason. Of course, there are still cases where a reboot is needed, but rebooting for every kernel security update seems unnecessary.

Socheat · August 10

I don't reboot myself, I let unattended-upgrade handles everything including kernel upgrades, and reboot. Because I had nothing critical on there, the system can reboot whenever it wants.

slowservers · August 10

I haven't had to do it yet as a provider. There haven't been any interesting VMM-level vulnerabilities that I've seen.

I have updated my own VPSs, of course, for other vulnerabilities/fixes.

PuDLeZ · August 10

I usually configure my things in a "set and forget" where it will patch daily and reboot if it's required. Sure, depending on the OS, it could be live patched but if something was that critical to me, I'd at least have two nodes so I can provide HA myself... As someone who joined a company where everything was manual, nothing HA, and patching was always a headache to get change tickets/etc (not to mention, many systems and software was really outdate and some way past eos/eol), I much prefer to stay up to date with the set and forget. Plus, that method enabled me to go on a trip/camping/etc. and disconnect without worrying about anything. Unless you're running some weird software that requires very specific versions of libraries/dependencies or very bleeding edge versions of everything, the risk of having some issue from patching is much lower than getting hacked...

For vendors, I don't expect them to live migrate/etc, especially with LET/LEB pricing. Outside of me requesting it, moving out of dc, or a hardware failure that requires extended down time (ex: ordering a part that will take a while to be delivered), I actually don't want them to migrate me to a different node. I've learned what to expect being on that node, I don't want to get moved to something unknown or have others moved to my node for what will be a very short outage if they just patch/reboot. Only thing I want from the providers is an email saying "urgent patch/maintenance required, we're starting at this time, expected to last x, and you can check status at y." As for how fast I expect them to apply them, it all depends on the vuln as many of the ones we see don't actually apply to them/the hypervisor level. For the ones that do apply, I'm pretty sure the vendor knows about it before I do and wants to get it applied so they don't dirty their name (especially the established ones).

vinhais · August 10

The biggest problem would be KernelCare no longer supporting PVE.

Howdy, Stranger!

Categories

In this Discussion

How often do you reboot because of kernel security patches? (both providers & customers)

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

How often do you reboot because of kernel security patches? (both providers & customers)

Comments