Spectre and Meltdown - The what is my provider going to do about it? thread!

AuroraZ

Ignore it like everyone else. Then bitch when shit hits the fan. S.O.P.

Awmusic12635

We ended up patching yesterday. Overall it was a pretty smooth update.

No real noticeable difference in CPU usage.

ariq01

https://github.com/speed47/spectre-meltdown-checker

Have you check it?

LjL

@Shazan said:

@tarasis said:
Nothing from

• Online.net (have a dedicated there)

Being a dedicated server, it is up to you to update it or not. I don't think they will do anything about it.

Well, I am on Scaleway, not full-blown Online.net, but the Scaleway-specific issue does say "For customers using BareMetal servers, we will also need to apply the microcode on each server to secure from Spectre".

I also received two successive emails (on the same day) with subject "Scaleway - Emergency security update required on all hypervisors". In the second one they elaborated a bit on the presence or lack of patches for the various things, and referred me to the above issue.

Interesting that Online's "poor man's subsidiary" sends out this information to users but the main provider would tell them nothing...

Maounique

Awmusic12635 said: No real noticeable difference in CPU usage.

Same here, where we expected the biggest hit in OVZ and we patched first due to the ease of exploitation, the variation is within the +/- 5% margin which happen everyday, especially mowing from weekend to weekday and from holidays to workdays.

TL;DR no difference.

nullnothere

@Awmusic12635 said: No real noticeable difference in CPU usage.

@Maounique said: TL;DR no difference

The difference for "normal" work loads shouldn't be much/noticeable. It is mainly in the syscall heavy workloads that the performance will start to really dip.

I think the average (non-idling set) VPS per host node (i.e. aggregated at the host node) doesn't too much by way of syscalls and even if we assume ~10-20% of load is syscall type of stuff, you're going to see (worst case?) 30% of that load degrade - so a worst case load increase from ~10% to ~15% which should imply that on average you shouldn't see more than a 1-5% change in load (for a good well balanced host node).

lurch

Netcup sent me an email on the 4th saying they were sorting it. My VM was rebooted 3 days ago.

Mr_Tom

I didn't recieve the email from ZX about it, but my VM was down this morning for patching. Down around 25 minutes in total.

That's 2 hosts done. Not heard from DO or Vultr though - thought they might have sent something out, even if they updated without any downtime.

Edit: just read a blog post from DO who say their KVM setup isn't vulnerable, or that they're waiting to fully test the patches first.

doghouch

OVH just did it for virtual servers:

sureiam

All my providers have had reboots this past week. However one in particular didn't email a warning or restart the VPS automatically after the reboot. Took me 2 days to realize that. It's not a critical vps so whatever but still annoying.

Oseri

Don't see it mentioned, but last Sunday @LaunchVPS applied related patches to their servers.

tarasis

Emails from both VirMach and Hosthatch yesterday. Snippets from both emails

VirMach

We will begin emergency kernel updates & reboots on all VirMach services, starting Thursday, January 11th, at 2PM Eastern US time. We apologize for the short notice, and hope you understand. VirMach usually uses rebootless kernel update services, but in this specific case we will need to manually apply the patch. If the rebootless patch becomes available from our provider, we will attempt to avoid a reboot. Please note that this is not an issue with VirMach servers, but is a problem with all Intel processors. While required maintenance does not usually qualify for uptime credits, we will be crediting all VPS customers for (1) day of service. We thank you for your continued support.

HostHatch

As you may be aware of the recently reported CPU vulnerabilities, namely Spectre and Meltdown, we will be performing reboots in order to patch the host nodes. We were waiting for Intel to release the microcode so we could patch all 3 variants of the vulnerabilities instead of 2 that are patched by the kernel at this time. According to our internel tests, all CPUs that we use on our Storage VPS series have the new Microcode available. If you have any other services with us (NVMe SSD or SSD), you will receive another email for their maintenance schedule once we have verified that their Microcodes are available.

Maounique

tarasis said: We were waiting for Intel to release the microcode so we could patch all 3 variants of the vulnerabilities instead of 2 that are patched by the kernel at this time. According to our internel tests, all CPUs that we use on our Storage VPS series have the new Microcode available.

yeah, we thought should patch OVZ immediately, so another round of reboots is in the cards. The new round will take all servers with it, but not all microcodes are available here yet.

Mr_Tom

tarasis said: HostHatch

Yeah, also had the same email from them. Just DO/Vultr left for me. They've put out announcements with some details but no official date (that I've seen).

tarasis

@Maounique said:
yeah, we thought should patch OVZ immediately, so another round of reboots is in the cards. The new round will take all servers with it, but not all microcodes are available here yet.

Understandable & appreciated.

jetchirag

Can someone confirm if there's a bug with

https://github.com/speed47/spectre-meltdown-checker

It shows it's vulnerable by Spectre Variant 2 whereas it's already patched:

- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, 
CVE-2017-5754) 
[root@server src]# 

eva2000

@jetchirag said:
Can someone confirm if there's a bug with

https://github.com/speed47/spectre-meltdown-checker

It shows it's vulnerable by Spectre Variant 2 whereas it's already patched:

> - Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, 
> CVE-2017-5754) 
> [root@server src]# 
> 

Variant 2 usually needs microcode updates. So you've updated microcode ?

Zerpy

@eva2000 said:

@jetchirag said:
Can someone confirm if there's a bug with

https://github.com/speed47/spectre-meltdown-checker

It shows it's vulnerable by Spectre Variant 2 whereas it's already patched:

> > - Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, 
> > CVE-2017-5754) 
> > [root@server src]# 
> > 

Variant 2 usually needs microcode updates. So you've updated microcode ?

I also wouldn't trust the https://github.com/speed47/spectre-meltdown-checker in regards to detect if IBRS is supported in microcode or not, because despite it says IBRS is supported both by microcode and kernel - if trying to actually enable IBRS during runtime, it will give the message "write error: No such device":

# echo 2 > /sys/kernel/debug/x86/ibrs_enabled
-bash: echo: write error: No such device

All these testing tools - I think we should take them with a grain of salt sometimes

eva2000

Zerpy said: because despite it says IBRS is supported both by microcode and kernel - if trying to actually enable IBRS during runtime, it will give the message "write error: No such device":

on cpus with actual microcode upate or without or both ?

hostdare

@eva2000 are the intel updates need to be run individually for all cpus (for vulnerability 3 I think ) ? I have many cpu versions

eva2000

@hostdare said:
@eva2000 are the intel updates need to be run individually for all cpus (for vulnerability 3 I think ) ? I have many cpu versions

list of cpus affected on intel side https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr

Zerpy

@eva2000 said:

Zerpy said: because despite it says IBRS is supported both by microcode and kernel - if trying to actually enable IBRS during runtime, it will give the message "write error: No such device":

on cpus with actual microcode upate or without or both ?

With CPUs with the actual microcode update.

So.. According to the https://github.com/speed47/spectre-meltdown-checker, it says "Hardware (CPU microcode) support for mitigation): YES"
"Kernel support for IBRS: YES"

But echo 2 > /sys/kernel/debug/x86/ibrs_enabled will fail, so either CloudLinux and CentOS made a kernel release that brings support for IBRS but does not allow enabling in any way (even during boot), or the script detects wrongly that the microcode is there.

Now, the microcode is from the initial 20171117 release from Intel - which I believe shouldn't contain any IBRS support (but might be wrong), yet the script detects it as being there.

Reality is hardware support + kernel support, according to script, but no way to enable it in /sys

WHT

So still no patch for centos7?

iwaswrongonce

We have a ton of OVH instances and none have been restarted...

jackb

@WHT said:
So still no patch for centos7?

It was released over a week ago?