Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Listen to your VPS Neighbours on intel Servers - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Listen to your VPS Neighbours on intel Servers

2

Comments

  • AnthonySmithAnthonySmith Member, Patron Provider

    psb777 said: you have to actively "listen" (sample) for very long and filter out lots of noises to have anything remotely interesting.

    Yep, but being able write a quick python script to filter and log the interesting noise is childs play, the danger is very real imo.

  • jsgjsg Member, Resident Benchmarker

    First, just because it's so funny: One of the new attacks on intel CPUs (Ridl) has become possible thanks to Meltdown mitigation.

    "You don't like copperheads? No problem, you just got a rattlesnake instead"

    Now, more seriously. Keep in mind that many providers sell (fraction of) threads as vCore. Will they give up 50% of their income? I don't hold my breath, particularly at the low end segment.

    "It's about time to switch to a non intel CPU" - Uhm, yes AMD does some things differently. But do they also do it better? My personal take is that intel is still such a dominant force that it's simply a more promising and rewarding research target. The absence of horror news about AMD vulnerabilites does however almost certainly not equate to AMD CPUs being really safer ...

    "It's about time to switch to a non x86 CPU" - Good luck! To which one if I may ask? Risc-V? Nope, too immature and I'm not even talking about main boards being available and at least a bit tested. Rasberry Pi etc? You are joking, right? Arm server processors? For one: what exactly is your assumption based that Arm does not have significant flaws? Plus - like for virtually all alternatives to x86: You also need software. And dev. tools. And swaths of experienced developers, and ... Similar for Mips, Sparc, Power (which is all but dead anyway).

    And then there is psychology and reality (read: pragmatism and facts like limited incomes).
    Hence, my take is that after some excitation someone will come up with some mitigation for at least some of the vulnerabilities of the month ... and almost everyone will just go on with their lives ... and their VPSs. Which actually is quite OK if you don't run sensitive stuff (like e.g. e-commerce) in which case you should switch to a dedi.

  • AnthonySmithAnthonySmith Member, Patron Provider

    @jsg yep thats all a fair opinion, it does seem the the 'immediate' best option is AMD though while appreciating that may not be a long term solution.

    given that we are now losing near 50% thread capacity in this environment on top of the previous "fix" causing about a 10 - 20% drop in performance, I cant see any other sensible option than going for AMD in the immediate term.

  • @AnthonySmith said:
    @jsg yep thats all a fair opinion, it does seem the the 'immediate' best option is AMD though while appreciating that may not be a long term solution.

    given that we are now losing near 50% thread capacity in this environment on top of the previous "fix" causing about a 10 - 20% drop in performance, I cant see any other sensible option than going for AMD in the immediate term.

    So there is no software fix for this? What is the purpose of the patches then?

    In case it IS true that vps hosting providers have to disable HyperThreading in order to prevent this, I am looking forward to confirmations of all providers on here, that they have done so.....

  • jsgjsg Member, Resident Benchmarker
    edited May 2019

    @AnthonySmith said:
    @jsg yep thats all a fair opinion, it does seem the the 'immediate' best option is AMD though while appreciating that may not be a long term solution.

    given that we are now losing near 50% thread capacity in this environment on top of the previous "fix" causing about a 10 - 20% drop in performance, I cant see any other sensible option than going for AMD in the immediate term.

    Yes. For a provider that seems to be a sensible solution, at least for some time. Besides, Epyc offers way more bang for the buck and some potentially interesting features due to having plenty of PCIe lanes.

    But that's new nodes. What about all those existing intel Xeon nodes?

    And there's something else, too: Based on what I've seen so far it seems very likely that more and more vulnerabilities will surface. Reason: The Spectre/Meltdown researchers opened a until then largely ignored can ... and now that that can has been opened there is more to come, much more and the current news is highly likely but the beginning of a news stream.

    Reason: Processors (plus context, e.g. AMT) are very complicated and very rich security research targets - and way more multi-layered and deeper rabbit holes than most could even imagine. A modern intel (or AMD) processor has very little in common with what we learned at university. It's actually more of a "computer with subcomputers within a processor".

    And YES!, of course virtually every manufacturer has f_cked up big time many times. In part due to a "features! features! features!" policiy and in part innocently because they were basically forced ("compatibility").

    I'll close with a personal remark: If I were a provider I'd look for ways to put a couple of small form factor Atom boards (think "NUCs") on a (cheap to get built) 2 HU metal frame along with a customized redundant power supply (less simple to do but still reasonably feasible), a simple drive cage, and probably a switch board, too.

    I'd end up with, say, 8 to 16 2 or 4 core Atoms or similar in a 2 HU slot, each with a (small) NVMe and 2 or more drives (optionally). Then I'd sell those as small dedis and I'd have the additional advantage of selling 8 to 16 small dedis within the power envelope of a single (current Xeon) 2 HU server (which is a very nice cost saver in hosting).

    Why? Because going dedi is about the only sensible thing to do (as a customer) in the current (and evolving!) situation. But dedis are way tio expensive for the VPS crowd, even the higher end one (paying like 25$ or 30$/mo for a VPS). Et voila: I would have a product for those and would collect them.

    Thanked by 1uptime
  • jsgjsg Member, Resident Benchmarker

    @packetnext said:
    So there is no software fix for this? What is the purpose of the patches then?

    In case it IS true that vps hosting providers have to disable HyperThreading in order to prevent this, I am looking forward to confirmations of all providers on here, that they have done so.....

    One - particularly on the software side - can only go so far into a processors internals. The "mitigations" now on offer are more emergency band aids than real mitigation. That can only come from processor people - and even they are limited by diverse factors.

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited May 2019

    jsg said: But that's new nodes. What about all those existing intel Xeon nodes?

    They get dropped and customers migrated to a safer space.

    I will probably keep the dual E5 nodes because halving the threads on a platform that rarely even touches 40% use to begin with is not the end of the world.

    E3's on the other hand, just became pretty pointless.

    As for the rest of your post... yeah maybe.

  • This is what DigitalOcean is saying about it (https://blog.digitalocean.com/may-2019-intel-vulnerability/)

    • We have received updated microcode from Intel and developed a set of kernel updates to mitigate the vulnerability, and we are rapidly rolling out these mitigations with no downtime to our users.

    • We also recommend taking steps to ensure your Droplet is up to date and secure. (https://www.digitalocean.com/docs/droplets/how-to/kernel/upgrade)

    The questions that arise in me;

    • Are these updates from Intel available for ALL platforms including vps providers running old hardware?

    • Are these providers actually going to receive and install these updates?

    Thanked by 1mrTom
  • angstromangstrom Moderator

    @packetnext said: This is what DigitalOcean is saying about it (https://blog.digitalocean.com/may-2019-intel-vulnerability/)

    DO's statement is pretty vague. For example, will they turn off hyperthreading? I wonder.

  • NeoonNeoon Community Contributor, Veteran

    Well some CPU's are already EOL and likely not get patched like the last flaw.

    Thanked by 1netomx
  • @Neoon said:
    Well some CPU's are already EOL and likely not get patched like the last flaw.

    cough 50 cent romanian vps cough

  • jackbjackb Member, Host Rep

    @packetnext said:
    This is what DigitalOcean is saying about it (https://blog.digitalocean.com/may-2019-intel-vulnerability/)

    • We have received updated microcode from Intel and developed a set of kernel updates to mitigate the vulnerability, and we are rapidly rolling out these mitigations with no downtime to our users.

    • We also recommend taking steps to ensure your Droplet is up to date and secure. (https://www.digitalocean.com/docs/droplets/how-to/kernel/upgrade)

    The questions that arise in me;

    • Are these updates from Intel available for ALL platforms including vps providers running old hardware?

    • Are these providers actually going to receive and install these updates?

    https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf

    Most people will be waiting for centos to release kernel updates and microcode_ctl updates which will contain this.

  • NeoonNeoon Community Contributor, Veteran
    edited May 2019

    @packetnext said:

    @Neoon said:
    Well some CPU's are already EOL and likely not get patched like the last flaw.

    cough 50 cent romanian vps cough

    Its OVZ, meh.

    Thanked by 1pike
  • jsgjsg Member, Resident Benchmarker
    edited May 2019

    @packetnext said:
    ... updated microcode from Intel

    I wouldn't bet more than $7 on that. Two main reasons: (a) intel is shooting too quickly. I understand their desperation but chances are low that a hurried microcode patch really mitigates. Chances are high, however, to introduce new problems.
    And no, that's not me wildly speculating. (At least) one of the current vulnerabilites has its very base in intels Meltdown/Spectre "fix". (b) you can mitigate problems rooted in a processors hardware only so far with microcode. And btw, that "fix" won't come free (in terms of performance).

    Keeping ones system updated is certainly a good policy. I doubt, however, that updating a VM's kernel can mitigate a vulnerability in the hosts processor.

  • NeoonNeoon Community Contributor, Veteran

    @jackb said:

    @packetnext said:
    This is what DigitalOcean is saying about it (https://blog.digitalocean.com/may-2019-intel-vulnerability/)

    • We have received updated microcode from Intel and developed a set of kernel updates to mitigate the vulnerability, and we are rapidly rolling out these mitigations with no downtime to our users.

    • We also recommend taking steps to ensure your Droplet is up to date and secure. (https://www.digitalocean.com/docs/droplets/how-to/kernel/upgrade)

    The questions that arise in me;

    • Are these updates from Intel available for ALL platforms including vps providers running old hardware?

    • Are these providers actually going to receive and install these updates?

    https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf

    Most people will be waiting for centos to release kernel updates and microcode_ctl updates which will contain this.

    What about the CPU's that are not listed there? schrödinger?

  • corbpiecorbpie Member

    Another reason why One Provider (dedi's) is the new future of hosting 😂

  • angstromangstrom Moderator

    For Ubuntu users, this is an informative page:

    https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS

  • psb777psb777 Member
    edited May 2019

    @AnthonySmith said:

    psb777 said: you have to actively "listen" (sample) for very long and filter out lots of noises to have anything remotely interesting.

    Yep, but being able write a quick python script to filter and log the interesting noise is childs play, the danger is very real imo.

    Afaik it's more nuanced than that. It's not like filtering leaked buffers from OpenSSL Heartbleed. In the ZombieLoad paper, they used lots of tricks because it can only smuggle 1 byte a time, and they had a very carefully designed filter inside the transient domain.

    As for RIDL, it took 24 hours to steal 16 bytes from the passwd file from a neighbor VM. I think that's not very practical because no hosting provider will allow you to abuse the CPU (which you share with your neighbor; if the VPS has any kind of dedicated CPU that's probably a physical core dedicated to you already) for 24 hours while you constantly brute forcing via SSH (in order to keep passwd loaded in cache).

    Thanked by 2angstrom uptime
  • NeoonNeoon Community Contributor, Veteran

    @psb777 said:
    As for RIDL, it took 24 hours to steal 16 bytes from the passwd file from a neighbor VM. I think that's not very practical because no hosting provider will allow you to abuse the CPU (which you share with your neighbor; if the VPS has any kind of dedicated CPU that's probably a physical core dedicated to you already) for 24 hours while you constantly brute forcing via SSH (in order to keep passwd loaded in cache).

    Now I do understand, why Netcup sells dedicated cores with no "mining allowed".
    They already saw it comming.

  • LeviLevi Member

    AMD stock is rissing like watter in the nile.

  • jsgjsg Member, Resident Benchmarker

    @psb777

    Careful! What we see now is but first proofs of concept. Also keep in mind that an attack going right at the jugular (the SSH password) shouldn't be compared to more general attacks.

    My professional understanding (I'm in IT sec dev) is certainly not that the heaven falls on our heads - but it's neither "no big thing. Just carry on".

    Thanked by 1psb777
  • NeoonNeoon Community Contributor, Veteran

    @LTniger said:
    AMD stock is rissing like watter in the nile.

    Intel stock has just dropped a single bit, like a few $, sad.
    Time for AMD to RYZE up.

  • datanoisedatanoise Member
    edited May 2019

    packetnext said: cough 50 cent romanian vps cough

    That's the problem with going super cheap. Sometimes it's better to spend a few more bucks and get something that might get upgraded to latest security updates.

    psb777 said: no hosting provider will allow you to abuse the CPU (which you share with your neighbor; if the VPS has any kind of dedicated CPU that's probably a physical core dedicated to you already) for 24 hours while you constantly brute forcing via SSH (in order to keep passwd loaded in cache).

    Customers of a provider that offers real dedicated cores should not have any problem with this security breach, but on some cases the cores can be "dedicated" rather than dedicated (aka not dedicated at all but sold as dedicated cores): some customers can feel secure while they aren't. If the core is sold as dedicated but isn't, you can use it 24/7 and won't get kicked out for abusing it while harvesting data: "dedicated" cores could be more dangerous than dedicated ones... :wink:

  • WebProjectWebProject Host Rep, Veteran

    @Neoon said:

    @LTniger said:
    AMD stock is rissing like watter in the nile.

    Intel stock has just dropped a single bit, like a few $, sad.
    Time for AMD to RYZE up.

    Not the first time issue with their processors, very interesting that no one suing them, as if they can’t develop processors without security bugs!

  • NeoonNeoon Community Contributor, Veteran
    edited May 2019

    @WebProject said:

    @Neoon said:

    @LTniger said:
    AMD stock is rissing like watter in the nile.

    Intel stock has just dropped a single bit, like a few $, sad.
    Time for AMD to RYZE up.

    Not the first time issue with their processors, very interesting that no one suing them, as if they can’t develop processors without security bugs!

    They had a lawsuit last time, Everything that is made by humans, has bugs.

  • psb777psb777 Member

    @jsg said:
    @psb777

    Careful! What we see now is but first proofs of concept. Also keep in mind that an attack going right at the jugular (the SSH password) shouldn't be compared to more general attacks.

    My professional understanding (I'm in IT sec dev) is certainly not that the heaven falls on our heads - but it's neither "no big thing. Just carry on".

    I agree. Attackers can get very creative, so it's best that we take precautions. But what we have in the OS is, as you pointed out, band aids. I won't blame VPS providers if they choose to err on the side of performance (or convenience, whatever) and my bytes get stolen. Really sensitive stuff shouldn't stay on a VPS in the first place.

  • jsgjsg Member, Resident Benchmarker

    @psb777 said:
    ... Really sensitive stuff shouldn't stay on a VPS in the first place.

    Yes.

  • @AnthonySmith said:

    pike said: Still Intel > AMD lol

    oh ok then, when you put it like that, i had not considered the greater than+lol perspective!

    Is Intel still that much better?

    I know performance on AMD desktop CPUs has improved greatly in the last few years, are they not suitable for servers?

    At least for LEBs

  • AC_FanAC_Fan Member

    @hostnoob said:

    @AnthonySmith said:

    pike said: Still Intel > AMD lol

    oh ok then, when you put it like that, i had not considered the greater than+lol perspective!

    Is Intel still that much better?

    I know performance on AMD desktop CPUs has improved greatly in the last few years, are they not suitable for servers?

    At least for LEBs

    AMD PR released a statement on the AMD subreddit that as far as they could tell after testing, Zen isn't affected by this. So Zen won't have to disable SMT and won't have the performance hit from the mitigation patch. That's a non-trivial gain in performance.

  • deankdeank Member, Troll

    @psb777 said:
    Really sensitive stuff shouldn't stay on a VPS in the first place.

    From what I can tell from behaviors / trends of LET, really sensitive (shady) stuff happen only on VPS.

Sign In or Register to comment.