Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Processor 'steal', the horrifying monster explained
New on LowEndTalk? Please Register and read our Community Rules.

Processor 'steal', the horrifying monster explained

jsgjsg Member, Resident Benchmarker
edited August 2021 in Tutorials

As the 'steal' ~ "provider XYZ totally brutally overselling!!!!" posts pop up often times, especially in my reviews, I'll put what I just explained in a thread here in it's own thread to get the "steal! Overselling!!!" scare stories taken care of, hopefully once and for all.

Quick and short explanation: 'steal' tells about the OS's view which is based on an OS vCPU which is a HWT. With a VDS that is what you get, a full vCore as seen by the OS. So, on say, a 16 core Epyc with 32 hardware threads (HWT) the node OS sees 32 vCores.

A VPS's vCore however is a shared HWT, typically 25% or, if you are lucky 33% of a HWT (and even less than 25% with some products/providers).

'top' and similar utilities report from the OS's perspective, not from the VPS perspective, or, to be more precise, they assume that a vCore is a full vCore/HWT. VPS however are "fair share" that is, a provider packs e.g. 3 VPS vCores on a HWT and then talks about something like "2 vCPU, fair share 33%" - and if on such a VPS 'steal' shows even 60% that is normal and to be expected.

You don't like that, you want a full HWT for yourself? No problem, just buy a VDS.

Side note: don't we all like to have some "burst capacity" like e.g. "you may use up to double your vCPU share for up to 5 minutes per hour"? Now, where do you guess that capacity comes from? Unless you believe in fairy tales it should be clear that there are only 2 potential sources, (a) capacity held in reserve by the provider, so in our example a provider not selling 32 x 3 = 96 vCores but only say 90, and (b) the fact that the vast majority of VPSs aren't fully used anyway and hence there is almost always some unused capacity.

If you want to be smart then you'll log the 'steal' over a reasonable period of time, say 48 hours, at least once per second (ideally even 10 times per second) and check whether, and if so how much, the 'steal' is greater than 100 - 'fair share' (e.g. 33). Because that - and only that - would indicate overselling and even then only if it happens not just rarely to occasionally and for short period of time (because those cases would indicate burst being used).

«1

Comments

  • jarjar Patron Provider

    Steal is so misunderstood because "steal" is such a combative word to begin with. It puts people in a bad place right from the start. If you don't understand linux terminology it's totally okay, but it should start from a place of "Is my app performing to it's capabilities, to my expectations, and am I paying a price I'm comfortable with?" Because if the answer is yes to all three, no other meaningless metric should be of value to you.

    Thanked by 2jsg chocolateshirt
  • @jsg
    Very nicely explained.

    I prefer VPS with dedicated execution thread (vCore), ideally dedicated core with both the HWT. Then there is no issue of cpu steal or cpu hitting 100%.

  • jsgjsg Member, Resident Benchmarker

    As some smartass and yabs fan just felt the need to lecture me in my Contabo review thread, I'll explain one decisive factor that seems to be very hard to grasp for some "experts" again and hopefully even clearer.

    Modern hypervisors can and do "portion" vCores - but the VPS still do not know about that. When running e.g. 'top' in a VPS it doesn't know that the VPS actually only has e.g. 33% of the vCore, so it reports 'steal'. To change that and to resolve that problem something like a vvCore would be needed where a vCore is a HWT and a vvCore is (in our example) 1/3rd of a vCore. But that's not how it's done currently.

  • jsgjsg Member, Resident Benchmarker

    @dev_vps said:
    @jsg
    Very nicely explained.

    I prefer VPS with dedicated execution thread (vCore), ideally dedicated core with both the HWT. Then there is no issue of cpu steal or cpu hitting 100%.

    Thank you, and: well, who wouldn't? But alas VDS are much more expensive ...

    Plus, let's be honest, the vast majority of even VPSs is not fully used (and that's not even talking about the idlers).
    I think two major factors in that are human psychology (we love to be "on the safe side" that is, to have reserves) and many if not most users not being able to guesstimate their real need correctly.

  • NeoonNeoon Member
    edited August 2021

    You see steal, if the host system can't give you the cpu time you want.
    Either its a limit you hit or the host system dosen't have the cpu time you desire available because its loaded to the roof.

    Any CPU steal will slow your application, to say xx% is fine, is bullshit.
    A loaded host system also impacts network traffic and latency on top of that.

    So your optimal goal is zero cpu steal.
    But on the low budget end its not always possible, see Virmach, Racknerd...

  • jarjar Patron Provider
    edited August 2021

    0.00% iowait is ideal but you won't sustain it even on dedicated hardware. It's part of running a server. Shared boxes are shared boxes, and thus they'll always be.

  • @jar said:
    0.00% iowait is ideal but you won't sustain it even on dedicated hardware. It's part of running a server. Shared boxes are shared boxes, and thus they'll always be.

    On Virmach, I often do see 60% indeed.

    However, a bunch of Virtual Machines I rented, got less than 1% steal.

    Usually it goes somewhere between 1-2%, some spike to 5% sometimes.

  • jsgjsg Member, Resident Benchmarker

    @Neoon said:
    You see steal, if the host system can't give you the cpu time you want.
    Either its a limit you hit or the host system dosen't have the cpu time you desire available because its loaded to the roof.

    Any CPU steal will slow your application, to say xx% is fine, is bullshit.
    A loaded host system also impacts network traffic and latency on top of that.

    So your optimal goal is zero cpu steal.
    But on the low budget end its not always possible, see Virmach.

    Pardon me but that's half BS and half day dreaming. Why do you think providers set a limit like e.g. fair share 33%?
    Simple reason: they can portion vCores in the hypervisor but the VPS doesn't know that it only has 33% of what it sees as its vCores, so it has to somehow be communicated and customers need to repect it.

    And NO, not any steal will slow the application. It will only slow the application if the application wants more than its share of resources. So if you see that your application is too slow then that's not a reason to complain about "steal!! Overselling!!!" but rather a reason to get a beefier VPS.

    Which of course boils down to "you don't get unlimited resources for a limited price", duh.

    Again, keep in mind that resources, particularly the processor, being shared is the very definition of VPS.

  • @jsg said:

    @Neoon said:
    You see steal, if the host system can't give you the cpu time you want.
    Either its a limit you hit or the host system dosen't have the cpu time you desire available because its loaded to the roof.

    Any CPU steal will slow your application, to say xx% is fine, is bullshit.
    A loaded host system also impacts network traffic and latency on top of that.

    So your optimal goal is zero cpu steal.
    But on the low budget end its not always possible, see Virmach.

    Pardon me but that's half BS and half day dreaming. Why do you think providers set a limit like e.g. fair share 33%?
    Simple reason: they can portion vCores in the hypervisor but the VPS doesn't know that it only has 33% of what it sees as its vCores, so it has to somehow be communicated and customers need to repect it.

    The limit only matters if you hit it, otherwise it dosen't.
    Especially not on idle, if I see 20% steal on idle, I am looking for a gun and the termination button.

    And NO, not any steal will slow the application. It will only slow the application if the application wants more than its share of resources.

    Have a look at this image, this was on idle right.

    The steal I saw in top, reflects on the network and vice versa.
    If you just run a webserver you might not care, if the latency jumps by 10-20ms but at some applications you do care.

    I swear by my balls, that was only 15% steal, it does impact stuff.

    So if you see that your application is too slow then that's not a reason to complain about "steal!! Overselling!!!" but rather a reason to get a beefier VPS.

    Which of course boils down to "you don't get unlimited resources for a limited price", duh.

    Stop bullshitting me, I got so many virtual machines for such a low price with a fucking SINGLE DIGIT number of STEAL.

    That argument, jesus, steal by my balls christ.

    Again, keep in mind that resources, particularly the processor, being shared is the very definition of VPS.

  • jsgjsg Member, Resident Benchmarker
    edited August 2021

    @Neoon

    You maybe put it better than I could when you said that you see 'steal' on idle VMs.
    Or in other words: How can one trust 'steal' values? How can an idling VM fail to get enough processor time?

    For me what you wrote is just one more nail in the 'steal' coffin.

    Well noted, I do not state that there is no overselling! In fact I'm quite convinced that quite some overselling is going on. Simple reason: the mixture of "smart capitalism", many idlers, most customers not fully using their VPS and a level of complexity that many can't fully grasp pretty much guarantee that some providers won't resist the temptation and will oversell.

    What I say is that - whatever anyone considers a high - steal is rarely and hardly a good basis, let alone evidence, for alleging overselling.

    Re Contabo / @contabo_m I happen to know (from private discussions) that they have a couple of largely unused HWT on all of their Epyc nodes (and presumably on other nodes as well). And unlike many "overselling!!!" crybabies I do have the data and plenty of them to stand on.

  • @jsg said:

    @dev_vps said:
    @jsg
    Very nicely explained.

    I prefer VPS with dedicated execution thread (vCore), ideally dedicated core with both the HWT. Then there is no issue of cpu steal or cpu hitting 100%.

    Thank you, and: well, who wouldn't? But alas VDS are much more expensive ...

    Plus, let's be honest, the vast majority of even VPSs is not fully used (and that's not even talking about the idlers).

    The VDS I have has one dedicated core with both HWT … 16gb RAM and 80gb SSD …. $25 per quarter. I think that is pretty reasonably priced

    I use this VDS with Visual Studio for C# programming, I don’t have to worry about CPU cores hitting 100% for extended times.

    I do have VPS from racknerd (@dustinc). Priced under $24 for the year. 3 vCPU with 2.5GB ram and 40 GB storage. I try to keep average cpu% load under 25%. Very pleased with the performance … and excellent value for the money

    In short, VDS and VPS are good based on the use case.

    Thanked by 1jsg
  • @Neoon said: I swear by my balls, that was only 15% steal, it does impact stuff.

    Stop bullshitting me, I got so many virtual machines for such a low price with a fucking SINGLE DIGIT number of STEAL.

    Over the years, I have had VPS from a lot of providers, I have never seen CPU steal greater than 5% (even after using the CPU for a couple of hours) except for Contabo.

    @jsg said: So if you see that your application is too slow then that's not a reason to complain about "steal!! Overselling!!!" but rather a reason to get a beefier VPS.

    So, I got a Contabo 4 core AMD EPYC VPS for $8.50. Within days, I get >50% steal, I see my applications are crawling, and its running slower than a $3 single core OVH VPS or a $2 BuyVM VPS. According to OP's logic, its my fault for the poor performance, and I need to spend more money on Contabo. This is total BS.

    @jsg said: What I say is that - whatever anyone considers a high - steal is rarely and hardly a good basis, let alone evidence, for alleging overselling.

    Overselling is when you make false promises like 4 cores while in reality delivering the performance of less than 1 core.

    @jsg said: Again, keep in mind that resources, particularly the processor, being shared is the very definition of VPS.

    Of course VPS is shared, but I have never had issues bursting and using all the cores with other providers.

    @jsg said: Re Contabo / @contabo_m I happen to know (from private discussions) that they have a couple of largely unused HWT on all of their Epyc nodes (and presumably on other nodes as well). And unlike many "overselling!!!" crybabies I do have the data and plenty of them to stand on.

    I wonder if OP works for Contabo the way he is singing praises for them. Please search for Contabo steal/overselling on LET and internet, and you will find plenty of posts where people have complained about this.

    Again, Contabo is not a bad provider, but they are very sketchy and unpredictable when it comes to CPU performance. If you care about performance, I would suggest look elsewhere.

  • jsgjsg Member, Resident Benchmarker

    @itsnotv said:

    @jsg said: So if you see that your application is too slow then that's not a reason to complain about "steal!! Overselling!!!" but rather a reason to get a beefier VPS.

    So, I got a Contabo 4 core AMD EPYC VPS for $8.50. Within days, I get >50% steal, I see my applications are crawling, and its running slower than a $3 single core OVH VPS or a $2 BuyVM VPS. According to OP's logic, its my fault for the poor performance, and I need to spend more money on Contabo. This is total BS.

    No, BS is that you say nothing about your application (nor about the load factor).

    @jsg said: What I say is that - whatever anyone considers a high - steal is rarely and hardly a good basis, let alone evidence, for alleging overselling.

    Overselling is when you make false promises like 4 cores while in reality delivering the performance of less than 1 core.

    For a start, no, that's not the definition of overselling (but possibly a symptom of it).

    As for false promises I'm with you - whenever I ask a provider to publicly declare how many VPS vCores are packed on the real HWTs ... they get quiet.

    But then, to be fair, we also must see that, pardon me, stupid customers just going for "many vCores, the more the better" without looking at (or asking for) what a vCore (as appraised) actually is in terms of HWT%, it's a temptation for providers to play those games.

    @jsg said: Again, keep in mind that resources, particularly the processor, being shared is the very definition of VPS.

    Of course VPS is shared, but I have never had issues bursting and using all the cores with other providers.

    I prefer hard data over "I had (or had not) issues".

    @jsg said: Re Contabo / @contabo_m I happen to know (from private discussions) that they have a couple of largely unused HWT on all of their Epyc nodes (and presumably on other nodes as well). And unlike many "overselling!!!" crybabies I do have the data and plenty of them to stand on.

    I wonder if OP works for Contabo the way he is singing praises for them. Please search for Contabo steal/overselling on LET and internet, and you will find plenty of posts where people have complained about this.

    IF I was a fanboy of a provider then of NexusBytes / @seriesn (it's not a secret).

    What I said wrt Contabo is - unlike your subjective, uhm, "reports" - based on hard data gathered in hundred of runs. And btw, the fact that I used Contabo as an example case is simply due to the fact that the "steal!!! Overselling!!!!" BS most recently happened to come up in my Contabo benchmark. My remarks and statements are not specific for any particular provider.

    As for "just search on the internet", sorry, but that seems to mainly show something about your state of mind and your way of thinking (applying the term loosely). You can find "confirmation" for everything in the internet, incl. for a vast array of nonsense.

  • hi, i read this .......

  • edited August 2021

    @Neoon said:
    I swear by my balls, that was only 15% steal, it does impact stuff.

    Sometimes I have 5-10% CPU steal on idle vps.. and yeah they affect the performance of ssh.

    Note:

    The person who easily judge other people as BS is the most BS person in the world..

  • Very good reading...

  • The issue is a-lot of hosts are not transparent with how much of the CPU you are being limited too. If its 25% and you order a plan with 4 vCores then you are getting the resources of 1 single core so its total marketing BS. I only work with providers who provide 100% of each core.

  • @jsg said: As for "just search on the internet", sorry, but that seems to mainly show something about your state of mind and your way of thinking (applying the term loosely). You can find "confirmation" for everything in the internet, incl. for a vast array of nonsense.

    You don't want to believe me, nor you want to believe what many others have written. That's fine, if Contabo is working great for you, I am happy for you. Maybe you got lucky or they really like you, and have given you a dedicated node for your VPS. Anyone else thinking of paying that setup fee and getting a Contabo, do your research. CPU steal will have a very adverse affect on the performance of your server, and good providers have almost none.

  • jsgjsg Member, Resident Benchmarker
    edited August 2021

    @noobjockeys said:
    The issue is a-lot of hosts are not transparent with how much of the CPU you are being limited too. If its 25% and you order a plan with 4 vCores then you are getting the resources of 1 single core so its total marketing BS. I only work with providers who provide 100% of each core.

    ... which is called VDS

    @itsnotv said:
    You don't want to believe me, nor you want to believe what many others have written. That's fine, if Contabo is working great for you, I am happy for you. Maybe you got lucky or they really like you, and have given you a dedicated node for your VPS. Anyone else thinking of paying that setup fee and getting a Contabo, do your research. CPU steal will have a very adverse affect on the performance of your server, and good providers have almost none.

    Nope, it's nothing personal nor is it pro-Contabo (or any other provider). The point is that it's an "everybody knows that" belief. Those are hard to counter because the believers really really believe it and will defend their belief.

    But you see, I don't just willy nilly make baseless assertions and certainly not pro or anti some given provider. If people want to continue to - wrongly - believe that the 'steal' number is evidence for oversell, fine with me. But for those who want to learn and know the facts I wrote this OP.

    And my base isn't belief and "on provider X VPS my application runs slow and top shows high steal". My base is knowledge, for example the knowledge required to write a complex benchmark software as well as the knowledge of OS, incl. how they work internally, along with tons of data gathered in thousands of test runs on diverse VPS and VDS of diverse providers, some of which value my software and my benchmarking enough to ask me to do it for them or with their product(s) and some of them do not even care about my review here, they just want the data, data they can rely on and that helped them multiple times to find flaws, sharp edges, thing to improve with their products.

    But again, if you want to continue to believe in 'steal' ~ overselling, fine with me. Just keep your mind open allow people with certainly not less experience than you to tell their view too.

  • rm_rm_ Member
    edited August 2021

    @jsg said: You don't like that, you want a full HWT for yourself? No problem, just buy a VDS.

    I wish that you didn't try to promote the Russian idiosyncrasy of "VDS", nobody else in the world has that as a meaningful distinction (I heard maybe a few hosts in Germany but that's it), so no need to invent one.

    You will find tons of hosts selling "VDS" who didn't even think to mean that it has dedicated cores or whatever (and oversell to the Moon). It's just how they used to call VPSes since 2000s, until the current terms were established yet.

    As such there's no important product difference of VPS vs VDS that you can rely on. For a dedicated cores product, everyone just calls it a "dedicated CPU core VPS", so let's just continue.

  • jsgjsg Member, Resident Benchmarker

    @rm_ said:

    @jsg said: You don't like that, you want a full HWT for yourself? No problem, just buy a VDS.

    I wish that you didn't try to promote the Russian idiosyncrasy of "VDS", nobody else in the world has that as a meaningful distinction (I heard maybe a few hosts in Germany but that's it), so no need to invent one.

    You will find tons of hosts selling "VDS" who didn't even think to mean that it has dedicated cores or whatever. It's just how they used to call VPSes, since 2000s, until the terms were established yet.

    There's no important product characteristic difference of VPS vs VDS. For dedicated cores, everyone just calls it a "dedicated CPU core VPS", so let's just continue.

    You have a point there, but the problem is that there are many, too many, terms out there for what I - and many providers - call 'VDS'. No matter which term I use, there will be some who complain (e.g. Germans who tend to call it "root server", which looking closer, doesn't even make sense).

    One thing is certain: I won't call it "dedicated CPU core VPS". Mainly 2 reasons, (a) V[whatever]S are not based on cores but on vCores (or vCPUs) ~ HWT, and (b) what a long snake of a term!
    I guess I'll stick with 'VDS'; it's short and it's similar to 'VPS' but with a difference - which actually fits well.

  • while it's somewhat derailing the thread, how the guest know it stolen CPU cycle? (assume 1vcpu $7 VPS) if host noped to execute guest code, wouldn't VM frozen in essence? is this something host will reply back to guest?

  • AlwaysSkintAlwaysSkint Member
    edited August 2021

    I prefer to view steal as a means of checking the general performance of my virtual servers. IME, whilst it can indicate overselling it can also show symptoms of other problems. Analyse the data along with system Load and I/O wait, to get a better overall picture of system performance.
    On a VPS with Magento bloatware, steal had a significant impact but the route to better performance was to increase IOPS by switching from HDD to SSD. The current VPS, runs with a larger steal percentage than I'm used to seeing but appears to be of little significance. CPU usage peaks frequently for relatively short durations.
    My main VPSes, running WHM/cPanel and multiple e-commerce stores are 'happy' to run on spinning rust and normally experience steal below 2%, all of the time. The only time Load is of concern, is during nightly backups, where I/O has the greatest influence and subsequent increase in Load. Overall, CPU usage peaks far less.

  • rustelekomrustelekom Member, Patron Provider

    I am afraid use VDS to separate shared core from dedicated core. Actually VPS and VDS talk about same virtual machine. If you are looking into dedicated cores then find host who provide such option and be ready that prices would be higher in several times. Why? Because if host do not share cores then he limited by core number to host virtual machines on node. Let's say 2 x Decade core node will serve only 19-20 client virtual machine. Compare this with shared cores - same node can host over 50 and even 100 virtual machines...

  • mostly
    backup process need burstable core (in minutes )

    so ..........

  • NeoonNeoon Member
    edited August 2021

    Well, there is a good quote:

    "Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor."

    Which brings it to the point.

  • jsgjsg Member, Resident Benchmarker
    edited August 2021

    @Neoon said:
    Well, there is a good quote:

    "Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor."

    Which brings it to the point.

    Yes, correct.
    But (a) we are talking about microseconds, and (b) if the virtual processor (aka vCore) of a VM actually is just say, 25% of a HWT - and again, it doesn't know about that - and the VM has workload for more than 25% of a HWT then 'steal' will happen/be shown without any overselling or other evil provider action.

    @rustelekom said:
    I am afraid use VDS to separate shared core from dedicated core. Actually VPS and VDS talk about same virtual machine. If you are looking into dedicated cores then find host who provide such option and be ready that prices would be higher in several times. Why? Because if host do not share cores then he limited by core number to host virtual machines on node. Let's say 2 x Decade core node will serve only 19-20 client virtual machine. Compare this with shared cores - same node can host over 50 and even 100 virtual machines...

    That's exactly what a VDS is. A VM with dedicated HWT/vCores.
    One can use different hypervisors and hence different VM types but at the end of the day when a processors cores are portioned then the VMs will use/run on vCores. The difference is just that a VDS vCores are full HWTs while a VPS vCores each are but fractions of a HWT. Btw, if one could dish out real cores the VM would still see vCores because that's how (at least) Unix OSs handle it.

  • rustelekomrustelekom Member, Patron Provider

    I am still afraid that such definition can confuse customers only. Because at many sites you can see such explanation "A virtual private server (VPS), also called a virtual dedicated server (VDS), is a virtual server that appears to the user as a dedicated server, but that is actually installed on a computer serving multiple websites. A single computer can have several VPSs, each one with its own operating system (OS) that runs the hosting software for a particular user.".
    So, i prefer more clear explanation like virtual machine with dedicated resources (like for example Hetzner use). Even if you dedicate core to customer virtual machine that no warranty all other resources will not shared (disks, ram, port). Of course this is only my personal view of point so anyone is free use any term and explanation as he want.

    Thanked by 1rm_
  • jsgjsg Member, Resident Benchmarker

    @rustelekom said:
    I am still afraid that such definition can confuse customers only. Because at many sites you can see such explanation "A virtual private server (VPS), also called a virtual dedicated server (VDS), ...

    Yes, that is confusing (IMO even intentionally) and nonsensical and false too.

    is a virtual server that appears to the user as a dedicated server, but that is actually installed on a computer serving multiple websites. A single computer can have several VPSs, each one with its own operating system (OS) that runs the hosting software for a particular user.".
    So, i prefer more clear explanation like virtual machine with dedicated resources (like for example Hetzner use). Even if you dedicate core to customer virtual machine that no warranty all other resources will not shared (disks, ram, port). Of course this is only my personal view of point so anyone is free use any term and explanation as he want.

    I agree - but: should we really stop using a term that is widely used and well known just because some idiots (or fraudsters) abuse it? I think no, we should not. We rather should have a serious word with those few providers and, if needed, bring them to court.

  • dev_vpsdev_vps Member
    edited August 2021

    For my VPS, the core is dedicated (with both its HWT). That is why it shows one socket with 2 virtual CPU.

    if the application is cpu intensive, VDS will save load of headaches.

Sign In or Register to comment.