New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Vultr patched to hide cpu steal
Not 100% sure this is the case since it's so hard to prove.
But I had a vps at Vultr Japan that was running a load that should have been 30% of a core and then after a certain point in time it went up to >80% even when there were no users (all the users were gone due to the lag). Yet the CPU steal was only listed at 1-10%
Support said that the level of cpu steal was normal and suggested that we buy their dedicated plan which cost 600% more.
What level of cpu steal is acceptable for a vps here and is there any way to find out which hosts have low cpu steal without buying it and hoping it doesn't become worse some months later?
Comments
Depends on the Application, for example wireguard is getting quite unhappy with cpu steal.
Ideally you want have none.
If its a cloud provider, just deploy again, get on a less busy node.
Either its a limit you hit or the Node is just loaded beyond repair.
If you have a consistent high load, get a VPS with dedicated cores or a smol dedi.
Cloud Providers are quite expensive in the long run, LET is cheaper.
I doubt that Vultr is "hiding" your CPU Steal, could be just the load that changes on the Node.
None, almost. We have max 0.1% at the node level, after that is alarm.
Well, TBH 0.2, then is the alarm triggered, so, technically, it is 0.2, but 0.1 is more or less accepted, otherwise we would get alarms all the time.
One of the most "abused" nodes running our very cheap big ass plans:
how is it possible that it only shows 10% cpu steal max while my load that used to take 30% is now doing 90% with no users?
Personally I don't think Vultr would cheat you, as they have lots of spare resources and no need to do such thing.
Vultr is very trustworthy and professional in most of time as I know.
Also steal time of each cpu in Linux is calculated by kernel directly, I don't think there exists an easy way to patch that in a proper way and cheat user,
Also, em...
For patch that in qemu (and here just for a tech discussion), did you have any idea for that?
This doesn't make sense to me, steal time is information sent from the hypervisor to the guest. The guest has no control over it.
https://lore.kernel.org/lkml/[email protected]/
Interesting,
I will dig into it and make a real qemu cheat patch for that later.
At first sight it shouldn't be difficult. But it might be possible to expose it somehow, like comparing clocks or similar.
I think what we need it the anti-cheat, that would probably be more useful and help OP find the truth along with many other people.
After a quick view, it seems that qemu maintains a phyiscal memory zone (and labeled its addr in msr 0x4b564d03 passing through into guest vm) and put this inside
:
then kernel get it maybe from func "paravirt_steal_clock"
The anti-cheat is to run something you know would take the known amount of cpu to run, which I was already doing.
My stuff should have been using only 30% of a core, yet it was showing up to 90% while only displaying 10% cpu steal. Unless there's some other unknown bug that causes this, this is a sure sign vultr is faking cpu steal.
Or I guess it could be massive cpu throttling.
10% is enormous anyway, it might indicate that many people are using a lot of CPU, like VMs infected with some cryptomining or someone suddenly deploying a ton of VMs to do that using fake cards to pay or similar.
If 10% st looks normal to you and only think it might be a problem if it is bigger and dissimulated somehow, then you are probably in need of a better advice regarding alternatives.
Doesn't 10% steal only mean that the cpus were oversubscribed by 10%?
No, unless you bought dedicated cores.
For example, if I run crypto mining and my VM shares a core with you, the hypervisor would find it hard to switch back in time for your VM to do its computing on the same core. The settings could be fair-share, but the hypervisor would have trouble facing such a situation.
It could be bad luck, i.e. someone sharing a core with you that is very aggressive, or overselling on the whole node and the hypervisor has trouble with the switching in time.
Managing the CPU is a very complicated task in shared environments and throttling is complicating it even further, if that is not mission critical, I would snapshot the VM and redeploy it just to test the theory. If bad luck, then the st would be gone or lower and your task would go pretty much as before, if everything is oversold/throttled, then nothing much would change.
It is hard, as guest vm is usually lack of a real physical wall clock observer. Unless you request to attach a hardware device into your vm for that. Some pci ethernet card have such api func.
Any way accessing host's wall clock inside qemu vm can be cheated technologically. Just like the film Martrix, vm running inside host is just a simulated world. Maybe our universe also works like this.
I was thinking more along the lines of counting the actual cycles your VM uses, then get the nominal core MHz against an external clock for a long time interval (the user says he knows exactly how long it should take his compute and therefore it is reasonable to assume this could be run for a long time and make an average). It is also highly unlikely a CPU with high st is actually idled by some power saving system, so the MHz should not be lower, but maybe higher as some CPUs support bursting for a while. But it could be throttled due to thermal issues and that is very difficult to catch.
It is all moot, anyway, because the cores are not dedicated, most likely, so, as the saying goes: YMMV.
That kind of performance loss is not surprising at all when the node is so overloaded that it constantly gives you 10% CPU steal time.
That doesn't mean you're losing 10% of theoretical performance. That means your VM wanted to run some code, but the hypervisor forced it to waste 10% of a unit time. It's relative to what you do on the VM - try using 100% of CPU time and watch CPU steal time% to go even higher. You also can't ignore performance loss due to CPU cache getting clobbered during context switches.
Vultr is cheating(?) you by putting your VM on a massively overloaded node. Tell them 10% steal is unacceptable and request moving your VM to another node.
OP already did that, but they said it is normal and tried to sell him dedicated plans. That is beyond the pale at double digits st, maybe at the level of summer hosts here it could be acceptable for 6 dollars a year, but not for a serious provider.
That's not my experience. But it's not hard to imagine support responds like this if OP writes they constantly use 30% CPU (well - it's more like 50% in the screenshot) and also something like "users reported lags". It indeed seems be on the borderline of considering moving to a larger plan.
Vultr has hourly pricing. Why do you ask them to move your VM to another node? Why don't you just create a new VM? That's faster for sure. No need to wait, and you can create as many VMs as you want. Pick the one which is the best performer. CMIIW.
At least that what have been doing when benchmarking "hourly" pricing providers. One of them is Vultr.
No no, sorry, please don't do that all the time, as everytime you create then quickly delete a VM, it takes not so little resources in host to deal with it, which disturbing your neighbours. Such thing should be considered as very bad abuse.
Sadly i'm a honest person, and i'm pretty sure, i'm not the only one do this. What i did? i just ran a single YABS and Geekbench 6 or PassMark's PerformanceTest, to see the score. That's it.
I think that's the flexibility of using hourly pricing providers. You can create VM and destroy it. Sadly there is no Rules or whatsoever on how many VMs you could create and destroy per hour or per day.
As a cloud provider (charge by minute) we know this very well and one of our main concern was the abuse possibilities too, i.e. someone spinning VMs for spam, delete, create new after the previous IPs have been deleted.
However, that risk never materialized as we have a high entry bar at 30 Eur minimum payment so the abusers are usually going elsewhere.
The correct answer is that any performance is acceptable if the server performs the function you set it to without any delays that you consider unacceptable, where the delays are system performance rather than application configuration or design issues. That sounds obvious but it's easy to lose sight over it.
Steal being a thing in a cloud environment is generally acceptable, shared resources for lower cost is the entire purpose of a cloud server, and shared resources are going to be at least mildly volatile. It should just be quite low on average. More than a tiny amount or the occasional spike is just evidence or a need to rebalance the host.
That is the crux of the matter, how low is low enough? I consider 10% to be enormous and unacceptable for any more than a few hours a month (shit could happen, but it should not take forever to find the abusers).
Yeah it's really hard to just nail down a number and say "This is it." To be honest the only time I've had cloud servers underperform based solely on their performance, it wasn't related to steal. If I've had steal and I have to assume that I have, it didn't impact my expectations based on the role of that server, so I'd have just had no reason to go looking for it. That's why I always question if it's an issue of graphs generating the complaint or did they go look at the graphs to explain a problem.
I mean, feel free to steal from my postfix box. If the mail queue stays low and the mail gets processed within 2 seconds, I won't be found looking at a CPU graph.
Look, I don't say that 10% would be a deal-breaker killing the VM, no, what I meant is that this kind of st level is a symptom of bigger problems. You can have shared cores and have 0 steal, it is not a given of a shared CPU environment, st means that the CPU has trouble switching and that is the kind of thing that would be snowballing into a stall.
You know, the cache is wiped, the switching takes more and more time, there could be some throttling getting in the way of the system's scheduling which is already under severe strain and the performance hit would be way bigger than 10%, the CPU is overwhelmed and all scheduling goes to hell. I do not doubt that the performance hit is much more than 10%, I would be surprised if it wouldn't have been, as far as I can tell from guesstimating out of personal experience with hundreds of nodes of any kind of hardware, anything above a couple % of st is really-really dangerous for the system's stability. It might work up to 5 or so, but above the CPU performance is dropping to the floor and sometimes it goes so badly that the system becomes unresponsive and you can't even shutdown the VMs safely.
In short, the system could go from an equilibrium into a crash in a few seconds, maybe minutes, it all depends on the actual tasks ran, some can be interrupted easier than others, the cache can help or be in the way, you COULD run a system like this if you know what your VMs are doing all the time and the load is stable, but when you have VMs you have no idea what are running, it is better to be way more conservative.
This is what they told me when I told them my node was overloaded. I was on a $40/month plan as well. I wasn't using a $5 plan and expecting 100% resources.
It is really disappointing, vultr had good performance for many years.