Overselling, and when the customer discovers the issue, the response is like this =))

DigitalFyre

So here's what I think is happening.

SSDs can't overwrite flash data in place. The flash is written in relatively small pages, roughly 16 KB, but it has to be erased in much larger blocks, several MB at a time. So when a file changes, the drive writes the new data somewhere else and leaves the old data there until it can be cleaned up. The FTL (flash translation layer) tracks where the current data actually is.

Eventually the drive runs out of clean space and has to do garbage collection. It takes a block, copies out anything that's still valid, then erases the block so it can be used again. If most of the data in that block is already invalid, that's not a big deal. The problem is that this drive has had 887 TB written to it, mostly from lots of small, random writes from multiple VMs at the same time. That tends to scatter valid data, so the drive has to move more data around before it can free up space.

That also fits with what I'm seeing in the diagnostics. Reads are still fast at 8.5 GB/s, so this isn't a general performance problem. Writes are down at 185 MiB/s, and individual writes were taking around 147 ms at the drive level. The identical drive next to it was doing the same work in about 10 ms.

The other checks look normal. SMART is clean, wear is only 2%, temperatures are fine, the PCIe link is running at full Gen4 x4, and the power state is normal. The drive also has the same model and firmware as the one that's performing normally.

I'm going with a secure format (nvme format --ses=1) as a potential fix. That should clear the drive's internal mapping and erase the blocks, effectively giving the drive a clean start.

Since I can't look inside the FTL and prove garbage collection is causing the slowdown, this remains a diagnosis rather than something I can directly confirm. But given the symptoms and everything else I've checked, it's the most plausible explanation.

If the drive is still slow after a full secure wipe, then I'll treat it as a hardware or firmware problem. I'll put in a new drive and send the old one back under warranty.

I'll be sending out a maintenance notice tonight and get this taken care of ASAP.

forest

@DigitalFyre said: I'm going with a secure format (nvme format --ses=1) as a potential fix. That should clear the drive's internal mapping and erase the blocks, effectively giving the drive a clean start.

No need to do that. It's definitely the fault of write amplification as you surmised, but there's already TRIM which indicates which blocks are free to the garbage collector. My guess is your configuration is such that VMs are not passing through TRIM commands, so all the free space on everyone's VM is being treated as in-use and those blocks are unavailable for allocation. Without taking advantage of TRIM, you'll see this performance degradation.

Start up one of your OS templates on that node and run fstrim -av as root and, if its output is silent or it doesn't return a message like /: 3.1 GiB (3370385408 bytes) trimmed on /dev/vda1, then you've found your problem.

WyvernCo

@host_c said:

@WyvernCo said:
The stats for $3/mo is fine BUT they are being advertised as "High-performance AMD Ryzen & EPYC KVM VPS with blazing NVMe storage" The speeds are fine for many use cases but they are not NVMe speeds. Provider has oversold versus what they advertised.

3$/Mo and some are pissed it is a bit oversold/cramed?

Even if a product is at deep discount, if it doesn't meet what it was advertised as that constitutes false advertising. Beyond the illegality of misrepresenting a commercial product being sold, it also is damaging to the brands who use or defend such tactics.

TimboJones

@tiendungdev said:
I appreciate the honesty shown in your ticket acknowledging the issue, but the practice of overselling to the point of overloading raises serious questions about business ethics?!. While I understand that performance might drop compared to when I first purchased the service (when the server had fewer clients), the current level of degradation is unacceptable.

How is it unacceptable? If it was critical, you'd have metrics to monitor when it was acceptable and when it wasn't.

Naturally, I will be moving my business elsewhere, as there are plenty of good providers out there.

Let us know the price and performance.

Please note that I am not a new customer—I have been with you since the initial launch—so this kind of profit-driven overcrowding is particularly disappointing.

That's all businesses. What charity provider will you be moving to?

DigitalFyre

@forest said: No need to do that. It's definitely the fault of write amplification as you surmised, but there's already TRIM which indicates which blocks are free to the garbage collector. My guess is your configuration is such that VMs are not passing through TRIM commands, so all the free space on everyone's VM is being treated as in-use and those blocks are unavailable for allocation. Without taking advantage of TRIM, you'll see this performance degradation.

Start up one of your OS templates on that node and run fstrim -av as root and, if its output is silent or it doesn't return a message like /: 3.1 GiB (3370385408 bytes) trimmed on /dev/vda1, then you've found your problem.

Thanks for the tip. I'll do this first!

rcy026

@alincupunct said:

@rcy026 said:

@alincupunct said:

@rcy026 said:
Is it just me, or was that not a very honest and to the point answer from the provider?

They simply state facts and tell it like it is, they do not try to cover it up or come with excuses. They admit the node is loaded and they can not fix it right now and therefor they offer a refund. I would without hesitation accept that answer and live with half-decent performance until they get the opportunity to upgrade. After all, this is a vps, it is all shared resources so half-decent performance is what you should expect.

I do not have any vps with @DigitalFyre but this thread makes me want to get some, an honest provider is hard to find these days. This was probably not what op intended but he just gave them a new customer.

Great idea when they can't be bothered to respond to tickets and just use AI )

Well, if the response is correct and answers my question I do not care who writes it.
And they claim to check each answer even if it's written by AI, so saying that they "can't be bothered to respond to tickets" is not really a fair judgement.

Customer service is one of those areas where I do not mind correctly implemented AI, since the average AI is usually more intelligent then the average customer. Not trying to offend anyone, but after 30 years in IT my expectation on the average customer is unfortunately extremely low not to say non-existent.

Enjoy your 8 vcore 9950x getting 1.1k in gb6 then, buy more from dogshit providers like this.

For $10 a month I absolutely will.
If I needed the guaranteed performance of a 9950x I would not buy a vps, I would buy a vds or dedicated. It still baffles me that people do not understand even the basic concept of what a vps is.