Strange slowdown of storage server

PineappleM · April 2025

I have a 1 TB storage server with Host-C. YABS is at the bottom of the post. I only have nginx installed with default settings and it serves a static website. There's nothing else running on it (I didn't even mount the 1 TB drive). The server runs great for 2 months, then the website stops responding.

When I log into the server to see what's going on, it is extremely slow. Each SSH prompt take seconds to show up as if the server is choking, yet RAM usage is about 300 MB and there's no heavy CPU usage. I have to reboot the server, after which everything is speedy and works fine again (website resumes working). This has happened twice now since I got this server (last year).

I have had many VPSes before from all sorts of providers and I've never seen this issue happen. Anyone have any recommendations on what I can try so that the VPS doesn't lag itself to death over time?

Two ideas I had:

Add a swap file (not sure why there isn't one by default)
Install the official Debian image and not the one provided by Host-C (dunno if this would make a difference)

Any other suggestions would be appreciated. Thank you!

# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
#              Yet-Another-Bench-Script              #
#                     v2025-04-20                    #
# https://github.com/masonr/yet-another-bench-script #
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #

Sun Apr 27 13:07:41 UTC 2025

Basic System Information:
---------------------------------
Uptime     : 0 days, 13 hours, 24 minutes
Processor  : Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
CPU cores  : 1 @ 2999.996 MHz
AES-NI     : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM        : 973.3 MiB
Swap       : 0.0 KiB
Disk       : 9.8 GiB
Distro     : Debian GNU/Linux 12 (bookworm)
Kernel     : 6.1.0-33-cloud-amd64
VM Type    : KVM
IPv4/IPv6  : ✔ Online / ✔ Online

IPv6 Network Information:
---------------------------------
ISP        : Andrei Tiberiu Holt
ASN        : AS211462 Andrei Tiberiu Holt
Host       : Andrei Tiberiu Holt
Location   : Oradea, Bihor County (BH)
Country    : Romania

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sda1):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 194.35 MB/s  (48.5k) | 1.19 GB/s    (18.6k)
Write      | 194.86 MB/s  (48.7k) | 1.19 GB/s    (18.7k)
Total      | 389.22 MB/s  (97.3k) | 2.38 GB/s    (37.3k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 1.17 GB/s     (2.2k) | 1.16 GB/s     (1.1k)
Write      | 1.23 GB/s     (2.4k) | 1.24 GB/s     (1.2k)
Total      | 2.40 GB/s     (4.7k) | 2.40 GB/s     (2.3k)

iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping
-----           | -----                     | ----            | ----            | ----
Clouvider       | London, UK (10G)          | 302 Mbits/sec   | 297 Mbits/sec   | 43.9 ms
Eranium         | Amsterdam, NL (100G)      | 289 Mbits/sec   | 297 Mbits/sec   | 42.8 ms
Uztelecom       | Tashkent, UZ (10G)        | 199 Mbits/sec   | 286 Mbits/sec   | 130 ms
Leaseweb        | Singapore, SG (10G)       | 135 Mbits/sec   | 240 Mbits/sec   | 191 ms
Clouvider       | Los Angeles, CA, US (10G) | 195 Mbits/sec   | 274 Mbits/sec   | 165 ms
Leaseweb        | NYC, NY, US (10G)         | 237 Mbits/sec   | 276 Mbits/sec   | 103 ms
Edgoo           | Sao Paulo, BR (1G)        | 130 Mbits/sec   | 264 Mbits/sec   | 236 ms

Geekbench 4 Benchmark Test:
---------------------------------
Test            | Value
                |
Single Core     | 3657
Multi Core      | 3534
Full Test       | https://browser.geekbench.com/v4/cpu/18687390

YABS completed in 9 min 27 sec

wadhah · April 2025

I mean everything looks fine in the YABS, If you have nothing important then I suggest doing a fresh install through netboot.xyz, make sure virtio drivers are installed

Did you have bbr as the tcp algorithm?

sh97 · April 2025

whats your latency to the VM?

PineappleM · April 2025

@wadhah said:
Did you have bbr as the tcp algorithm?

I'm going to guess no since I didn't explicitly set the TCP algorithm to anything. Any way I can check for that?

@sh97 said:
whats your latency to the VM?

131 ms. I've had better responsiveness with Singapore VMs (250 ms). I live near New York and the VPS is in Romania.

wadhah · April 2025

@PineappleM said: I'm going to guess no since I didn't explicitly set the TCP algorithm to anything. Any way I can check for that?

>

sysctl net.ipv4.tcp_congestion_control

somik · April 2025

Have you contacted your provider? Open a support ticket with them and ask? It could be due to someone else abusing the server or the node being oversold and being out of resources.

lukast__ · April 2025

Your YABS looks fine.
While I personally haven't seen either on my Host-C VPSes in any significant amounts (actually they have some of the lowest steal from all of my VPSes), you could try to monitor IOwait and steal and look whether there are spikes before or while these issues occur.
Another possibility would be that somehow something in your install got corrupted, maybe even the kernel (but as you seem to be keeping your system up-to-date (current debian stable kernel installed), and it occurs even though you rebooted (likely to a new kernel), that doesn't seem as likely), but also something like a corrupted systemd binary could cause this.
I would recommend that you simply reinstall, and monitor IOwait/steal either from the beginning or only if the problem reoccurs.

And I'll tag @host_c, could also be an issue with a specific qemu version or something, maybe it's a known problem.

PineappleM · April 2025

@wadhah said:

@PineappleM said: I'm going to guess no since I didn't explicitly set the TCP algorithm to anything. Any way I can check for that?

>

sysctl net.ipv4.tcp_congestion_control

The value is cubic.

@somik said:
Have you contacted your provider? Open a support ticket with them and ask? It could be due to someone else abusing the server or the node being oversold and being out of resources.

@lukast__ said:
Your YABS looks fine.
I would recommend that you simply reinstall, and monitor IOwait/steal either from the beginning or only if the problem reoccurs.

And I'll tag @host_c, could also be an issue with a specific qemu version or something, maybe it's a known problem.

Thank you both for your comments! I don't like asking providers for tech support because support time is expensive, and I like to be a good quiet customer who simply pays the invoice on time and doesn't cause problems for the provider (i.e. be an easy money customer).

I will do a little more plumbing myself and will contact Host-C after I feel I put a good faith effort into addressing the issue so they have as much detail as possible. Of course, I'm still happy to take any other suggestions.

barbaros · April 2025

@PineappleM said:

@wadhah said:

@PineappleM said: I'm going to guess no since I didn't explicitly set the TCP algorithm to anything. Any way I can check for that?

>

sysctl net.ipv4.tcp_congestion_control

The value is cubic.

@somik said:
Have you contacted your provider? Open a support ticket with them and ask? It could be due to someone else abusing the server or the node being oversold and being out of resources.

@lukast__ said:
Your YABS looks fine.
I would recommend that you simply reinstall, and monitor IOwait/steal either from the beginning or only if the problem reoccurs.

And I'll tag @host_c, could also be an issue with a specific qemu version or something, maybe it's a known problem.

Thank you both for your comments! I don't like asking providers for tech support because support time is expensive, and I like to be a good quiet customer who simply pays the invoice on time and doesn't cause problems for the provider (i.e. be an easy money customer).

I will do a little more plumbing myself and will contact Host-C after I feel I put a good faith effort into addressing the issue so they have as much detail as possible. Of course, I'm still happy to take any other suggestions.

Think it this way, if you can tell them specific date and time, they can check things on their site. And on the long run, throttling or getting rid of a resource abuser neighbour would benefit to both provider and customers.

But if they say all is good, then you can keep doing your own research. If you are not monitoring the server, try something like Hetrixtools.

Also a slight chance that there might be issues between you and your server (or vice versa) so doing MTR would be also beneficial.

artxs · April 2025

I have storage server dedis and when the hard drive starts going bad, the CPU locks when it tries to access hard drive with hardware failures and I/O wait goes up artificially. I'd imagine this is whats happening to your node. the I/O wait is high so less CPU for VMs.

wadhah · April 2025

@PineappleM said: The value is cubic.

>

You can change it to bbr using these commands, run each line separately

sudo sh -c 'echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf'
sudo sh -c 'echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf'
sudo sysctl -p

Then verify it's bbr by running

sysctl net.ipv4.tcp_congestion_control

relogin to your server, a full restart might be even better.

note: bbr will only help it the problem is with the network of your server. It will not help with anything else.

layer7 · April 2025

Hi,

next time if you experience this, check with netstat or something similar the number of active/open/closed connections.

If your hardware is fine ( disk IO good, CPU good, RAM good ) and its still slow, then its coming usually from the network ( not too much left anyway ). And if your throughput in the iperf looks fine and maybe your ping also looks fine ( i assume that for now ) then one thing could be that you have too many open connections running that your system has to handle.

That does usually not happen, as your system will automatically close them. But who knows, better check it to be sure....

Dedihoster · April 2025

Could be some slow disk or filesystem issues creeping up. Swap file + reinstall with a fresh Debian image should make a big difference!

ole_hansen · April 2025

Have you enabled error log in nginx and if whats is the latest entry ?

emperor · April 2025

Monitor IO and steal like @lukast__ said, probably someone is hammering the node. Best is to open ticket. IF you or other wont report it can go on for days till provider take action. Also its better not to use provider OS, install fresh from iso or netboot

Howdy, Stranger!

Categories

In this Discussion

Strange slowdown of storage server

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Strange slowdown of storage server

Comments