New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hetzner AX52 constantly crashing
Hello LET,
Recently we migrated some clients to AX52s. One of the hypervisors have been constantly crashing with no indication to why.
We contacted Hetzner on the 10th regarding this, with logs of PCIE errors and they replaced the server. This did not resolve it, they replaced it again. Still nothing.
So we had them replace the drives too, still nothing.
They're now telling us that it's our issue, they don't support custom systems/installations, and we should use the AX101 instead which we migrated from.
These are logs during the crash, nothing.
Is anyone else having similar issues?
Comments
What motherboard? Did you upgrade to ECC?
We've tried all of them, but at the minute it's B650M-PLUS. We're not using ECC, although we do have other hypervisors running fine without it.
my vps every day is down. can i request refund?
,,😅🤭
hostaris services is very bad. sadly
Do you have the same or similar problems on AX102 or just AX52? Generally our 7000 series servers have been fine, including one Hetzner AX102 which we have, but we've had some instability with AM5 on a different provider before (that provider used ASRockRack or Gigabyte boards and we had the same problem where it would randomly crash with 0 indication why in the logs). AM5 is still a very new platform and we've noticed some issues with it, AM4 is way more mature.
what no ECC?
What OS you using? Try with different OS and see if you face similar issues. In the past we had issue with server and it was related OS and network port/adapter.
Both times I've had instability with Hetzner servers it was due to faulty RAM. But if you mean they replaced the server several times then that is likely not the case.
Do you have any pre crash logs you could look into? That could tell you what is going on pre crash if it is software related.
I had the same problems with there AX101.
I applied the GRUB fixes described here, https://lowendtalk.com/discussion/185740/hetzner-ax101-ax102-spontaneously-reboots#latest
It solved my problems.
We haven't tried 102 yet.
Nope. We did find the following once but this has not been repeated and has only happened once, all the other crashes showed nothing. At the bottom of the logs there is 7 minutes undocumented for which is the crash.
We have many AM5 servers online with IPMI boards (Gigabyte, Asrock etc) and have only had this problem with defective hardware, so either CPU (most common) or defective motherboard. In fact, we had more problems with AM4.
DM me your IP, I will migrate you to a stable machine.
NOT SURE IF WORKS FOR YOU TRY TO UPDATE graphics driver OR DEGRADE IT.
DM sent...
hmm... lots of fails at the moment...
speed within frankfurt... aweful!
---------------------------- network-speed.xyz ---------------------------- A simple script to test network performance using speedtest-cli --------------------------------------------------------------------------- Version : v2023.09.30 Global Speedtest : wget -qO- network-speed.xyz | bash Region Speedtest : wget -qO- network-speed.xyz | bash -s -- -r --------------------------------------------------------------------------- Basic System Info --------------------------------------------------------------------------- CPU Model : AMD Ryzen 7 7700 8-Core Processor CPU Cores : 1 @ 3792.860 MHz CPU Cache : 1024 KB AES-NI : ✔ Enabled VM-x/AMD-V : ✔ Enabled Total Disk : 78.7 GB (7.1 GB Used) Total RAM : 3.8 GB (354.5 MB Used) System uptime : 4 days, 19 hour 25 min Load average : 0.00, 0.00, 0.01 OS : Debian GNU/Linux 12 Arch : x86_64 (64 Bit) Kernel : 6.1.0-13-amd64 Virtualization : KVM --------------------------------------------------------------------------- Basic Network Info --------------------------------------------------------------------------- IPv6 Access : ❌ Offline IPv4 Access : ✔ Online ISP : Hostaris Limited ASN : AS199765 Hostaris Limited Host : Hostaris Limited Location : Frankfurt am Main, Hesse-HE, Germany --------------------------------------------------------------------------- Speedtest.net (Region: EUROPE) --------------------------------------------------------------------------- Location Latency Loss DL Speed UP Speed Server ISP: Hostaris Limited Nearest 5.24 ms N/A 907.74 Mbps 896.71 Mbps Netprotect - Frankfurt London, UK 16.97 ms 0.0% 847.01 Mbps 895.83 Mbps M247 Ltd - London Edinburgh, UK 29.37 ms N/A 899.35 Mbps 889.88 Mbps Vodafone UK - Edinburgh Dublin, IE 37.94 ms 0.0% 919.81 Mbps 882.26 Mbps Three Ireland - Dublin Amsterdam, NL FAILED Dronten, NL 14.02 ms 0.0% 910.06 Mbps 903.03 Mbps Serverius Connectivity - Amsterdam Paris, FR 16.81 ms 0.0% 921.35 Mbps 891.65 Mbps ORANGE FRANCE - Paris Marseille, FR 20.97 ms 0.0% 910.47 Mbps 874.86 Mbps GSL Networks - Marseille Madrid, ES 35.15 ms 0.0% 916.49 Mbps 815.88 Mbps Orange - Jazztel - Madrid Barcelona, ES 32.63 ms 0.3% 804.73 Mbps 833.67 Mbps Adamo - Barcelona Lisbon, PT 40.38 ms 0.0% 749.47 Mbps 836.90 Mbps Edgoo Networks - Lisbon Rome, IT FAILED Milan, IT FAILED Zurich, CH FAILED Frankfurt, DE 5.79 ms 0.0% 7.03 Mbps 41.21 Mbps 23M GmbH - Frankfurt am Main Berlin, DE FAILED Vienna, AT FAILED Budapest, HU FAILED Gdansk, PL FAILED Warsaw, PL FAILED Lviv, UA FAILED Kyiv, UA FAILED Minsk, BY FAILED Bucharest, RO 33.60 ms 0.0% 8.04 Mbps 82.28 Mbps Orange Romania Communications - Bucharest Iasi, RO FAILED Helsinki, FI FAILED Stockholm, SE FAILED Oslo, NO FAILED Moscow, RU FAILED Petersburg, RU FAILED Istanbul, TR FAILED Tbilisi, GE FAILED --------------------------------------------------------------------------- Avg DL Speed : 733.46 Mbps Avg UL Speed : 737.01 Mbps Total DL Data : 11.30 GB Total UL Data : 9.95 GB Total Data : 21.25 GB --------------------------------------------------------------------------- Duration : 8 min 29 sec System Time : 15/10/2023 - 16:00:56 CEST Total Script Runs : 25290 ---------------------------------------------------------------------------To be fair, it's 700% faster than "failed"
That shouldn't be happening - no one else has reported issues like this. I will look into it.
You're not actually on the unstable node either.
Running this myself and so far I'm having no issues. The no IPv6 does stand out though, perhaps for whatever reason your V6 isn't working and some of the speedtests are trying to use V6?
next try, very good:
---------------------------- network-speed.xyz ---------------------------- A simple script to test network performance using speedtest-cli --------------------------------------------------------------------------- Version : v2023.09.30 Global Speedtest : wget -qO- network-speed.xyz | bash Region Speedtest : wget -qO- network-speed.xyz | bash -s -- -r --------------------------------------------------------------------------- Basic System Info --------------------------------------------------------------------------- CPU Model : AMD Ryzen 7 7700 8-Core Processor CPU Cores : 1 @ 3792.860 MHz CPU Cache : 1024 KB AES-NI : ✔ Enabled VM-x/AMD-V : ✔ Enabled Total Disk : 78.7 GB (7.1 GB Used) Total RAM : 3.8 GB (365.6 MB Used) System uptime : 4 days, 20 hour 39 min Load average : 0.00, 0.00, 0.00 OS : Debian GNU/Linux 12 Arch : x86_64 (64 Bit) Kernel : 6.1.0-13-amd64 Virtualization : KVM --------------------------------------------------------------------------- Basic Network Info --------------------------------------------------------------------------- IPv6 Access : ❌ Offline IPv4 Access : ✔ Online ISP : Hostaris Limited ASN : AS199765 Hostaris Limited Host : Hostaris Limited Location : Frankfurt am Main, Hesse-HE, Germany --------------------------------------------------------------------------- Speedtest.net (Region: EUROPE) --------------------------------------------------------------------------- Location Latency Loss DL Speed UP Speed Server ISP: Hostaris Limited Nearest 5.23 ms N/A 918.51 Mbps 906.10 Mbps Netprotect - Frankfurt London, UK 16.93 ms 0.9% 867.07 Mbps 880.41 Mbps M247 Ltd - London Edinburgh, UK 29.36 ms N/A 801.24 Mbps 875.16 Mbps Vodafone UK - Edinburgh Dublin, IE 36.06 ms 0.0% 923.62 Mbps 736.28 Mbps Three Ireland - Dublin Amsterdam, NL FAILED Dronten, NL 14.69 ms 0.0% 889.64 Mbps 859.34 Mbps Serverius Connectivity - Amsterdam Paris, FR 21.45 ms 0.0% 890.01 Mbps 780.90 Mbps ORANGE FRANCE - Paris Marseille, FR 20.81 ms 0.0% 873.16 Mbps 791.47 Mbps GSL Networks - Marseille Madrid, ES 33.42 ms 0.0% 889.73 Mbps 773.42 Mbps Orange - Jazztel - Madrid Barcelona, ES 35.84 ms 0.0% 578.20 Mbps 478.10 Mbps Adamo - Barcelona Lisbon, PT 49.04 ms 0.0% 190.25 Mbps 504.39 Mbps Edgoo Networks - Lisbon Rome, IT 27.34 ms 0.0% 922.19 Mbps 841.67 Mbps TIM SpA - Rome Milan, IT 14.90 ms 0.0% 874.37 Mbps 859.05 Mbps Fastweb SpA - Milan Zurich, CH 12.11 ms 0.0% 868.29 Mbps 822.74 Mbps Sunrise Communication AG - Zurich Frankfurt, DE 11.83 ms 0.7% 885.38 Mbps 890.02 Mbps 23M GmbH - Frankfurt am Main Berlin, DE 15.76 ms N/A 896.70 Mbps 631.07 Mbps Misaka Network, Inc. - Berlin Vienna, AT 22.24 ms 0.0% 894.96 Mbps 845.02 Mbps A1 Telekom Austria AG - Vienna Budapest, HU 33.17 ms 0.0% 667.16 Mbps 537.23 Mbps ATW Internet Kft. - Budapest Gdansk, PL 24.89 ms 0.0% 593.41 Mbps 753.74 Mbps T-Mobile Polska S.A. - Gdańsk Warsaw, PL 33.76 ms 0.0% 770.23 Mbps 778.40 Mbps Orange Polska S.A. - Warsaw Lviv, UA 35.03 ms 0.0% 883.89 Mbps 768.90 Mbps Kyivstar - Lviv Kyiv, UA 31.81 ms 0.0% 862.56 Mbps 813.99 Mbps o3 - Kyiv Minsk, BY 60.06 ms 0.0% 911.58 Mbps 444.42 Mbps A1 - Minsk Bucharest, RO 37.47 ms 0.0% 497.36 Mbps 382.43 Mbps Orange Romania Communications - Bucharest Iasi, RO 34.51 ms 1.3% 313.76 Mbps 41.73 Mbps Digi Helsinki, FI 34.34 ms 0.0% 810.25 Mbps 632.56 Mbps Elisa Oyj - Helsinki Stockholm, SE 36.65 ms 0.0% 833.38 Mbps 621.15 Mbps i3D.net - Stockholm Oslo, NO 31.36 ms 0.0% 880.12 Mbps 880.21 Mbps Speedtest.net - Oslo Moscow, RU 39.28 ms 0.0% 844.97 Mbps 727.64 Mbps Rostelecom - Moscow Petersburg, RU 36.08 ms 0.0% 718.57 Mbps 739.43 Mbps MTS OJSC - Saint Petersburg Istanbul, TR 39.99 ms 0.0% 877.30 Mbps 843.55 Mbps Turkcell - Istanbul Tbilisi, GE 63.54 ms N/A 742.13 Mbps 718.21 Mbps Silknet JSC - Tbilisi --------------------------------------------------------------------------- Avg DL Speed : 786.14 Mbps Avg UL Speed : 714.79 Mbps Total DL Data : 31.19 GB Total UL Data : 28.11 GB Total Data : 59.30 GB --------------------------------------------------------------------------- Duration : 15 min 44 sec System Time : 15/10/2023 - 17:21:40 CEST Total Script Runs : 25301 ---------------------------------------------------------------------------Wouldn’t be comfy with that in a home server let alone datacentre
Yeah, I don't really know what happened. We also built our own AM5 servers and they have all been fine/stable so maybe it was a configuration issue with the provider we used to use, or maybe it was some issue with Proxmox/Debian 11 back then. They tried replacing the CPU/motherboard/etc but the problem never got fixed.
Are there desktop motherboard that don't look like a Steven Spielberg alien spaceship? They are cool ok, but suppose for a minute that I don't want my gf to think I'm a total wally just because I keep a spaceship with LEDs on my desk. Maybe I should just tell her that it's not a spaceship but a serious and sober """""server""""".
It was caused by Datapacket's DDoS protection, we've sorted it now anyhow.
root@vps3:~# curl -sL yabs.sh | bash
## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ##
Yet-Another-Bench-Script
v2023-09-06
https://github.com/masonr/yet-another-bench-script
## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ##
Mon Oct 16 06:46:14 PM WIB 2023
Basic System Information:
Uptime : 0 days, 13 hours, 34 minutes
Processor : AMD Ryzen 7 7700 8-Core Processor
CPU cores : 1 @ 3792.892 MHz
AES-NI : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM : 1.9 GiB
Swap : 0.0 KiB
Disk : 39.9 GiB
Distro : Debian GNU/Linux 12 (bookworm)
Kernel : 6.1.0-9-amd64
VM Type : KVM
IPv4/IPv6 : ✔ Online / ✔ Online
IPv6 Network Information:
ISP : Hostaris Limited
ASN : AS199765 Hostaris Limited
Host : Hostaris
Location : Frankfurt am Main, Hesse (HE)
Country : Germany
Running fio random mixed R+W disk test with 4k block size.Running fio random mixed R+W disk test with 64k block sizeRunning fio random mixed R+W disk test with 512k block sizRunning fio random mixed R+W disk test with 1m block size.fio Disk Speed Tests (Mixed R/W 50/50):
iperf3 Network Speed Tests (IPv4):
Provider | Location (Link) | Send Speed
| Recv Speed | Ping
----- | ----- | ----
| ---- | ----
Performing IPv4 iperf3 send test to Clouvider (Attempt #1
Performing IPv4 iperf3 send test to Clouvider (Attempt #2
Performing IPv4 iperf3 recv test from Clouvider (Attempt #Performing IPv4 iperf3 recv test from Clouvider (Attempt #Clouvider | London, UK (10G) | 902 Mbits/sec | 906 Mbits/sec | 18.2 ms
Performing IPv4 iperf3 send test to Scaleway (Attempt #1 oPerforming IPv4 iperf3 send test to Scaleway (Attempt #2 oPerforming IPv4 iperf3 recv test from Scaleway (Attempt #1Performing IPv4 iperf3 recv test from Scaleway (Attempt #2Performing IPv4 iperf3 recv test from Scaleway (Attempt #3Scaleway | Paris, FR (10G) | 897 Mbits/sec | busy | 16.9 ms
Performing IPv4 iperf3 send test to NovoServe (Attempt #1
Performing IPv4 iperf3 recv test from NovoServe (Attempt #NovoServe | North Holland, NL (40G) | 914 Mbits/sec | 915 Mbits/sec | 12.6 ms
Performing IPv4 iperf3 send test to Uztelecom (Attempt #1
Performing IPv4 iperf3 recv test from Uztelecom (Attempt #Uztelecom | Tashkent, UZ (10G) | 843 Mbits/sec | 632 Mbits/sec | 96.6 ms
Performing IPv4 iperf3 send test to Clouvider (Attempt #1
Performing IPv4 iperf3 send test to Clouvider (Attempt #2
Performing IPv4 iperf3 recv test from Clouvider (Attempt #Clouvider | NYC, NY, US (10G) | 829 Mbits/sec | 608 Mbits/sec | 90.4 ms
Performing IPv4 iperf3 send test to Clouvider (Attempt #1
Performing IPv4 iperf3 recv test from Clouvider (Attempt #Performing IPv4 iperf3 recv test from Clouvider (Attempt #Clouvider | Dallas, TX, US (10G) | 570 Mbits/sec | 344 Mbits/sec | 122 ms
Performing IPv4 iperf3 send test to Clouvider (Attempt #1
Performing IPv4 iperf3 recv test from Clouvider (Attempt #Clouvider | Los Angeles, CA, US (10G) | 764 Mbits/sec | 330 Mbits/sec | 144 ms
iperf3 Network Speed Tests (IPv6):
Provider | Location (Link) | Send Speed
| Recv Speed | Ping
----- | ----- | ----
| ---- | ----
Performing IPv6 iperf3 send test to Clouvider (Attempt #1
Performing IPv6 iperf3 recv test from Clouvider (Attempt #Clouvider | London, UK (10G) | 884 Mbits/sec | 891 Mbits/sec | 18.2 ms
Performing IPv6 iperf3 send test to Scaleway (Attempt #1 oPerforming IPv6 iperf3 recv test from Scaleway (Attempt #1Performing IPv6 iperf3 recv test from Scaleway (Attempt #2Performing IPv6 iperf3 recv test from Scaleway (Attempt #3Scaleway | Paris, FR (10G) | 899 Mbits/sec | busy | 19.1 ms
Performing IPv6 iperf3 send test to NovoServe (Attempt #1
Performing IPv6 iperf3 recv test from NovoServe (Attempt #NovoServe | North Holland, NL (40G) | 900 Mbits/sec | 899 Mbits/sec | 12.8 ms
Performing IPv6 iperf3 send test to Uztelecom (Attempt #1
Performing IPv6 iperf3 recv test from Uztelecom (Attempt #Uztelecom | Tashkent, UZ (10G) | 807 Mbits/sec | 490 Mbits/sec | 96.8 ms
Performing IPv6 iperf3 send test to Clouvider (Attempt #1
Performing IPv6 iperf3 recv test from Clouvider (Attempt #Clouvider | NYC, NY, US (10G) | 828 Mbits/sec | 494 Mbits/sec | 90.4 ms
Performing IPv6 iperf3 send test to Clouvider (Attempt #1
Performing IPv6 iperf3 recv test from Clouvider (Attempt #Clouvider | Dallas, TX, US (10G) | 779 Mbits/sec | 267 Mbits/sec | 122 ms
Performing IPv6 iperf3 send test to Clouvider (Attempt #1
Performing IPv6 iperf3 recv test from Clouvider (Attempt #Clouvider | Los Angeles, CA, US (10G) | 751 Mbits/sec | 362 Mbits/sec | 144 ms
Geekbench 6 Benchmark Test:
Test | Value
|
Single Core | 2226
Multi Core | 2228
Full Test | https://browser.geekbench.com/v6/cpu/3108474
YABS completed in 12 min 48 sec
root@vps3:~#
thanks 👍