All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hetzner AX101/AX102 spontaneously reboots
We have had over 70 AX101 servers at Hetzner for a long time, and we are observing spontaneous server reboots. With the AX101 they happened quite rarely. There are no messages in the logs, the server just reboots as if it was powered off and then turned back on.
We recently rented some AX102 servers and they rebooted every 12 hours! It's just unbearable. I contacted support and after diagnostics, they said there was no hardware problems and suggested replacing the server. However, the problem persisted even after replacing the server.
We have our own water cooled 7950X servers with identical settings and they never reboot like Hetzner. I know that Hetzner uses its own BIOS which reduces the performance. Could this be a problem? Or are they overloading the power lines in the rack?
Has anyone else encountered this problem?
Comments
Can you look in the Bios and see if there are any Temperature control settings where you can see if it's getting overheated or what the power output is? also, what are the specs of the server
You probably haven't dealt with Hetzner before, there are very few settings in their BIOS
I can confirm this is happening with my AX101 servers. Some just freeze randomly once in a few months. If this is happening every 12 hours with AX102 for you, you can ask Hetnzer support to run a stress test for 24 hours.
If there is no setup fee, I will rent your server
Hey @hostiko , please check ticket #2535. Thanks.
What distro/version/kernel are you using ? Did you try anything new like 5.10+ ?
I expect these issues to be present with anything RedHat based or older Debian/Ubuntu with older kernels.
I had similar issues with 101 1.5years ago with Arch, not sure what kernel was at that time.
Had to add few options to GRUB CMD which fixed issue:
GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 nomodeset noapic pci=assign-busses apicmaintimer idle=poll reboot=cold,hard"Without knowing anything else, it's likely overheating. One downside of using consumer grade cpu in hosting, you (the host) are on your own in figuring out the heat situation. As opposed to say epyc or xeon which are guaranteed to work given the right chassis + sink.
New user here but I have an AX102 however I have not had this issue myself, have had the CPU at 80% load for over 24 hours, and It's still just chugging away, the only issue I do have is the clock speed which in many different tests I have been unable to get it to go above 4.5, but I also expected that
This is happening with me also , I thought it might be Proxmox issue ,
I am using Proxmox 7.4.3 and there are no logs simply -Reboot- showing in logs.
Ax101 , now waiting for Ax102 for to be delivered will check if the issue persist there also.
We have Few other Ax101 with windows , on Production from year back and those never had issue .
Even a Debian Docker , but specifically Proxmox One is facing issues.
I have a few Ax101 with RockyLinux 8, no problem. Uptime 3 months+
Have had this issue aswell with centos OS, by installing kernel-ml or kernel-lt the issue was fixed. It was mostly kernel panic which freezes up the whole server and only solution was to restart the server to bring it back online...
We use Almalinux 8 and always updating the kernel. Our AX102s are currently running on 6.2.11 and this does not prevent the server from rebooting spontaneously.
No kernel panic
Can't you ask Hetzner for a replacement or something? Someone mentioned it here but they do a small testing and then come back to you. Something like this happened to one mate few months ago but it was actually with an Intel server, the stress test's even did not return anything useful and they kept on saying there's no problem with the server. But at the very end they offered hardware replacements IIRC (one option to keep the drives even, another for a totally fresh server)
They already replaced the server. And that server rebooted an hour ago
. That doesn't solve the problem.
Given that the AX102 have only been available for a week or so, they might be more trusting of reports of problems like this, as it's possible they haven't ironed out all the kinks yet.
rent some 101 in Nuremberg - never seen. what is your dc?
At 120 €/month is it even convenient to go with an AX101 on consumer hardware?
In proportion, I was paying 10 €/day for the 120 cores Epyc on Azure with 512 GB memory on spot. On demand it would be 20 €/day, or 5 €/month per core. The AX101 is 7.5 €/month per core on crap consumer hardware. Let me reiterate: crap consumer hardware, which is the reason why stuff reboots and crashes.
It feels brutally overpriced.
Stick to Intel + ECC if you want Hetzner. They can’t cheap out on motherboards that way
you know, we had a problem with hetzner with bandwidth unlimited != bandwidth unlimited, and i said before that hetzner will become next year (2023) unlimited electricity != unlimited electricity
they are just saving what they can 
liars
Leaving this here for someone's future problems, but it seems to be a different issue from the one that @hostiko has.
We've found that Hetzner's SX134 will randomly lose connectivity when the 10G add-on is ordered with the server. Issue manifests itself when bandwidth is higher than 1Gbps, and may be each other day, or a few days in between. Servers will require reboot to become functional again. Syslog has messages mentioning AMD-Vi followed by a hang of the interface and unsuccessful attempts of the system to restore connectivity. Solution so far seems to be to add the "immou=pt" flag to GRUB to disable AMD's virtualisation of I/O resources. We are 8 days shy of a month with no downtime so far and will be submitting this finding to Hetzner after that.
What's that?
How do you get 120 Epyc cores + 512 GB memory for €600/mo on-demand on Azure? We have a HB120rs v3 at work on Azure (120 Epyc cores, 448GB RAM, 2TB Storage) and even with a 3-year reserve it's €1,200/mo, on-demand would have been €2,400/mo. Are we missing some dramatically cheaper way to get it (not spot)?
It was the price of July 2022 either in Canada Central or US West.
How often roughly does it happen on the AX101? You're scaring me as I'm running some important shit on an AX101 I bought like two weeks ago lol.
You must be insane to think that the AX101 or AX102 is expensive, Hetzner does some of the cheapest servers on the entire market. Any other provider would easily charge 200-250 euro for the same specifications.
They are 2 completely different products, there is absolutely no comparison between a full bare metal server and a spot VM instance that can be taken down at any time. Not to mention, Hetzner includes 1 Gbps Unmetered which would easily cost thousands on Azure, and have you considered the 8TB of dedicated NVMe storage?
No one also wants to pay almost 600 euro for a server, when their use case will likely work on a 100 euro server. Sure, the price per core is cheaper, but AX101 is better performance per core and is also 1/6th the price of that instance (not even a dedicated server) you're mentioning.
Hetzner also sells "proper" servers with server-grade hardware like Intel Xeon Silver/Gold and EPYC, but most people go with the AX101/102 instead because it's very affordable and has high single core results which make it ideal for running primarily single threaded applications.
Can't agree more.
Probably not thousands before Hetzner does a little suspend prank on you though.
A correct term for bandwidth allocation at Hetzner is closer to ”undefined” then it is to ”unlimited”.
Don't worry. I am running a production app in one AX101 with 0 reboots/problems.
@Advin
Hetzner is overpriced for the main reason that they are subject to a German income tax of 45% plus 19% VAT and inane excise duties on electricity. They can only appear to be competitive against international providers by offering "comparable specs", but on fluff consumer hardware that they pay one-fifth of server grade hardware. The result? They crash, and reboot, and SATA hard drives aren't detected, and so on, and it takes days to diagnose and fix problems on consumer hardware. And the 120€ setup fee? Come on. Azure gets even cheaper with pre-paid plans.
Regarding storage and bandwidth you may have a better argument, I haven't considered that, because personally I've only needed cpu-performance, so that's what I look for.