New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HostHatch Singapore outage again? Anyone knows what's going on?
Hi everyone,
It looks like my HostHatch VPS in the Singapore location is down again.
I'm unable to SSH into the server, and even the console shows multiple kernel "hung task" messages such as:
INFO: task nginx blocked for more than 122 seconds
INFO: task kworker blocked for more than 122 seconds
systemd-timesyncd.service: Watchdog timeout
I'm not sure whether this is caused by a storage issue, a host node problem, or something else on the infrastructure side.
Is anyone else with HostHatch Singapore experiencing the same issue today?
Has HostHatch posted any status update or explained what's causing these recurring outages?
I'd appreciate it if anyone could share their experience or any information.
Thanks!
Comments
Did try rebooting?
I tried, but it didn't work.
I wrote the ticket.
My hosthatch SG has 100% uptime for the last 30 days. Could be a node issue.
Do they have a "power off" type function to hard shutdown? What does it do?
I can see three buttons in the control panel: Boot, Reboot, and Power Off, but none of them are working.
The current status of my VPS is:
Status: Shutting down...
The status has been stuck on "Shutting down...", and I'm also unable to access the server via SSH.
Last year I faced the same in my Japan server. I tried vmc a few times. then tried to reinstall the OS multiple times. One time, resintall work and VPS became responsive again.
We had a node reboot today in Singapore, and it took us a few hours before we responded to it - since our monitoring showed no issues, which is something we definitely need to improve as part of our monitoring processes.
In the past two months, we've dealt with multiple node reboot issues (one node in LA, one in Amsterdam, one in Hong Kong). All of them are now resolved, and were caused due to a AMD Zen3 bug, that is not very well documented (the whole internet has exactly one thread on the exact issue), so it took a lot of trial and error, and troubleshooting to get to the proper fix of that.
As for the better monitoring, we will be pushing out the changes this week (not soon™), so we can respond to similar issues within minutes, rather than the number of hours that it took today before we responded to the problem. You should definitely hold us to a higher standard, and I am sorry that we failed to be up to the mark.
I would like to add that this was a single node that were affected. "HostHatch Singapore outage" is multiple full racks worth of hardware and thousands of customers, and is not a fair description of the issue.
I just received an email about SG node reboot but the server is accessible now.
Except for IPv6 on the E5 NVMe Amsterdam, which went down after the host node outage two weeks ago and never came back. 🙂