New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HostHatch Chicago data lost ?
This discussion has been closed.
Comments
I'm cursing myself with this comment but I have two storage servers there: a 3T and a 10T. Both of them have been 100% stable for me. My use case is pretty light though, primarily archive storage and some plex streaming, so I haven't noticed network issues.
I don't have a separate partition for O/S...
Now I am jealous.
ticket replied
Mike W — HostHatch Representative
Hello,
Apologies for the delayed response.
We're still investigating the ongoing issue affecting the node STOR4.CHI.
The node had disk errors and was hard rebooted, causing most VMs to have errors when booting up.
We are still doing our best to restore all VMs back to their working state.
If it is possible for you, we would recommend that we spin up a new VM for you with the same specifications and boot up RescueCD on the current/affected VM. You can then move over the data to the new VM. We can also move over your old IP address to the new VM if needed. Please let us know.
I have one Virmach storage VPS too, what my experience is, it's ok, but as you may know, Virmach have a strict resource policy than other providers, so when I use it, I need to take extra precautions to not hit the limit, which sometimes gives a little bit of inconvenience for me.
Performance is average, no downtime till now.
After checking through all data partitions there is much corrupted data.
For others on this node before relying on your data I would suggest you check your data partitions with fsck (ext*) or xfs_repair (xfs) or similar.
Yeah this thing is toast. My btrfs setup has no redundancy or error correction so I don't trust any of this data. I'm going to request a new VM and toss this one.
Here is a scan 17gb out of 6.42tb lol. For anyone who doesn't know the error summary should be "Error summary: no errors found"
It's usually not needed on a VPS if the host node has RAID and a filesystem like zfs or btrfs. Since the host handles it in that case, each individual virtual server shouldn't need to also do it. HostHatch say their storage VPSes have either RAID6 or RAID60 (can't remember which), so I'm not quite sure how the data could get so corrupted. Maybe a bad RAID card?
How strict? in terms of IO, CPU and network usage.
I've never used their VPS before. I ordered a Tokyo storage VPS from them last week.
I've had quite a few times of downtime from hosthatch LA storage server. Not going to renew it anymore.
According to some old emails they replaced the raid card back in August 2021.
The only way I could get BTRFS recovery would have been to split the server in half and ran a raid 1 setup. That would have protected me against bitrot but this problem would have still wiped me out.
I don't really care. I prepared for hardware failures that are out of my control. I have another 10TB instance in LA that mirrors this one so I didn't lose anything. Biggest issue is that this Chicago server was the primary backup server that all my servers backup to. Quick DNS update and a reset of ssh host keys and now LA is the primary backup server. Once this is resolved Ill have Chicago be the backup mirror.
I've got a 4TB storage server pre-ordered with VirMach so I'll have some resiliency between providers but they're both in the LAX Psychz datacenter so I'm sure an earthquake will ruin this plan someday.
I live in the San Francisco Bay Area so I'm not looking forward to 'the big one'
On the positive side, I get 16ms ping to my HostHatch LA servers and haven't had any major issues with them
(other than long-running connections dropping sometimes)
My second storage VPS is with Servarica in Montreal, Canada. 3.5TB for US$84/year. I've also got a lifetime 4TB pcloud account that I
rclonemy important encrypted Borg backups into.https://virmach.com/terms-conditions/
D) High Usage Policy: Any usage by Customer that disrupts the overall performance of our server(s) is not permitted. Customer agrees to operate within the VirMach usage parameters, which are as follows:
High CPU: Customer’s Service cannot burst to 95-100% usage for more than five (5) minutes and cannot average higher than 50% usage within any two (2) hour period. Packages advertised to include dedicated CPU, Services with the high CPU option, and any customized Service plans that include high CPU option may burst to 100% at all times.
High Load: Customer’s Service cannot have a 15-minute load average higher than the number of full logical cores assigned and cannot have a 1-day load average higher than 70% of the number of full logical cores assigned.
High Mail Volume: VirMach reserves the right to block port 25 on Customer’s Service. Customer cannot send more than 100 maximum e-mails per hour, and must maintain a similar average volume of mail on a week-to-week basis—no bursting permitted. VirMach reserves the right to waive this requirement for the purpose of a customized Service plan.
High I/O: Customer’s Service cannot average more than 80 IOPS within any two (2) hour period, cannot burst above 300MB/s disk write average for more than ten (10) minutes, cannot average more than 300 write operations per second for more than 1 hour, and cannot be above 20% average utilization within any six (6) hour period.
High Network Usage: Customer’s Service cannot have more than 50,000 conntrack sessions at any given time, and cannot use more than the allocated bandwidth. Customer understands that the network is shared and utilizing maximum network speed will not always be possible.
Those terms and conditions are for the old servers and were not strictly enforced.
@foitin - I expect the T&C to be updated for the new Ryzen based servers with less restrictions. Of course this is not guaranteed. But IMHO worrying about this is premature. I've always found that if you are reasonable, you will find that VirMach is too. The inverse is also true.
This is really strange, I don’t know how they can lose so much data at rest and still recover some meaningful data. This failure mode should not happen
Okay. Hopefully they can provision 1TB storage VPS in Tokyo.
server down for over 48 hours, still not resume yet...
And their network status page is no longer updated or have I not found the right link?
Same here. I've got a reply from them and requested spinning a new one, but got no response after 20 hours again. It's a 3rd level backup for me, so, the data loss would hurt but not that bad. Though, it might need a day or two to rebuild all the fresh backups again after the new install.
To tell the truth, I'm now thinking of not renewing it this year. It's all fine when the servers are running well. But when the server has problems, Hosthatch's response time is still a bad example of service quality...sigh....
I had a similar issue happen in the last few days bit on the Sydney node to one of my VMs.
Went to restart VM after the 'Dirty Pipe' vulnerability in Ubuntu, on restart VM wouldn't respond, no VNC, ping or SSH - down for 12 hours - then it all came good, didn't get to the bottom of it.
What I'm concerned about is whether the same thing can happen in other locations... It's still unclear as to whether this failure mode (with large chunks of @cablepick's data being corrupted) is isolated to the particular hardware or software configuration used in Chicago, or whether it can potentially happen in their other locations. Seems strange that this'd happen with a RAID setup unless multiple drives failed / encountered corruption at the same time.
Your concern is not misplaced. Two weeks ago, my HostHatch storage VPS in Los Angeles experienced a similar kind of data corruption. I had separate partitions for OS and data (both XFS), and they both went FUBAR (at least beyond xfs_repair). I believe the corruption happened while the VM was online, because cleanly mounted XFS partitions suddenly detected serious data structure corruptions and panicked. To save myself time and energy, I chose not to open any tickets, and simply reinstalled the whole VM.
It is isolated to a single server in Chicago. I have a storage vps in Chicago which has been idling for 1+ year. When I checked, it had 100+ days uptime. Anyway, when you go for low end storage vps like these which are dirt cheap, the cost savings should easily justify making backups of backups.
Fixed that for you. Since the cause isn't yet known we can't know whether it might also be an issue for other locations which is what Daniel was talking about.
See many players lose data, be careful to buy
@hosthatch or @Emil Do you have any updates as to what happened to STOR4.CHI and if it's an isolated case?
I've got backups so I can restore everything in the case of any issues, but I also have close to 5TB of data on my VPS, and restoring 5TB of stuff over the internet is not fun.
I don't think they have proactive monitoring. I can say LSI/Broadcom absolutely sucks for that and most people script something up to grab status weekly. Also, they likely don't have background consistency checks scheduled since performance takes a hit during.
Server is still down. No updates from support.
same here as well, "invalid arch-independent elf magic"
I woke up to a new $80 overdue bill from HostHatch, My storage service was not due to expire until November 2022.
A ticket was open by them, as I did not open a ticket for this issue, stating.
Anybody else get billed?
EDIT: Nice, the new server has only less then half the RAM as the old server, (1.8GB verse 3.9GB).
Also for those of you who had account credit or a c/c on file, I expect they took your account credit, as they did mine, and paid or partially paid the new two year bill.
I get a bill too but without any ticket or explanation. It's for a new VM for another two years. Not going to pay it rn.