New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
It sounds like the affected customers weren't notified at all?
I'd agree with limiting certain details until you've reached a resolution or fully know the impact; but if you need to hard bounce a server I'd recommend sending an email out to let people know there was an outage and a hard reboot. In my experience the thing that annoys people the most is being completely in the dark.
Servarica's architecture is entirely different. They have all the storage in a SAN (Storage Area Network) rather than in the servers themselves, and likely a separate dedicated network just for the storage. This can have disadvantages - in theory it's a bit slower since reads/writes have to go over a network of some sort, but in practice Servarica has unusually high fio 4k and 64k read/write speed and IOPs so it actually seems faster than some other storage VPSes ¯\_(ツ)_/¯
BuyVM's storage slabs and Letbox's block storage are similar.
**Hi, So, I tried the same. Booted from Centos live CD, set my Luks password, tried the rescue option and I cant seem to find the disks to run fsck / xfs_repair... Any pointers? Thanks
**
Boot Rescue
Start Rescue?
Luks Passphrase:
fdisk status:
I see it as part of a 3-2-1 backup policy. The three copies of important files are on my PC, on a storage VPS, and in "the cloud". Essentially instead of having two copies locally and one offsite (like traditional 3-2-1), I have one copy locally and two offsite. I have some important file in Seafile, and I've configured the Seadrive client to always have those files available locally on my PC.
You should have backups of files "in the cloud" too - That means either having two clouds (e.g. Google and Backblaze), or having a copy on DVD/BluRay, or USB HDD stored somewhere secure, whatever.
What hard drive models were these? Sounds like consumer drives and not RAID supported ones. And also that your RAID cards are failing, possibly from heat. Sure would be good to know why the RAID kicked them out, should say in the logs. Since it kicks out good drives and has corruption despite ECC memory, your RAID cards are very much suspect. The lack of root cause analysis is an issue.
Correct. I immediately opened a ticket on March 19 and got the response:
And no further ticket responses or status updates.
Stop using that dogshit fucking troubleshooter, grab a shell and do it yourself.
It's literally just
cryptsetup open /dev/vda3 my-root, inspect the unlocked container, then diagnose from there.If he was doing full disk encryption then he'd have backups, isn't just easier and faster to just restore from backups?
Who has time to check and hope there isn't hidden corruption instead of reliable restoration?
Ofc, but considering he is on this thread asking then I think you already know why
To be fair, at their price point I don't expect everything to be enterprise-grade hardware.
Although I also don't expect so much data to become corrupted, even on non-enterprise hardware. This is going to take me at least a week to fully restore things I think. I've restored the backup of files on the server itself, now I have to verify all backups that are stored on the server, and finally re-enable all the backups and cronjobs (I turned them all off as soon as this issue happened).
I have a few other servers that back up to this server using borgbackup, and then those backups are then copied elsewhere for redundancy. Eventually I want to set up two entirely separate backups using two different apps across all my servers, but I've only done that for one server so far.
Anyways, since I have the "good" copy of the backups, I can copy those back to this server to revert them to a known good state. I'm using
rsync -avc, the-coption meaning it performs a checksum of every file on both sides and uses that to determine if the file is different, rather than just using size and update time. I've done this for one backup so far, and it ended up having to transfer ~1.6GB out of ~103GB, meaning around 1.5% of the backup was corrupt (actually slightly less since that number includes rsync's metadata including all the checksums, but metadata would be <100MB total)You don't run hardware RAID without RAID supported drives. That's been well known for a decade when they began differentiating desktop and NAS drives. I'm not talking about the excessively priced Dell drives locked down to PERC models, I'm talking Red brand and NAS designed drives.
I'm in the process of upgrading my home machine, which included a 10.4TB backup and the f'n restore started at 1.2Gbps and dropped to 200Mbps. It's mostly from the 8x10TB RAID 10 initializing while restoring the 10TB. So much for having 10Gbe NICs.
Both the HostHatch disasters and my home upgrade really pushing towards setting up a deal with a local friend to host each others backups so I can drive and pick up the server and bring home if needed.
The thing that stops me from setting up a NAS or something similar at home is that I'm in California and electricity is very expensive. For me, the monthly cost of the HostHatch 10TB VPS ($240 / 2 years = $10 / month) is cheaper than just the electricity to run a PC/NAS with decent CPU power 24/7. I'm also limited to 40Mbps upload (thanks, Comcast) and there's no other internet options in my apartment building
@hosthatch said:
(Cleared to reduce the response)
Do you really care if the door hits him on the way out? - how you answered your current customer seems rude (that statement is usually used in my country when someone is trying to be rude not sure about yours). B
You seem skewed to thinking your statistics are good since its only 5% of customers in that location affected but you are missing the main point that some of your current customers are now unhappy AND someone in your company knew that the disks were hopping off the raid a little after a year (which sounds like something you all knew about? And you guys are responsible since you knew of the possibilities).
Are you waiting for this kind of setup to continue to corrupt other customers data? Hopefully you have an action plan to avoid this right like getting rid of that old config and provisioning new better designed storage for your old loyal customers? I mean should I wait till next year when I might be the unlucky one to lose data from my LA vps with you?
RE Hosthatch wondering why a thread was made:
Your 'customer service' is horrid, I have a ticket over 1 month old now with no response about your broken panel preventing actions on a service. So this service cannot be used at present and the ticket is over a month old, what are you doing?
But 40Mbps upload is enough - NBN said so
NBN is a laugh.
Ok. This first step worked.. I was able to unlock my luks container.
I tried xfs_repair and its throwing some other errors and asking me to mount the disk... i tried that too but get another error - "Structure needs cleaning". I need help.. Please.
At this point there is zero chance you can recover this to a working and trustworthy system. The best you can hope for is getting some files off and using them to start over on a fresh install
Take a backup of the raw block device over ssh, as the next step may will destroy it further. Then, pass
-Lparameter to xfs_repair as it suggestsIf there is some reason you can't take a backup, at a pinch you could use LVM snapshots (delete the swap LV if there is no free space)
Whoosh. When you don't fix problems in any sort of reasonable time frame the average user would have resolved the issue and you haven't, it's fucking natural to try and help you fix our shit faster.
This doesn't make sense, btw. If disks were kicked out and went read-only (they'd be kicked out for I/O errors and then I/O errors stop, not the other way around as you say), they wouldn't be corrupt AF. Data that wasn't changed in a year was corrupted. That's bad writes right there. Like 5% of data or more.
So I'm not sure if you truly understand you know the problem. It clearly didn't immediately detect catastrophic I/O errors and you definitely had extensive bad writes instead of your read-only belief.
Are we going to find out someone actually set these up in RAID 0 and not RAID 10? That actually fits the symptoms more than battery backed hardware controller that isn't considered defective and needs to be replaced having momentary fuck ups randomly after a year.
So I finally restored most of the data on my affected VPS. I had already restored a backup of the most important files on the VPS, but today I was restoring the backups from other systems that are stored on this one, from a mirror stored on another server. I went through every backup that's stored on this server and used
borg checkto verify they're no longer corrupted. For some of the data, I only have a second copy of it locally (not "in the cloud") so I'll have to upload those again over my relatively slow 40Mbps upload (thanks, Comcast).Overall, some directories had no corrupted files at all, whereas others had as many as 5-10% of files corrupted. In total, maybe 4-5% of the files were corrupted. Maybe it's based on where the files are located on the physical disks.
@Daniel15 - grab more storage offers, from different providers, so you can copy the data from one server to another when a provider gets involucrated. I also have 40mbps bandwidth at home, and I use multiple servers with encrypted data on different providers.
I try to stay away from cloud, due to privacy concerns. On VPS I have more control over what is encrypted and how.
This is what I fear. Saving bucks in setting them up for raid0 and letting non-existent RAID cards take the blame for 01 whole year, I really hope thats not the case.
I've got other storage, however I'm making extensive use of HostHatch's private network to access the storage from some of their fast NVMe servers in the same location. That makes it more difficult to switch to another provider in case of issues.
The only thing I'm using the public cloud for is a mirror of encrypted Borgbackup backups.
The private net bit is fantastic, really wish more providers would start doing this.
Servarica can add a private network on request, they just don't advertise it. Their bandwidth billing is done on their edge routers meaning any internal traffic (between any Servarica VPSes) is free, which removes one of the use cases for a private network (avoiding fees for transferring a lot of data between VPSes at the same provider).
As far as I know, they own the entire data center, so they have more freedom in terms of what they can do. They also have fully redundant systems, AFAIK also including a warm spare for failover of their storage.
The only reason I don't use them more is because the speeds between me near San Francisco and their data center in Montreal are not great. HostHatch's Los Angeles location works well enough (as well as Psychz can work) with low ping times, and I've mostly been happy with it.
Interesting to know, I'll have to ticket and ask.
They don't own the DC, but they do have a private cage. They do own all their hardware to my knowledge. Their actual infra is really good, probably the best setup I've seen out of a LET provider.
My LA has been unavailable for more than three days
Thanks for quoting 60 lines of text for a 1 sentance reply
Seeing as how you spent zero time "sharing information as compared to actually solving it", I guess you see zero importance in timely customer notification because
I guess this part says it all:
5 users at a location? Okay, I can agree. But 5%?
You don't need to write up a 10 paragraph status notification for these things until you've tried to solve an issue and learned that you can't, at which point the customers deserve one since they'll have to clean up after your own failures (which you ended up doing here, I'm not criticizing that). A simple one-liner in your status page saying "We are investigating issues with some storage nodes on XXX" within 15 minutes of seeing the issue helps a lot. Then another one-liner update a few hours later. It takes 30 seconds, which is probably less time than you spend trying "to decide how much time we should spend sharing information....". Simply refusing to communicate will just bring frustrated people to LET since your support seems to be a big cobwebbed black hole.
UPDATE: Examples from RamNode
https://twitter.com/NodeStatus
http://status.ramnode.com
Genuinely amazed there's no status page for HH yet.