New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Oh man, good luck, hope it works out. Tha word corruption is my greatest nightmare! Thankfully my single project at the moment does both back ups and replication - hope it works as intended!
This is what I was able to grab... Any suggestions/ help to fix? Thanks
For some reason the connection between HostHatch and Servarica (where one of my backups is stored) caps out at ~90Mbps even though it gets ~600Mbps from another server in LA (blaming Psychz for that), plus HostHatch's disks are a bit slow on this server, so this restore is taking a while. I started restoring ~1TB of files a few hours ago and it's 20% complete so far.
Most files I've looked at seem fine though, so the restore is just in case some random files on disk got corrupted and I didn't notice.
Are you booting into the system or into a live CD?
I'm not familiar with XFS but maybe someone else has an idea.
Yeah it's bizarre! I would have checked what's in the systemd journal at that point, but it's also corrupted
I think this is from the live Rescue upon bootup of the vm... Running Centos7 - with FDE
Booting from the OS won't work well if the partition containing
/sbin
and/usr/sbin
(usually a root partition) is corrupted in any way, since some of the mount/fsck tools may themselves be corrupted. You might need to boot from a CentOS live CD and go into a recovery mode. That's what I did initially - Since I run Debian, I mounted the latest Debian ISO, booted into the rescue mode available in the "advanced options" in the installer boot menu, chose not to mount anything, and ran fsck from the terminal it started.Anyone knows how to fix this error? this happened after shutting down system and booting it up again from website
I will probably ditch my storage VPS with HH and move over to hetzner considering they are far more reliable and transparent. I just wonder why this particular node went down. Was it because of a disk failure, power failure, or what? And how that has affected user data? Shouldnt the host let users know how much data is affected, whether the drives failed, or were replaced etc. I thought the drives were in raid configuration which should help in case of drive failure.
You sure this is from a LiveCD/rescue? This looks like the initrd boot process (mounting root into
/sysroot
thenswitch_root
-ing into it).Try to boot into a LiveCD, unlock the LUKS volume then fsck the XFS volume.
It means your kernel image (
/boot/vmlinuz-*
) is corrupt. See if there's an older kernel available in the GRUB menu. Otherwise you might have to boot into a live CD and copy a clean version of the kernel across. Which distro are you using?Getting this error, and googling it tells me to check the RAM. May this be a host node issue?
@hosthatch Given the number of people reporting corrupted data, can you please clarify what "an issue" refers to in your email (what was the actual issue and how was it resolved), why it had to be a hard reboot, and why it has resulted in data corruption?
Received ~20 hours ago:
I'm thinking something similar right now too. There are too many issues without proper communication.
This sucks, I've been a happy customer of HH for a while and have multiple storage VMs in each location. I wish they'd be more transparent about outages like this, not only does it suck for endusers but it's also terrible PR for them. I still love my services and I got them cheaply enough to where I can just replicate across locations, which is fantastic (and what they recommended doing in another thread). I'm glad the email eventually went out (a bit late), but it's really troubling how they haven't acknowledged the customers who are experiencing corruption. I'll probably still be a happy customer, but at the same time I'll also be on the lookout for storage VMs from other providers.
Are you fucking serious?
Because you guys are fucking defective in terms of network status updates or replying to tickets.
(Never received a ticket update after involucrated Chicago server after initial "we're looking into it" and I've only now seeing a new server provisioned on a new node, but without private network interface).
So, next time you're not sure why such a thread exists, just pull up your Network Status page. Oh wait...
I'll crack the shits if this happens in Sydney. I've been rather content with my 2-yearly deal but it's up for renewal this year and last time I used it I felt like it was very slow.
Something I still don't quite understand is that normally if it was just an unexpected power-off (for example if the host kernel panics or halts), in theory just the files that were in the write buffer/cache at the time should have the potential to get corrupted, if they weren't
fsync
'd to disk yet. Often those files can be repaired using the journal, if it's a journaling file system like ext4.However, I'm seeing corruption in files that haven't been modified in over a year, and other people in this thread have had their Linux kernel image corrupted, which isn't something that's updated that often.
Seems like it's something deeper than just that? Maybe HDD / RAID failure as well...
Modified title based on OP's request
Ooof.
lol
I've got an error on mine as well 'the root filesystem on /dev/vda1 requires a manual fsck' when booting... this does not look promising!
Interesting, @Nekki had rightly predicted that title edits would be reserved for Host Reps and it’s interesting to see it play out.
Restoring from backup (another storage VPS via Borgbackup) is still network IO bound, going at ~90Mb/s so it's taking a loooong time. 45% done so far.
So far so good, but now I'm thinking about what if my backups are corrupt... so I think after this I'm going to set up two different backup systems (eg. borg and duplicacy) to two different servers, as right now I just backup to another storage VPS then sync that backup to "cloud storage" (pCloud).
That's what I saw originally, and I got it into a bootable state by booting off a Debian installer CD, going to advanced → rescue mode and running
fsck.ext4
(e.g.fsck.ext4 /dev/vda1
) in the rescue shell. Of course the command will be different if you use something other than ext4.but I'm not a host rep? lol
I just clicked the flag link and asked a mod to change the title
Well, I take the OP (= @Daniel15 ) to be an upright guy, and I think that he wanted a more accurate/transparent title
(This is not to say that any title-change request by any OP in any thread will be honored)
Their website is still the old one.
That's right. My original thought was that it was some "regular" downtime where everything comes back fine, e.g. the server had to be rebooted, the network went out, etc. Now that I know that it wasn't a regular outage and there's actual data corruption, I wanted the title to be more accurate
Now the title is kinda similar to the post about a similar issue in Chicago (https://lowendtalk.com/discussion/comment/3395419).
Just gonna re-install and not trust previous data state...
I spent most of the day recovering from issues similar to others in the thread I had the xz-compression error. I ended up with 300000 entries in lost+found, which wasn't much fun! Here's a general outline of how I handled it:
Are you serious?
I've received no email or notification of the data corruption event. Had I not seen this thread, I would have continued to assume my backups were safe. SNMP continued to report as okay up until I checked that the filesystem had become read-only. On reboot I received 'the root filesystem on /dev/vda1 requires a manual fsck.'
I've recommended people to HostHatch and have bought many services. Your responses on Chicago and now LA storage have been underwhelming to say the least. I know this is lowend but I've still bought $500+ worth of HostHatch VPS in the last 12months which I'm also happy to spend with other providers I use.
404 - Drama not found.