New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Filesystem corruption is a serious issue. You can have some pretty bulletproof setups (ZFS can be really good) but there's no guarantees. Backups are always the responsibility of the customer at the end of the day. A provider offers convenience unless they explicitly offer guarantees. And even then, your data is probably worth more than what you're paying the provider.
As part of disaster recovery, you need a way to verify that your data is correct. You can't assume that there's been no bit rot or other silent corruption. Ideally there isn't, but is it worth gambling with your data that way?
In developing Slow Servers, I ended up mounting my host's partitions with sync enabled. This slows down disk a lot, even with SSD, but if there is a power cut or a crash, it significantly reduces the chances of corruption. Another bonus is that it's less likely for the fsck to fail and the host to never come back. I don't like driving almost 2 hours (one way) out of the blue to deal with that! Thankfully, I haven't yet.
RAID 1 is interesting because it looks better on paper than it is in reality. Under OpenBSD, I found (with 512e/4K sector drives anyway) that it made filesystem corruption considerably more likely than without. And while drive failures definitely happen, power cuts and crashes do as well -- perhaps more frequently in some cases.
One tricky element to RAID 1 is that you don't know which copy of the data is correct. Now sometimes one drive will know that a sector is bad and report that, and you'll just get the good sector. That isn't guanteed all of the time, however, and if there was a power cut during writes, each drive may have two "valid" blocks that are different.
Now let's say you replace one drive but you're missing sectors. Now you're missing data. The only way to make RAID 1 fairly bulletproof is to have it act like a democracy where the majority wins. This forces you to have 3 or 4 drives in a pool, too. And it removes all of the read speed benefits of RAID 1. RAID Z1 or Z2 is a much nicer setup for most people. But OpenBSD doesn't have ZFS, so it's a no-go for me.
I worry a bit about sparse qcow files when crashes or corruption happens. It seems like there's more failure mechanisms. I ended up fully preallocating raw files for drives with Slow Servers. It's slow and hard on the drives, but seems like the safest method involving filesystems on filesystems. Raw block devices are obviously better on the whole. At a previous employer we were using LVM and it worked pretty well on the whole. You can do that with ZFS as well.
Anyway, this is a lot of tangents. I know it's not fun being responsible for everything that might happen with your data. There are tools that can help make some of this easier.
In practice, you're better off without RAID and having two or three hosts with data that is checksummed and signed (or the checksums are signed) so that you know you're getting back what you put into it.
One thing I like about BTRFS is the ability to run a scrub to check the data integrity.
If you haven't verified a restore works it is just hopes and dreams.
LOL You have never done a RAID5 rebuild then? It goes like this, especially on larger capacity arrays;
1) A drive fails
2) You replace the drive
3) Tell it to rebuild the array.
4) That rebuild takes then takes out the entire array.
Had that happen to me exactly once. Learned the hard lesson.. Do not rebuild arrays. DO NOT EVER REBUILD A RAID5 ARRAY.
Copy the data off, make a new array, then copy the data back to the array.