New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Filesystem corruption is a serious issue. You can have some pretty bulletproof setups (ZFS can be really good) but there's no guarantees. Backups are always the responsibility of the customer at the end of the day. A provider offers convenience unless they explicitly offer guarantees. And even then, your data is probably worth more than what you're paying the provider.
As part of disaster recovery, you need a way to verify that your data is correct. You can't assume that there's been no bit rot or other silent corruption. Ideally there isn't, but is it worth gambling with your data that way?
In developing Slow Servers, I ended up mounting my host's partitions with sync enabled. This slows down disk a lot, even with SSD, but if there is a power cut or a crash, it significantly reduces the chances of corruption. Another bonus is that it's less likely for the fsck to fail and the host to never come back. I don't like driving almost 2 hours (one way) out of the blue to deal with that! Thankfully, I haven't yet.
RAID 1 is interesting because it looks better on paper than it is in reality. Under OpenBSD, I found (with 512e/4K sector drives anyway) that it made filesystem corruption considerably more likely than without. And while drive failures definitely happen, power cuts and crashes do as well -- perhaps more frequently in some cases.
One tricky element to RAID 1 is that you don't know which copy of the data is correct. Now sometimes one drive will know that a sector is bad and report that, and you'll just get the good sector. That isn't guanteed all of the time, however, and if there was a power cut during writes, each drive may have two "valid" blocks that are different.
Now let's say you replace one drive but you're missing sectors. Now you're missing data. The only way to make RAID 1 fairly bulletproof is to have it act like a democracy where the majority wins. This forces you to have 3 or 4 drives in a pool, too. And it removes all of the read speed benefits of RAID 1. RAID Z1 or Z2 is a much nicer setup for most people. But OpenBSD doesn't have ZFS, so it's a no-go for me.
I worry a bit about sparse qcow files when crashes or corruption happens. It seems like there's more failure mechanisms. I ended up fully preallocating raw files for drives with Slow Servers. It's slow and hard on the drives, but seems like the safest method involving filesystems on filesystems. Raw block devices are obviously better on the whole. At a previous employer we were using LVM and it worked pretty well on the whole. You can do that with ZFS as well.
Anyway, this is a lot of tangents. I know it's not fun being responsible for everything that might happen with your data. There are tools that can help make some of this easier.
In practice, you're better off without RAID and having two or three hosts with data that is checksummed and signed (or the checksums are signed) so that you know you're getting back what you put into it.
One thing I like about BTRFS is the ability to run a scrub to check the data integrity.
If you haven't verified a restore works it is just hopes and dreams.
LOL You have never done a RAID5 rebuild then? It goes like this, especially on larger capacity arrays;
1) A drive fails
2) You replace the drive
3) Tell it to rebuild the array.
4) That rebuild takes then takes out the entire array.
Had that happen to me exactly once. Learned the hard lesson.. Do not rebuild arrays. DO NOT EVER REBUILD A RAID5 ARRAY.
Copy the data off, make a new array, then copy the data back to the array.
Nooo, don't mount as sync! The performance hit is not worth it and it will increase fragmentation badly and puts significantly (sometimes up to 100x) more wear on solid state drives. Mount it with
data=journalinstead which will give you the same reliability at a much lower (but still not-insignificant) overhead. That will commit all data to the journal before being written to the filesystem. It's far more aggressive than the default ofdata=orderedwhich only journals metadata and transaction info.You can use dm-integrity for that. Then any attempt to read the invalid block will return EIO.
Are you sure you're preallocating correctly? Most modern filesystems will allow you to use something like
fallocate(1)which does instant preallocation and shouldn't be hard or slow on any drive:There's no such options on OpenBSD.
Because I write one VPS volume with dd at a time, fragmentation should be minimal.
This definitely is harder on SSDs. I'm very particular about using MLC SSDs like the Samsung 850 Pro.
That's good to know! Also not something on OpenBSD.
I'm specifically not doing that. I'm using
ddfor OpenBSD images installed via a ISO, and this for images that are originally qcow.This takes quite a while.
It's deliberate because blocks written will prevent fragmentation. It's also another way to ensure I never overcommit disk. It's certainly possible to prevent that in other ways, but this is another way to make it more foolproof.
Ah, I forgot you use OpenBSD. I guess FFS2 doesn't support holes then? And no authenticated disk encryption either.
I sure hope OpenBSD supports TRIM, otherwise you'll start ending up in a situation where a write of even a single byte causes an entire erase block's worth of data being written (often 16 MiB or more).
I'm curious what kind of corruption you saw. I don't know much about OpenBSD's softraid implementation. Did it occur when rebuilding the array? Or did the array somehow get silently out of sync without being detected?
Sorry, I have a backup.
What a pity!
The thing is, external 3.5-inch HDD enclosures are not very portable because most of them require external power.
That's why you purchase this and put it at work, home, etc
So? Majority of the systems you would be plugging it into also use external power. This means that external power is in the work area.
I would need to buy a UPS if it doesn't power with laptop's ports just in case electricity is out while backing up data.
I don't follow, why would I need it? And it requires external power. Anyway, I need to get a 4TB CMR HDD, they have gotten so expensive.
LOL. It beats bringing power supply everywhere. So, the solution is to have it where you want to access it.
4TB CMR HDD, expensive? It's still cheaper than buying 10-30TB drives.