New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Software RAID Resyncs Can Be So Painful
raindog308
Administrator, Veteran
in General
Had 3x12TB drives in a Synology DS418. Decided to add a fourth...
Personalities : [raid1] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sdd5[3] sda5[0] sdc5[2] sdb5[1]
23416309760 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[=>...................] reshape = 7.9% (931304416/11708154880) finish=5084.5min speed=35325K/sec
That's with /proc/sys/dev/raid/speed_limit_max set at 600000. This is using Synology's SHR RAID with 1 drive fault tolerance.
The CPU is a Realtek RTD1296 quad-core 1.4GHz, which is probably a big part of why it's slow. Or maybe the fact that these are 7200rpm SATA drives. The original 3x12TBs are WD Gold, but I got an IronWold Pro for the 4th drive because it was $200 cheaper and still had a 5-year warranty (same 256MB cache).
This is just my backup NAS so I'm less performance-sensitive...but I wasn't expect a 4-day rsync odyssey.
All the drives are at 32C/90F. Sync started about 7 hours ago.
Comments
Set
speed_limit_minto that as well, or even better both to something like 2000000. The speed should pick up, but then don't expect to access the NAS a lot in the meantime.There's nothing inherent about "software" RAID being bad here compared to hardware.
Finally, not a great decision to run 4x12TB RAID5, hope you have backups. One drive could fail even right now for you (being startled by an increased load), leading to a full data loss.
echo 50000 > /proc/sys/dev/raid/speed_limit_min
(you set max but there is also min, default is usually low)
echo 2048 > /sys/block/md2/md/stripe_cache_size
(use more RAM for active RAID blocks, default is usually 256)
If anything, software RAID can be faster if it's tightly integrated with the filesystem (which mdraid is not), but hardware RAID will never be integrated with the filesystem since it's by definition filesystem agnostic.
In the future, look towards bcachefs being more mature! It is designed to be very fast with the use of a small SSD cache and has RAID support tightly integrated with full support for erasure coding, so you lose that RAID5 write hole.
If you've got four drives and want a level of redundancy where you need to tolerate the failure of just one drive, you should go with RAID10. As a bonus, you get enhanced multithreaded read performance, no write penalty, and half the time you can handle the failure of two drives. But yes even then, HDD resyncs can be slow.
It's pretty much just doing a lot of XOR, so even a slow CPU should handle the speeds of spinning rust. But to be sure, check
topto see how much CPU time the mdraid resync kernel thread is using.