Software RAID Resyncs Can Be So Painful

raindog308

Had 3x12TB drives in a Synology DS418. Decided to add a fourth...

Personalities : [raid1] [raid6] [raid5] [raid4] [raidF1] 
md2 : active raid5 sdd5[3] sda5[0] sdc5[2] sdb5[1]
      23416309760 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  reshape =  7.9% (931304416/11708154880) finish=5084.5min speed=35325K/sec

That's with /proc/sys/dev/raid/speed_limit_max set at 600000. This is using Synology's SHR RAID with 1 drive fault tolerance.

The CPU is a Realtek RTD1296 quad-core 1.4GHz, which is probably a big part of why it's slow. Or maybe the fact that these are 7200rpm SATA drives. The original 3x12TBs are WD Gold, but I got an IronWold Pro for the 4th drive because it was $200 cheaper and still had a 5-year warranty (same 256MB cache).

This is just my backup NAS so I'm less performance-sensitive...but I wasn't expect a 4-day rsync odyssey.

All the drives are at 32C/90F. Sync started about 7 hours ago.

rm_

@raindog308 said: /proc/sys/dev/raid/speed_limit_max set at 600000

Set speed_limit_min to that as well, or even better both to something like 2000000. The speed should pick up, but then don't expect to access the NAS a lot in the meantime.

There's nothing inherent about "software" RAID being bad here compared to hardware.

Finally, not a great decision to run 4x12TB RAID5, hope you have backups. One drive could fail even right now for you (being startled by an increased load), leading to a full data loss.

luckypenguin

echo 50000 > /proc/sys/dev/raid/speed_limit_min
(you set max but there is also min, default is usually low)

echo 2048 > /sys/block/md2/md/stripe_cache_size
(use more RAM for active RAID blocks, default is usually 256)

forest

@rm_ said: There's nothing inherent about "software" RAID being bad here compared to hardware.

If anything, software RAID can be faster if it's tightly integrated with the filesystem (which mdraid is not), but hardware RAID will never be integrated with the filesystem.

forest

@raindog308 said: The original 3x12TBs are WD Gold, but I got an IronWold Pro for the 4th drive because it was $200 cheaper and still had a 5-year warranty (same 256MB cache).

If you've got four drives and want a level of redundancy where you need to tolerate the failure of just one drive, you should go with RAID10. As a bonus, you get enhanced multithreaded read performance, no write penalty, and half the time you can handle the failure of two drives.

@raindog308 said: The CPU is a Realtek RTD1296 quad-core 1.4GHz, which is probably a big part of why it's slow.

It's pretty much just doing a lot of XOR, so even a slow CPU should handle the speeds of spinning rust.