Software RAID Resyncs Can Be So Painful

raindog308 · June 23

Had 3x12TB drives in a Synology DS418. Decided to add a fourth...

Personalities : [raid1] [raid6] [raid5] [raid4] [raidF1] 
md2 : active raid5 sdd5[3] sda5[0] sdc5[2] sdb5[1]
      23416309760 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  reshape =  7.9% (931304416/11708154880) finish=5084.5min speed=35325K/sec

That's with /proc/sys/dev/raid/speed_limit_max set at 600000. This is using Synology's SHR RAID with 1 drive fault tolerance.

The CPU is a Realtek RTD1296 quad-core 1.4GHz, which is probably a big part of why it's slow. Or maybe the fact that these are 7200rpm SATA drives. The original 3x12TBs are WD Gold, but I got an IronWold Pro for the 4th drive because it was $200 cheaper and still had a 5-year warranty (same 256MB cache).

This is just my backup NAS so I'm less performance-sensitive...but I wasn't expect a 4-day rsync odyssey.

All the drives are at 32C/90F. Sync started about 7 hours ago.

rm_ · June 23

@raindog308 said: /proc/sys/dev/raid/speed_limit_max set at 600000

Set speed_limit_min to that as well, or even better both to something like 2000000. The speed should pick up, but then don't expect to access the NAS a lot in the meantime.

There's nothing inherent about "software" RAID being bad here compared to hardware.

Finally, not a great decision to run 4x12TB RAID5, hope you have backups. One drive could fail even right now for you (being startled by an increased load), leading to a full data loss.

luckypenguin · June 23

echo 50000 > /proc/sys/dev/raid/speed_limit_min
(you set max but there is also min, default is usually low)

echo 2048 > /sys/block/md2/md/stripe_cache_size
(use more RAM for active RAID blocks, default is usually 256)

forest · June 23

@rm_ said: There's nothing inherent about "software" RAID being bad here compared to hardware.

If anything, software RAID can be faster if it's tightly integrated with the filesystem (which mdraid is not), but hardware RAID will never be integrated with the filesystem since it's by definition filesystem agnostic.

In the future, look towards bcachefs being more mature! It is designed to be very fast with the use of a small SSD cache and has RAID support tightly integrated with full support for erasure coding, so you lose that RAID5 write hole.

forest · June 23

@raindog308 said: The original 3x12TBs are WD Gold, but I got an IronWold Pro for the 4th drive because it was $200 cheaper and still had a 5-year warranty (same 256MB cache).

If you've got four drives and want a level of redundancy where you need to tolerate the failure of just one drive, you should go with RAID10. As a bonus, you get enhanced multithreaded read performance, no write penalty, and half the time you can handle the failure of two drives. But yes even then, HDD resyncs can be slow.

@raindog308 said: The CPU is a Realtek RTD1296 quad-core 1.4GHz, which is probably a big part of why it's slow.

It's pretty much just doing a lot of XOR, so even a slow CPU should handle the speeds of spinning rust. But to be sure, check top to see how much CPU time the mdraid resync kernel thread is using.

nullnothere · June 23

@forest said: bcachefs being more mature

I've not been following it too much, but there's been enough drama over it already. I don't use it at all - so any insights would be welcome. I'm not sure at all on why I (or anyone) would pick it over ZFS for anything worth preserving (and of course how much time to invest in it).

forest · June 23

@nullnothere said:

@forest said: bcachefs being more mature

I've not been following it too much, but there's been enough drama over it already. I don't use it at all - so any insights would be welcome. I'm not sure at all on why I (or anyone) would pick it over ZFS for anything worth preserving (and of course how much time to invest in it).

The drama has been largely resolved now that development has been taken (temporarily) out of tree. There was a lot of friction earlier due to the fast pace of development that caused arguments on LKML when it was still in-tree. Now it's temporarily installed with dkms until it goes back in tree.

The main reason to pick it over ZFS, other than being GPL instead of CDDL, is its focus on performance (Even ZFS with L2ARC and SLOG on SSD doesn't come anywhere near it) while simultaneously having an extremely rugged design with data safety as a priority. Its main selling point is that it's extremely reliable and very fast.

nullnothere · June 23

@forest said: ... simultaneously having an extremely rugged design with data safety as a priority. Its main selling point is that it's extremely reliable and very fast.

Fair points but I tend to be very conservative when it comes to trusting filesystems with my data. So it'll take me some time (and a LOT more other users using it regularly and consistently for some extended amount of time across many kernel versions and distro releases) before I'd be willing to wade into such territory. Till then ext4, mdadm, lvm, luks and ZFS (where possible/useful) for me...

forest · June 23

@nullnothere said:

@forest said: ... simultaneously having an extremely rugged design with data safety as a priority. Its main selling point is that it's extremely reliable and very fast.

Fair points but I tend to be very conservative when it comes to trusting filesystems with my data. So it'll take me some time (and a LOT more other users using it regularly and consistently for some extended amount of time across many kernel versions and distro releases) before I'd be willing to wade into such territory. Till then ext4, mdadm, lvm, luks and ZFS (where possible/useful) for me...

Me too. It's promising so far, but for high-reliability, I still go with ZFS (or ext4/xfs + mdraid). It is definitely something to look forward to in the near future, though!

DataRecovery · June 23

@raindog308 said: Synology's SHR RAID

This could also be the reason.

Looks like you have the drives of the same size, which involves less Synology's proprietary tricks, but still.

raindog308 · June 23

@luckypenguin said:
echo 2048 > /sys/block/md2/md/stripe_cache_size (use more RAM for active RAID blocks, default is usually 256)

Default was set to 1024, but doubling to 2048 shaved ~600 mins off the estimate.

rm_ · June 23

@forest said: The drama has been largely resolved now that development has been taken (temporarily) out of tree. There was a lot of friction earlier due to the fast pace of development that caused arguments on LKML when it was still in-tree. Now it's temporarily installed with dkms until it goes back in tree.

That guy is not OK in the head unfortunately, meanwhile it seemed like a promising project.

Bcachefs creator insists his custom LLM is female and 'fully conscious'
we have full AGI, and now my life has been reduced from being perhaps the best engineer in the world to just raising an AI that in many respects acts like a teenager

Now I'd trust ReiserFS more than this.

david · June 23

Recently, one of my vps experienced a RAID failure. After it became available again after rebuilding, I ran a btrfs scrub, and the data was corrupt. A good reason to use a filesystem with the ability to check data integrity (zfs would be good, too).

Mainfrezzer · June 23

@david said:
Recently, one of my vps experienced a RAID failure. After it became available again after rebuilding, I ran a btrfs scrub, and the data was corrupt. A good reason to use a filesystem with the ability to check data integrity (zfs would be good, too).

which one are we talking about? hopefully you didnt run raid5 btrfs

david · June 23

@Mainfrezzer said:

@david said:
Recently, one of my vps experienced a RAID failure. After it became available again after rebuilding, I ran a btrfs scrub, and the data was corrupt. A good reason to use a filesystem with the ability to check data integrity (zfs would be good, too).

which one are we talking about? hopefully you didnt run raid5 btrfs

The RAID was managed by the provider; I'm not sure what they used.

totally_not_banned · June 23

@raindog308 said:
The CPU is a Realtek RTD1296 quad-core 1.4GHz, which is probably a big part of why it's slow.

It's hard to tell what chip https://www.cpubenchmark.net/cpu.php?cpu=ARM+Cortex-A53+4+Core+1400+MHz&id=4143 exactly refers to but if the RTD1296 is that range it being slow is sadly not much of surprise. I kind of would expect a lot of expensive operations (encryption and such) to be implemented in hardware though, which at least in theory should somewhat offset the slowness.

Considering how readily available low power ARM boards are these days it shouldn't be too hard to mcgyver something more powerful. If interest/time aligns with that is obviously a completely different question.

layer7 · June 23

Hi,

as long as the process has no 100% CPU usage, the CPU should not be the (primary) problem.

So that leaves configuration settings and the disks itself as potential sources of slowliness.

But since waiting activity seems also not to exist, most probably the settings are slowing down things here.

I suggest asking the good old AI and/or your favorite search engine howto speed up raid 5 stuff.

host_c · June 23

@raindog308 said: active raid5

4-core ARM Cortex-A53 @ 1.4 GHz, I bet your phone has more horsepower then that CPU.

Synolgy OS might throttle the CPU time also for this task, but that really depends on what other services you have started on that NAS.

Also, even if some think that RAID5 is simple to calculate, for rebuild, that cpu will have to do a shit load math and it ain't a storage ASIC.

Larger the drives, the longer it will take.

At 12TB / drive, I would give it ~2-3 days probably.

As some context, a 24 Drive Raid60 Array ( 8x3 Raid 6 ) of 22TB SAS Drives 7200 RPM on a modern ASIC with light load took ~ 8 days to complete.

So I think you are good.

EDIT:

Also, if it makes you fell better, I have a QNAP TS-435XEU that has a Marvell OCTEON TX2 CN9131 and 32 GB ram, it is a piece of garbage that cost ~2000 USD with the shelf it has. It is slow as a snail. We upgraded the drives from 12 GB to 24 TB SATA, it was a 3 week project.

raindog308 · June 23

@totally_not_banned said: It's hard to tell what chip https://www.cpubenchmark.net/cpu.php?cpu=ARM+Cortex-A53+4+Core+1400+MHz&id=4143 exactly refers to but if the RTD1296 is that range it being slow is sadly not much of surprise.

This is my backup Synology, so I don't mind it being slow...well, until now.

My main ones are

a DS1522+ which has an AMD Ryzen Embedded R1600
a DS920+ which has an Intel(R) Celeron(R) J4125 CPU @ 2.00GHz

I kind of would expect a lot of expensive operations (encryption and such) to be implemented in hardware though, which at least in theory should somewhat offset the slowness.

processor   : 0
model name  : ARMv8 Processor rev 4 (v8l)
BogoMIPS    : 54.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd03
CPU revision    : 4

totally_not_banned · June 23

@raindog308 said:

> processor : 0
> model name    : ARMv8 Processor rev 4 (v8l)
> BogoMIPS  : 54.00
> Features  : fp asimd evtstrm aes pmull sha1 sha2 crc32
> CPU implementer   : 0x41
> CPU architecture: 8
> CPU variant   : 0x0
> CPU part  : 0xd03
> CPU revision  : 4
>

Yeah, kind of expected but in general hardware acceleration goes a long way.

vendor_id   :   CentaurHauls
cpu family  :   6
model   :   13
model name  :   VIA C7 Processor 1500MHz
stepping    :   0
flags   :   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge cmov pat clflush acpi mmx fxsr sse sse2 tm nx pni est tm2 xtpr rng rng_en ace ace_en ace2 ace2_en phe phe_en pmm pmm_en

This ~20 year old snail gets dwarfed by the RTD1296 (assuming the passmark data is representative) but still manages to push 30mbits of encrypted traffic as long as VIA's proprietary padlock engine is used.

jsg · June 23

@totally_not_banned said:
I kind of would expect a lot of expensive operations (encryption and such) to be implemented in hardware though ...

What for? AES is implemented pretty much everywhere nowadays (incl. even quite a few MCUs) and the core of Raid 5 are two simple binary operations (and even error correction can be done by normal processors).

totally_not_banned · June 23

@jsg said:

@totally_not_banned said:
I kind of would expect a lot of expensive operations (encryption and such) to be implemented in hardware though ...

What for? AES is implemented pretty much everywhere nowadays (incl. even quite a few MCUs) and the core of Raid 5 are two simple binary operations (and even error correction can be done by normal processors).

Well, encryption is obviously useless for raid but nothing is stopping manufacturers from implementing stuff that's actually useful. I mean, CPUs could do AES before intel added their ops too. It's just faster now. How a raid version would look like... i don't really know. Something bulk, i guess. If VIA can cook their own proprietary extension others can do it too

gremeyer · June 23

One time, I had an initial software raid sync take several days to complete. After giving it the old turn it off and back on (reboot) the ETA went from several days to a few hours.

Howdy, Stranger!

Categories

In this Discussion

Software RAID Resyncs Can Be So Painful

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Software RAID Resyncs Can Be So Painful

Comments