Getting Slow Disk Speed on RAID-1 (Only ~200 MB/s) – Anyone Else Facing This?

hostomega · November 2025

I’m facing a strange issue on a new server from GorillaServers.

Issue:

When using RAID-1 (mdadm) on AlmaLinux 9, I get only ~200 MB/s write speed.

If I test each NVMe individually (no RAID), I easily get ~1 GB/s+.

I cross-checked with a friend who has a very similar setup — same result:
RAID-1 = ~200 MB/s
Single disk = 1 GB/s

The datacenter replaced the server, but the issue is still the same.
So I’m wondering if anyone else has experienced this?

⸻

Server Specs
• Asrock Rack B650D4U
• AMD Ryzen 9 9950X
• 2 × 1.92TB NVMe (Kioxia KCD6XLUL1T92)
• AlmaLinux 9
• VirtFusion Platform
• mdadm RAID-1 (default installer)

⸻

What I Tested
• dd test: ~200 MB/s on RAID-1
• On raw NVMe (no RAID): 1 GB/s
• FIO confirms that individual NVMe drives are fast
• Tried reinstalling different times
• Tried another identical server — same issue

⸻

Question

Is anyone else seeing this slow mdadm RAID-1 performance on AlmaLinux 9 with NVMe?

Is this:
• an AlmaLinux 9 kernel issue?
• mdadm RAID-1 sync/write bottleneck?
• something specific with these Kioxia enterprise NVMe drives?
• or something related to the B650D4U board?

Any insights would be appreciated.

-------------------- A Bench.sh Script By Teddysun -------------------
 Version            : v2025-05-08
 Usage              : wget -qO- bench.sh | bash
----------------------------------------------------------------------
 CPU Model          : AMD Ryzen 9 9950X 16-Core Processor
 CPU Cores          : 32 @ 5603.756 MHz
 CPU Cache          : 1024 KB
 AES-NI             : ✓ Enabled
 VM-x/AMD-V         : ✓ Enabled
 Total Disk         : 1.7 TB (359.0 GB Used)
 Total Mem          : 185.8 GB (55.0 GB Used)
 Total Swap         : 16.0 GB (4.4 GB Used)
 System uptime      : 0 days, 9 hour 54 min
 Load average       : 1.33, 1.46, 1.35
 OS                 : AlmaLinux release 9.6 (Sage Margay)
 Arch               : x86_64 (64 Bit)
 Kernel             : 5.14.0-570.62.1.el9_6.x86_64
 TCP CC             : cubic
 Virtualization     : Dedicated
 IPv4/IPv6          : ✓ Online / ✓ Online
 Organization       : AS53850 GorillaServers, Inc.
 Location           : Los Angeles / US
 Region             : California
----------------------------------------------------------------------
 I/O Speed(1st run) : 200 MB/s
 I/O Speed(2nd run) : 199 MB/s
 I/O Speed(3rd run) : 199 MB/s
 I/O Speed(average) : 199.3 MB/s
----------------------------------------------------------------------

adelsmith331 · November 2025

I want to know about this.

MikeA · November 2025

I don't know but my whole setup runs Almalinux 9/10 with mdadm raid but I've never used Kioxia and only use onboard M.2 generally. I've never had an issue like this with the WD/Samsung consumer drives I mostly use. Multi-gig writes in raid-1, and resyncs with 1-2GB/s sustained. Might just have something to do with the fact these Kioxia models are for read-intensive workloads, and there's some quirk with mdadm raid and a drive/partition setting with them. Compare the 30K write IOPS max with the typical non-DC NVMe I use that have like 1M-1.2M write IOPS max. Maybe you're hitting some bottleneck with the write IOPS and something with the mdadm raid. Tldr you should rule out the operating system and motherboard really.
Edit: AI says some mdadm bitmap policy could be the issue with but I don't know Wish I had one to test on though, would be interesting to know the cause.

hostomega · November 2025

@MikeA said:
I don't know but my whole setup runs Almalinux 9/10 with mdadm raid but I've never used Kioxia and only use onboard M.2 generally. I've never had an issue like this with the WD/Samsung consumer drives I mostly use. Multi-gig writes in raid-1, and resyncs with 1-2GB/s sustained. Might just have something to do with the fact these Kioxia models are for read-intensive workloads, and there's some quirk with mdadm raid and a drive/partition setting with them. Compare the 30K write IOPS max with the typical non-DC NVMe I use that have like 1M-1.2M write IOPS max. Maybe you're hitting some bottleneck with the write IOPS and something with the mdadm raid. Tldr you should rule out the operating system and motherboard really.
Edit: AI says some mdadm bitmap policy could be the issue with but I don't know Wish I had one to test on though, would be interesting to know the cause.

I am also using same Motherboard and on board NVMe in another DC. I am getting 3 GB/s speed.

zakkuuno · November 2025

check cat /proc/mdstat

mdadm’s probably initializing the raid1 in the background

PuDLeZ · November 2025

Dumb question but are you sure the disks completed syncing after setting up the raid1 and you testing? Not sure with almalinux but standard will be something like 'cat /proc/mdstat' without the quotes.

If it's fully synced, can you try the single disk mode again and use tmux/screen/multiple shells to test both drives at the same time to see if that causes the same issue to happen?

Lastly, can you temporarly use a different distro (debian/etc) to see if it experiences the same issue? Basically just to confirm if it is or isn't almalinux9 specific.

PuDLeZ · November 2025

@zakkuuno said:
check cat /proc/mdstat

mdadm’s probably initializing the raid1 in the background

hehe, got distracted with work stuff and didn't refresh the page before typing/posting my comment. Honestly, I'd bet this is the reason. I know when I first started using mdadm and raid1/5/10 on drives, I was annoyed that it took "forever" to sync the disks/array.

hostomega · November 2025

@zakkuuno said:
check cat /proc/mdstat

mdadm’s probably initializing the raid1 in the background

I have been facing this issue for a week. So no chance for raid sync

hostomega · November 2025

@PuDLeZ said:
Dumb question but are you sure the disks completed syncing after setting up the raid1 and you testing? Not sure with almalinux but standard will be something like 'cat /proc/mdstat' without the quotes.

If it's fully synced, can you try the single disk mode again and use tmux/screen/multiple shells to test both drives at the same time to see if that causes the same issue to happen?

Lastly, can you temporarly use a different distro (debian/etc) to see if it experiences the same issue? Basically just to confirm if it is or isn't almalinux9 specific.

It's not a raid sync issue. For your satisfaction, I am sharing again

[root@104 ~]# cat /proc/mdstat
Personalities : [raid1] 
md124 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      1049536 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md125 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 nvme0n1p4[0] nvme1n1p4[1]
      1856348160 blocks super 1.2 [2/2] [UU]
      bitmap: 8/14 pages [32KB], 65536KB chunk

md127 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      16776192 blocks super 1.2 [2/2] [UU]

unused devices: <none>
[root@104 ~]#

PuDLeZ · November 2025

@hostomega said:

@PuDLeZ said:
Dumb question but are you sure the disks completed syncing after setting up the raid1 and you testing? Not sure with almalinux but standard will be something like 'cat /proc/mdstat' without the quotes.

If it's fully synced, can you try the single disk mode again and use tmux/screen/multiple shells to test both drives at the same time to see if that causes the same issue to happen?

Lastly, can you temporarly use a different distro (debian/etc) to see if it experiences the same issue? Basically just to confirm if it is or isn't almalinux9 specific.

It's not a raid sync issue. For your satisfaction, I am sharing again
[root@104 ~]# cat /proc/mdstat
Personalities : [raid1] 
md124 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      1049536 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md125 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 nvme0n1p4[0] nvme1n1p4[1]
      1856348160 blocks super 1.2 [2/2] [UU]
      bitmap: 8/14 pages [32KB], 65536KB chunk

md127 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      16776192 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
[root@104 ~]# 

Cool, so that's not it. I figured it was a dumb question but I had to ask... I did copy/paste part of your first post into chatgpt and a short version of the answer is to try something more accurate than dd like
fio --name=seqwrite --filename=/mnt/raid/testfile --rw=write --bs=1M --size=4G --direct=1 --numjobs=1 --iodepth=32

advinservers · November 2025

Yes, we faced this exact same issue but with KCD6XLUL3T84 drives on ASRockRack B650D4U motherboards.

For some weird reason, one of the disks would always run at 230 MB/s according to hdparm/yabs.sh benchmarks.

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/md2):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 283.75 MB/s  (70.9k) | 283.03 MB/s   (4.4k)
Write      | 284.50 MB/s  (71.1k) | 284.52 MB/s   (4.4k)
Total      | 568.25 MB/s (142.0k) | 567.56 MB/s   (8.8k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 270.25 MB/s    (527) | 266.10 MB/s    (259)
Write      | 284.61 MB/s    (555) | 283.82 MB/s    (277)
Total      | 554.87 MB/s   (1.0k) | 549.92 MB/s    (536)

It was weird because the slow performance would sometimes "switch" drives (e.g., /dev/nvme0n1 would be slow, we'd reboot, and then /dev/nvme1n1 would be slow). Sometimes the slow performance wouldn't show on the disks individually and sometimes it would.

We replicated it across multiple motherboards and systems where one of the disks would always be a lot slower than the other when using U.2 NVMe disks on the ASRockRack B650D4U platform.

The one solution that we found was to remove one of the disks on the M.2 (we were using M.2 -> U.2) slot closest to the socket and instead move it to the PCIe x4 slot (we are using PCIe -> M.2 -> U.2). This is only an issue with the B650D4U and not on any of our Supermicro AM5 motherboards.

We use Kioxia U.2 Enterprise NVMe on almost all of our Ryzen VPS hypervisors and it has not been an issue since moving one of the disks to the PCIe x4 slot on the board.

We were only able to reproduce this issue when using U.2 NVMe SSD, regular M.2 seems to be fine. We have only tested it with Kioxia U.2 so perhaps it could be some incompatibility between Kioxia and B650D4U, or it could just be a general issue with all U.2 drives, we're not really sure.

hostomega · November 2025

@PuDLeZ said:
@hostomega said:

@PuDLeZ said:
Dumb question but are you sure the disks completed syncing after setting up the raid1 and you testing? Not sure with almalinux but standard will be something like 'cat /proc/mdstat' without the quotes.

If it's fully synced, can you try the single disk mode again and use tmux/screen/multiple shells to test both drives at the same time to see if that causes the same issue to happen?

Lastly, can you temporarly use a different distro (debian/etc) to see if it experiences the same issue? Basically just to confirm if it is or isn't almalinux9 specific.

It's not a raid sync issue. For your satisfaction, I am sharing again
[root@104 ~]# cat /proc/mdstat
Personalities : [raid1] 
md124 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      1049536 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md125 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 nvme0n1p4[0] nvme1n1p4[1]
      1856348160 blocks super 1.2 [2/2] [UU]
      bitmap: 8/14 pages [32KB], 65536KB chunk

md127 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      16776192 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
[root@104 ~]# 
Cool, so that's not it. I figured it was a dumb question but I had to ask... I did copy/paste part of your first post into chatgpt and a short version of the answer is to try something more accurate than dd like
fio --name=seqwrite --filename=/mnt/raid/testfile --rw=write --bs=1M --size=4G --direct=1 --numjobs=1 --iodepth=32

[root@104 ~]# fio --name=seqwrite --filename=/mnt/raid/testfile --rw=write --bs=1M --size=4G --direct=1 --numjobs=1 --iodepth=32
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=32
fio-3.35
Starting 1 process
seqwrite: Laying out IO file (1 file / 4096MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 1 (f=1): [W(1)][100.0%][w=187MiB/s][w=187 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=250102: Fri Nov 14 14:41:06 2025
  write: IOPS=198, BW=198MiB/s (208MB/s)(4096MiB/20685msec); 0 zone resets
    clat (usec): min=2465, max=10263, avg=5044.50, stdev=204.90
     lat (usec): min=2471, max=10268, avg=5049.22, stdev=205.04
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 5014], 10.00th=[ 5014], 20.00th=[ 5014],
     | 30.00th=[ 5014], 40.00th=[ 5014], 50.00th=[ 5014], 60.00th=[ 5014],
     | 70.00th=[ 5014], 80.00th=[ 5014], 90.00th=[ 5080], 95.00th=[ 5145],
     | 99.00th=[ 5669], 99.50th=[ 6325], 99.90th=[ 7767], 99.95th=[ 8225],
     | 99.99th=[10290]
   bw (  KiB/s): min=182272, max=204800, per=100.00%, avg=202851.90, stdev=3575.25, samples=41
   iops        : min=  178, max=  200, avg=198.10, stdev= 3.49, samples=41
  lat (msec)   : 4=0.02%, 10=99.95%, 20=0.02%
  cpu          : usr=0.12%, sys=0.81%, ctx=4178, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=4096MiB (4295MB), run=20685-20685msec

Disk stats (read/write):
    md126: ios=113/35386, merge=0/0, ticks=16/118397, in_queue=118413, util=99.19%, aggrios=56/35891, aggrmerge=0/162, aggrticks=7/57464, aggrin_queue=57471, aggrutil=97.95%
  nvme1n1: ios=0/35891, merge=0/162, ticks=0/109356, in_queue=109356, util=97.95%
  nvme0n1: ios=113/35891, merge=0/162, ticks=14/5573, in_queue=5587, util=7.48%
[root@104 ~]#

zakkuuno · November 2025

@hostomega said:

@zakkuuno said:
check cat /proc/mdstat

mdadm’s probably initializing the raid1 in the background

I have been facing this issue for a week. So no chance for raid sync

That’s pretty strange then. I had to ask too because you mentioned new server and reinstalls so my first thought was the first syncing never completed.

I would try the same on debian with debian’s default settings on mdadm to see if it’s specific to the alma/rhel software stack. ChatGPT however suggested to try disabling bitmap and see if that makes any difference.

Have you tried yabs? Is it giving poor results too?

hostomega · November 2025

@zakkuuno said:

@hostomega said:

@zakkuuno said:
check cat /proc/mdstat

mdadm’s probably initializing the raid1 in the background

I have been facing this issue for a week. So no chance for raid sync

That’s pretty strange then. I had to ask too because you mentioned new server and reinstalls so my first thought was the first syncing never completed.

I would try the same on debian with debian’s default settings on mdadm to see if it’s specific to the alma/rhel software stack. ChatGPT however suggested to try disabling bitmap and see if that makes any difference.

Have you tried yabs? Is it giving poor results too?

yabs is also giving the same result.

zakkuuno · November 2025

@hostomega said:

@zakkuuno said:

@hostomega said:

@zakkuuno said:
check cat /proc/mdstat

mdadm’s probably initializing the raid1 in the background

I have been facing this issue for a week. So no chance for raid sync

That’s pretty strange then. I had to ask too because you mentioned new server and reinstalls so my first thought was the first syncing never completed.

I would try the same on debian with debian’s default settings on mdadm to see if it’s specific to the alma/rhel software stack. ChatGPT however suggested to try disabling bitmap and see if that makes any difference.

Have you tried yabs? Is it giving poor results too?

yabs is also giving the same result.

Just saw the comment by advinservers after I typed it.

hostomega · November 2025

@advinservers said:
Yes, we faced this exact same issue but with KCD6XLUL3T84 drives on ASRockRack B650D4U motherboards.

For some weird reason, one of the disks would always run at 230 MB/s according to hdparm/yabs.sh benchmarks.
fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/md2):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 283.75 MB/s  (70.9k) | 283.03 MB/s   (4.4k)
Write      | 284.50 MB/s  (71.1k) | 284.52 MB/s   (4.4k)
Total      | 568.25 MB/s (142.0k) | 567.56 MB/s   (8.8k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 270.25 MB/s    (527) | 266.10 MB/s    (259)
Write      | 284.61 MB/s    (555) | 283.82 MB/s    (277)
Total      | 554.87 MB/s   (1.0k) | 549.92 MB/s    (536)
It was weird because the slow performance would sometimes "switch" drives (e.g., /dev/nvme0n1 would be slow, we'd reboot, and then /dev/nvme1n1 would be slow). Sometimes the slow performance wouldn't show on the disks individually and sometimes it would.

We replicated it across multiple motherboards and systems where one of the disks would always be a lot slower than the other when using U.2 NVMe disks on the ASRockRack B650D4U platform.

The one solution that we found was to remove one of the disks on the M.2 (we were using M.2 -> U.2) slot closest to the socket and instead move it to the PCIe x4 slot (we are using PCIe -> M.2 -> U.2). This is only an issue with the B650D4U and not on any of our Supermicro AM5 motherboards.

We use Kioxia U.2 Enterprise NVMe on almost all of our Ryzen VPS hypervisors and it has not been an issue since moving one of the disks to the PCIe x4 slot on the board.

We were only able to reproduce this issue when using U.2 NVMe SSD, regular M.2 seems to be fine. We have only tested it with Kioxia U.2 so perhaps it could be some incompatibility between Kioxia and B650D4U, or it could just be a general issue with all U.2 drives, we're not really sure.

@GorillaServers You can also diagnose the system using this method.

PuDLeZ · November 2025

@hostomega said:

@PuDLeZ said:
@hostomega said:

@PuDLeZ said:
Dumb question but are you sure the disks completed syncing after setting up the raid1 and you testing? Not sure with almalinux but standard will be something like 'cat /proc/mdstat' without the quotes.

If it's fully synced, can you try the single disk mode again and use tmux/screen/multiple shells to test both drives at the same time to see if that causes the same issue to happen?

Lastly, can you temporarly use a different distro (debian/etc) to see if it experiences the same issue? Basically just to confirm if it is or isn't almalinux9 specific.

It's not a raid sync issue. For your satisfaction, I am sharing again
[root@104 ~]# cat /proc/mdstat
Personalities : [raid1] 
md124 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      1049536 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md125 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 nvme0n1p4[0] nvme1n1p4[1]
      1856348160 blocks super 1.2 [2/2] [UU]
      bitmap: 8/14 pages [32KB], 65536KB chunk

md127 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      16776192 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
[root@104 ~]# 
Cool, so that's not it. I figured it was a dumb question but I had to ask... I did copy/paste part of your first post into chatgpt and a short version of the answer is to try something more accurate than dd like
fio --name=seqwrite --filename=/mnt/raid/testfile --rw=write --bs=1M --size=4G --direct=1 --numjobs=1 --iodepth=32

[root@104 ~]# fio --name=seqwrite --filename=/mnt/raid/testfile --rw=write --bs=1M --size=4G --direct=1 --numjobs=1 --iodepth=32
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=32
fio-3.35
Starting 1 process
seqwrite: Laying out IO file (1 file / 4096MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 1 (f=1): [W(1)][100.0%][w=187MiB/s][w=187 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=250102: Fri Nov 14 14:41:06 2025
  write: IOPS=198, BW=198MiB/s (208MB/s)(4096MiB/20685msec); 0 zone resets
    clat (usec): min=2465, max=10263, avg=5044.50, stdev=204.90
     lat (usec): min=2471, max=10268, avg=5049.22, stdev=205.04
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 5014], 10.00th=[ 5014], 20.00th=[ 5014],
     | 30.00th=[ 5014], 40.00th=[ 5014], 50.00th=[ 5014], 60.00th=[ 5014],
     | 70.00th=[ 5014], 80.00th=[ 5014], 90.00th=[ 5080], 95.00th=[ 5145],
     | 99.00th=[ 5669], 99.50th=[ 6325], 99.90th=[ 7767], 99.95th=[ 8225],
     | 99.99th=[10290]
   bw (  KiB/s): min=182272, max=204800, per=100.00%, avg=202851.90, stdev=3575.25, samples=41
   iops        : min=  178, max=  200, avg=198.10, stdev= 3.49, samples=41
  lat (msec)   : 4=0.02%, 10=99.95%, 20=0.02%
  cpu          : usr=0.12%, sys=0.81%, ctx=4178, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=4096MiB (4295MB), run=20685-20685msec

Disk stats (read/write):
    md126: ios=113/35386, merge=0/0, ticks=16/118397, in_queue=118413, util=99.19%, aggrios=56/35891, aggrmerge=0/162, aggrticks=7/57464, aggrin_queue=57471, aggrutil=97.95%
  nvme1n1: ios=0/35891, merge=0/162, ticks=0/109356, in_queue=109356, util=97.95%
  nvme0n1: ios=113/35891, merge=0/162, ticks=14/5573, in_queue=5587, util=7.48%
[root@104 ~]#

Wow, I find it interesting nvme1n1 was like 97.95% while the other disk is only 7.48%
no clue if your results means one of your drives is bad/slow or if it's some type of bus/lane/etc. bottleneck that's only experienced when both drives are being used at the same time.

I did a similar test on a local machine I have (four 2TB nvme drives in a raid10) and as you can see towards the bottom, they're all fairly close in util %

foodrelease:/mnt/nvme # fio --name=seqwrite --filename=/mnt/nvme/benchtest.img --rw=write --bs=1M --size=4G --direct=1 --numjobs=1 --iodepth=32
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=32
fio-3.38
Starting 1 process
seqwrite: Laying out IO file (1 file / 4096MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 1 (f=1)
seqwrite: (groupid=0, jobs=1): err= 0: pid=192963: Fri Nov 14 15:02:40 2025
  write: IOPS=3927, BW=3927MiB/s (4118MB/s)(4096MiB/1043msec); 0 zone resets
    clat (usec): min=134, max=28289, avg=249.88, stdev=914.39
     lat (usec): min=137, max=28297, avg=254.03, stdev=914.41
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  137], 10.00th=[  137], 20.00th=[  137],
     | 30.00th=[  137], 40.00th=[  139], 50.00th=[  141], 60.00th=[  143],
     | 70.00th=[  143], 80.00th=[  143], 90.00th=[  145], 95.00th=[  145],
     | 99.00th=[ 5735], 99.50th=[ 6652], 99.90th=[ 6783], 99.95th=[ 9896],
     | 99.99th=[28181]
   bw (  MiB/s): min= 3842, max= 3968, per=99.44%, avg=3905.00, stdev=89.10, samples=2
   iops        : min= 3842, max= 3968, avg=3905.00, stdev=89.10, samples=2
  lat (usec)   : 250=98.17%, 500=0.07%, 750=0.02%
  lat (msec)   : 4=0.07%, 10=1.61%, 20=0.02%, 50=0.02%
  cpu          : usr=4.22%, sys=16.41%, ctx=4166, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=3927MiB/s (4118MB/s), 3927MiB/s-3927MiB/s (4118MB/s-4118MB/s), io=4096MiB (4295MB), run=1043-1043msec

Disk stats (read/write):
    md0: ios=0/6784, sectors=0/6946816, merge=0/0, ticks=0/1348, in_queue=1348, util=71.69%, aggrios=0/4162, aggsectors=0/4194825, aggrmerge=0/0, aggrticks=0/786, aggrin_queue=894, aggrutil=73.13%
  nvme3n1: ios=0/4162, sectors=0/4194825, merge=0/0, ticks=0/731, in_queue=809, util=60.72%
  nvme4n1: ios=0/4162, sectors=0/4194825, merge=0/0, ticks=0/814, in_queue=936, util=67.12%
  nvme5n1: ios=0/4162, sectors=0/4194825, merge=0/0, ticks=0/831, in_queue=956, util=73.13%
  nvme2n1: ios=0/4162, sectors=0/4194825, merge=0/0, ticks=0/771, in_queue=877, util=65.15%
foodrelease:/mnt/nvme #

DBA · November 2025

The B650D4U has 2xNVME slots with different speeds and IO paths - one PCIe5.0 x4 from the CPU and the other PCIe4.0 x4 from the FCH. Not a good pairing for RAID 1 - the second drive would do better connected though a PCIe slot that also has lanes directly from the CPU.

hostomega · November 2025

@advinservers said:
Yes, we faced this exact same issue but with KCD6XLUL3T84 drives on ASRockRack B650D4U motherboards.

For some weird reason, one of the disks would always run at 230 MB/s according to hdparm/yabs.sh benchmarks.
fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/md2):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 283.75 MB/s  (70.9k) | 283.03 MB/s   (4.4k)
Write      | 284.50 MB/s  (71.1k) | 284.52 MB/s   (4.4k)
Total      | 568.25 MB/s (142.0k) | 567.56 MB/s   (8.8k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 270.25 MB/s    (527) | 266.10 MB/s    (259)
Write      | 284.61 MB/s    (555) | 283.82 MB/s    (277)
Total      | 554.87 MB/s   (1.0k) | 549.92 MB/s    (536)
It was weird because the slow performance would sometimes "switch" drives (e.g., /dev/nvme0n1 would be slow, we'd reboot, and then /dev/nvme1n1 would be slow). Sometimes the slow performance wouldn't show on the disks individually and sometimes it would.

We replicated it across multiple motherboards and systems where one of the disks would always be a lot slower than the other when using U.2 NVMe disks on the ASRockRack B650D4U platform.

The one solution that we found was to remove one of the disks on the M.2 (we were using M.2 -> U.2) slot closest to the socket and instead move it to the PCIe x4 slot (we are using PCIe -> M.2 -> U.2). This is only an issue with the B650D4U and not on any of our Supermicro AM5 motherboards.

We use Kioxia U.2 Enterprise NVMe on almost all of our Ryzen VPS hypervisors and it has not been an issue since moving one of the disks to the PCIe x4 slot on the board.

We were only able to reproduce this issue when using U.2 NVMe SSD, regular M.2 seems to be fine. We have only tested it with Kioxia U.2 so perhaps it could be some incompatibility between Kioxia and B650D4U, or it could just be a general issue with all U.2 drives, we're not really sure.

@advinservers my new server is build with Asrock Rack EPYC4000D4U !

Howdy, Stranger!

Categories

In this Discussion

Getting Slow Disk Speed on RAID-1 (Only ~200 MB/s) – Anyone Else Facing This?

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Getting Slow Disk Speed on RAID-1 (Only ~200 MB/s) – Anyone Else Facing This?

Comments