Anyone with RAID-1/ax101 willing to help me ?

Liso

Background

Long story short, I create this bench script written in shell that works similarly to yabs— except I employ additional feature and one of them being able to support multi disk layout.

So a user just reported issues with the script, he said there is a formatting problem. I have look into the code and first thing I thought was problem with loop control (this is the code if anyone want to take a look → https://github.com/L1so/benchy/blob/ec2649009f0e878304ef6f8b73e6ef843fe504ef/benchy#L477).

Later I realized I was wrong and actually the original code is correct, continue instead of continue 2 because the function is placed on outer loop and not nested one.

Actual problem

I decided to look back on the issues and found the culprit, it lies with RAID-1 configuration that the user was using in his hetzner ax101 dedi, it create a mirror of a disk to another disk so there was two disk hold the same data. My suspicion is confirmed when I ask the user to run a command to troubleshoot the problem.

Before I PM him, he also provide lsblk and df -h output, at that time I haven't noticed raid1 type from his lsblk.

---

With that being said, if anyone has access to a server with raid1 or ax101 itself, I kindly ask you to help me diagnose this problem. You can participate by running this command and post the output.

lsblk -nPpbo KNAME,SIZE,PKNAME,MOUNTPOINT,FSTYPE

With that, I will try eliminate the duplicate raid 1 disk with awk.

Yeah I can just pipe the output to this, but that feels like the wrong way and I probably will use that as last resort.

Thanks !

Falzo

I think this hasn't anything to do with the server type (or provider). It's simply how md/soft-raid works.
lsblk doesn't help much in that regard as it lists all block devices and that kind of raid sits on top of it.

here is an lsblk from a system using a mix of soft-raid (raid10) and zfs:

KNAME="/dev/loop0" SIZE="34359738368" PKNAME="" MOUNTPOINT="" FSTYPE="ext4"
KNAME="/dev/loop1" SIZE="34359738368" PKNAME="" MOUNTPOINT="" FSTYPE="ext4"
KNAME="/dev/sda" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sda1" SIZE="1073741824" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sda2" SIZE="493921239040" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sda3" SIZE="1048576" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sda4" SIZE="2218602922496" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE="zfs_member"
KNAME="/dev/sda5" SIZE="3287574715904" PKNAME="/dev/sda" MOUNTPOINT="/data/sda" FSTYPE="xfs"
KNAME="/dev/sdb" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sdb1" SIZE="1073741824" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sdb2" SIZE="493921239040" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sdb3" SIZE="1048576" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sdb4" SIZE="2218602922496" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE="zfs_member"
KNAME="/dev/sdb5" SIZE="3287574715904" PKNAME="/dev/sdb" MOUNTPOINT="/data/sdb" FSTYPE="xfs"
KNAME="/dev/sdc" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sdc1" SIZE="1073741824" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sdc2" SIZE="493921239040" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sdc3" SIZE="1048576" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sdc4" SIZE="2218602922496" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE="zfs_member"
KNAME="/dev/sdc5" SIZE="3287574715904" PKNAME="/dev/sdc" MOUNTPOINT="/data/sdc" FSTYPE="xfs"
KNAME="/dev/sdd" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sdd1" SIZE="1073741824" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sdd2" SIZE="493921239040" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE="linux_raid_member"
KNAME="/dev/sdd3" SIZE="1048576" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/sdd4" SIZE="2218602922496" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE="zfs_member"
KNAME="/dev/sdd5" SIZE="3287574715904" PKNAME="/dev/sdd" MOUNTPOINT="/data/sdd" FSTYPE="xfs"
KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sdd1" MOUNTPOINT="/boot" FSTYPE="ext3"
KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sdb1" MOUNTPOINT="/boot" FSTYPE="ext3"
KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sdc1" MOUNTPOINT="/boot" FSTYPE="ext3"
KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sda1" MOUNTPOINT="/boot" FSTYPE="ext3"
KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sdd2" MOUNTPOINT="/" FSTYPE="ext4"
KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sdb2" MOUNTPOINT="/" FSTYPE="ext4"
KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sdc2" MOUNTPOINT="/" FSTYPE="ext4"
KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sda2" MOUNTPOINT="/" FSTYPE="ext4"
KNAME="/dev/zd0" SIZE="68719476736" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/zd0p1" SIZE="1023410176" PKNAME="/dev/zd0" MOUNTPOINT="" FSTYPE="ext3"
KNAME="/dev/zd0p2" SIZE="2047868928" PKNAME="/dev/zd0" MOUNTPOINT="" FSTYPE="swap"
KNAME="/dev/zd0p3" SIZE="65646100480" PKNAME="/dev/zd0" MOUNTPOINT="" FSTYPE="ext4"
KNAME="/dev/zd16" SIZE="128849018880" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/zd16p1" SIZE="575668224" PKNAME="/dev/zd16" MOUNTPOINT="" FSTYPE="ntfs"
KNAME="/dev/zd16p2" SIZE="128270204928" PKNAME="/dev/zd16" MOUNTPOINT="" FSTYPE="ntfs"
KNAME="/dev/zd32" SIZE="128849018880" PKNAME="" MOUNTPOINT="" FSTYPE=""
KNAME="/dev/zd32p1" SIZE="510656512" PKNAME="/dev/zd32" MOUNTPOINT="" FSTYPE="ext3"
KNAME="/dev/zd32p2" SIZE="7999586304" PKNAME="/dev/zd32" MOUNTPOINT="" FSTYPE="swap"
KNAME="/dev/zd32p3" SIZE="63999836160" PKNAME="/dev/zd32" MOUNTPOINT="" FSTYPE="ext4"
KNAME="/dev/zram0" SIZE="6442450944" PKNAME="" MOUNTPOINT="[SWAP]" FSTYPE=""

I'd suspect the depending on the zpool config you can also run into troubles.

may I ask, why you don't parse mount output for usable/benchable dirs? ;-)

Liso

@Falzo said:
I think this hasn't anything to do with the server type (or provider). It's simply how md/soft-raid works.
lsblk doesn't help much in that regard as it lists all block devices and that kind of raid sits on top of it.

here is an lsblk from a system using a mix of soft-raid (raid10) and zfs:

KNAME="/dev/loop0" SIZE="34359738368" PKNAME="" MOUNTPOINT="" FSTYPE="ext4"
> KNAME="/dev/loop1" SIZE="34359738368" PKNAME="" MOUNTPOINT="" FSTYPE="ext4"
> KNAME="/dev/sda" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sda1" SIZE="1073741824" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sda2" SIZE="493921239040" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sda3" SIZE="1048576" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sda4" SIZE="2218602922496" PKNAME="/dev/sda" MOUNTPOINT="" FSTYPE="zfs_member"
> KNAME="/dev/sda5" SIZE="3287574715904" PKNAME="/dev/sda" MOUNTPOINT="/data/sda" FSTYPE="xfs"
> KNAME="/dev/sdb" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sdb1" SIZE="1073741824" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sdb2" SIZE="493921239040" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sdb3" SIZE="1048576" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sdb4" SIZE="2218602922496" PKNAME="/dev/sdb" MOUNTPOINT="" FSTYPE="zfs_member"
> KNAME="/dev/sdb5" SIZE="3287574715904" PKNAME="/dev/sdb" MOUNTPOINT="/data/sdb" FSTYPE="xfs"
> KNAME="/dev/sdc" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sdc1" SIZE="1073741824" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sdc2" SIZE="493921239040" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sdc3" SIZE="1048576" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sdc4" SIZE="2218602922496" PKNAME="/dev/sdc" MOUNTPOINT="" FSTYPE="zfs_member"
> KNAME="/dev/sdc5" SIZE="3287574715904" PKNAME="/dev/sdc" MOUNTPOINT="/data/sdc" FSTYPE="xfs"
> KNAME="/dev/sdd" SIZE="6001175126016" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sdd1" SIZE="1073741824" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sdd2" SIZE="493921239040" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE="linux_raid_member"
> KNAME="/dev/sdd3" SIZE="1048576" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/sdd4" SIZE="2218602922496" PKNAME="/dev/sdd" MOUNTPOINT="" FSTYPE="zfs_member"
> KNAME="/dev/sdd5" SIZE="3287574715904" PKNAME="/dev/sdd" MOUNTPOINT="/data/sdd" FSTYPE="xfs"
> KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sdd1" MOUNTPOINT="/boot" FSTYPE="ext3"
> KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sdb1" MOUNTPOINT="/boot" FSTYPE="ext3"
> KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sdc1" MOUNTPOINT="/boot" FSTYPE="ext3"
> KNAME="/dev/md0" SIZE="1071644672" PKNAME="/dev/sda1" MOUNTPOINT="/boot" FSTYPE="ext3"
> KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sdd2" MOUNTPOINT="/" FSTYPE="ext4"
> KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sdb2" MOUNTPOINT="/" FSTYPE="ext4"
> KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sdc2" MOUNTPOINT="/" FSTYPE="ext4"
> KNAME="/dev/md1" SIZE="987571945472" PKNAME="/dev/sda2" MOUNTPOINT="/" FSTYPE="ext4"
> KNAME="/dev/zd0" SIZE="68719476736" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/zd0p1" SIZE="1023410176" PKNAME="/dev/zd0" MOUNTPOINT="" FSTYPE="ext3"
> KNAME="/dev/zd0p2" SIZE="2047868928" PKNAME="/dev/zd0" MOUNTPOINT="" FSTYPE="swap"
> KNAME="/dev/zd0p3" SIZE="65646100480" PKNAME="/dev/zd0" MOUNTPOINT="" FSTYPE="ext4"
> KNAME="/dev/zd16" SIZE="128849018880" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/zd16p1" SIZE="575668224" PKNAME="/dev/zd16" MOUNTPOINT="" FSTYPE="ntfs"
> KNAME="/dev/zd16p2" SIZE="128270204928" PKNAME="/dev/zd16" MOUNTPOINT="" FSTYPE="ntfs"
> KNAME="/dev/zd32" SIZE="128849018880" PKNAME="" MOUNTPOINT="" FSTYPE=""
> KNAME="/dev/zd32p1" SIZE="510656512" PKNAME="/dev/zd32" MOUNTPOINT="" FSTYPE="ext3"
> KNAME="/dev/zd32p2" SIZE="7999586304" PKNAME="/dev/zd32" MOUNTPOINT="" FSTYPE="swap"
> KNAME="/dev/zd32p3" SIZE="63999836160" PKNAME="/dev/zd32" MOUNTPOINT="" FSTYPE="ext4"
> KNAME="/dev/zram0" SIZE="6442450944" PKNAME="" MOUNTPOINT="[SWAP]" FSTYPE=""
> 

I'd suspect the depending on the zpool config you can also run into troubles.

may I ask, why you don't parse mount output for usable/benchable dirs? ;-)

I'm trying to fetch following data: partition name, mountpoint, size, filesystem type. lsblk provide that four data while mount only has 3 of them, with no size information in the output (though I can easily feed the mountpoint to df to fetch its size).

If it's okay with you, can you show me mount output in your system ?

Falzo

@Liso said: If it's okay with you, can you show me mount output in your system ?

sure, same box:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16368564k,nr_inodes=4092141,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3285816k,mode=755)
/dev/md1 on / type ext4 (rw,relatime,stripe=256)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15698)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sdc5 on /data/sdc type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdb5 on /data/sdb type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/md0 on /boot type ext3 (rw,relatime)
/dev/sda5 on /data/sda type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
zfs on /opt type zfs (rw,noatime,xattr,noacl)
zfs/subvol-590-disk-0 on /opt/subvol-590-disk-0 type zfs (rw,noatime,xattr,posixacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3285812k,mode=700,uid=1000,gid=1000)
/dev/sdd5 on /data/sdd type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

PS: and yes, I understand the thing with the sizes in lsblk being convenient, however with things like raid-10 or even raid-5 raid-6 this would not work, as you'll only see the size of each block-device and not the mounted size. apart from that there possibly is also LVM or btrfs which allow for their own 'grouping' ...
what about parsing df -T ? otherwise yes, you'll need at least two requests for those four fields ;-)