Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How to debug IO delay
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How to debug IO delay

v3ngv3ng Member, Patron Provider

Hi,

I have a small SSD NAS (based on a Pi 4) which occasionally has a very high IO wait (20-40) and much lower write speed (to a samba share).

Even when the cache of the Crucial MX500 250GB is full it shouldn't drop below 200MB/s

The SSD is connected via SATA to USB and formatted as XFS as it seems to be the best performing FS.

Does anyone have an idea how to debug this?
It's not a network issue as iperf3 shows a constant 1Gbit/s between the Pi and my Mac

Thanks!

Comments

  • I don't think there is much to debug. from what can be seen in your screenshot it could be just a flush of (any) cache happening in that moment, therefore the operation slows down and have to wait for that being done. keep in mind that usually ssd devices come with some kind of internal cache as well which could add to that.

  • v3ngv3ng Member, Patron Provider

    @Falzo said:
    I don't think there is much to debug. from what can be seen in your screenshot it could be just a flush of (any) cache happening in that moment, therefore the operation slows down and have to wait for that being done. keep in mind that usually ssd devices come with some kind of internal cache as well which could add to that.

    I just tested the same SSD with the USB adapter on my Mac and it never dropped below 200MB/s.
    Even when writing over 100GB

  • my bets would still be on the cache though. probably done differently on your pi/linux vs your macbook. for instance could be that the pi's RAM is running full and needs to be flushed.

    could also be that another process is writing to a different drive/location and the cpu can't handle all that in parallel but needs to prioritize and therefore you see IOwait - but it is actually not waiting for your usb ssd but another storage.

    you could try running iotop to maybe catch that moment where it is dropping and see what other processes there might access something different, which queues the process to the USB drive.

  • MaxKVMMaxKVM Member, Host Rep

    You might be able to catch whatever this is using atop. Set the atop interval to 30sec/1min (or whatever your dstat interval is) and flag -t in dstat to show time, then check the atop log at the time of the iowait.

  • v3ngv3ng Member, Patron Provider

    @MaxKVM said:
    You might be able to catch whatever this is using atop. Set the atop interval to 30sec/1min (or whatever your dstat interval is) and flag -t in dstat to show time, then check the atop log at the time of the iowait.

    Thank you, do you notice something unusual?
    I took this screenshot during the transfer.

  • MaxKVMMaxKVM Member, Host Rep

    Nothing stands out. You can try looking at sorted disk stats with atop by starting it with atop -d (or simply type "d" if you are already in atop).

  • I'd consider this pretty much eaxactly what you coudl expect out of an SMB share on a Gbit network connection. 1Gbit equals roughly 120MB/s may and smb as protocol might add some overhead. dstat number might not be perfectly averaged out, depending on which numbers it pulls at what moment to do it's calculations... that's why you might see 133MB/s as well as 80 MB/s ... did you try a bigger interval like 5 seconds?
    there is no IOwait in these screenshots anymore anyway ;-)

  • I believe it's a limitation of the PI, just few days ago I saw a video about SATA vs USB3 performance on Pi 4 Compute module, and it mentioned something about speed dropping occasionally to speeds you are reporting

Sign In or Register to comment.