fio to test I/O

So looking for a better way to get a look at I/O other than using dd and ioping I came across https://www.linux.com/learn/tutorials/442451-inspecting-disk-io-performance-with-fio/ in a google search, so gave it a try. My test bed is the newest node in Los Angeles which contains an SSD for readcache using flashcache

I made two passes to make sure the data was cached to see what help flashcache was

Test 1

[root@e3la17 data]# fio ./random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MB)
Jobs: 1 (f=1): [r] [100.0% done] [16084K/0K /s] [3927 /0  iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=465974
  read : io=131072KB, bw=16531KB/s, iops=4132 , runt=  7929msec
    clat (usec): min=96 , max=14014 , avg=237.62, stdev=674.84
     lat (usec): min=96 , max=14014 , avg=237.83, stdev=674.83
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  108], 10.00th=[  110], 20.00th=[  113],
     | 30.00th=[  116], 40.00th=[  117], 50.00th=[  119], 60.00th=[  121],
     | 70.00th=[  124], 80.00th=[  129], 90.00th=[  135], 95.00th=[  183],
     | 99.00th=[ 4256], 99.50th=[ 5216], 99.90th=[ 6688], 99.95th=[ 7520],
     | 99.99th=[ 8896]
    bw (KB/s)  : min=12414, max=18392, per=100.00%, avg=16529.80, stdev=1481.70
    lat (usec) : 100=0.01%, 250=96.14%, 500=0.14%, 750=0.13%, 1000=0.21%
    lat (msec) : 2=0.65%, 4=1.57%, 10=1.15%, 20=0.01%
  cpu          : usr=1.97%, sys=8.87%, ctx=33212, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=131072KB, aggrb=16530KB/s, minb=16530KB/s, maxb=16530KB/s, mint=7929msec, maxt=7929msec

Disk stats (read/write):
    dm-3: ios=31990/18, merge=0/0, ticks=7358/0, in_queue=7363, util=93.90%, aggrios=16384/16393, aggrmerge=0/0, aggrticks=1070/2522, aggrin_queue=3558, aggrutil=61.93%
    dm-2: ios=32768/18, merge=0/0, ticks=2141/0, in_queue=2141, util=26.65%, aggrios=32768/10, aggrmerge=0/10, aggrticks=2140/0, aggrin_queue=2134, aggrutil=26.57%
  sda: ios=32768/10, merge=0/10, ticks=2140/0, in_queue=2134, util=26.57%
  sdb: ios=1/32768, merge=0/0, ticks=0/5044, in_queue=4975, util=61.93%

Test 2

[root@e3la17 data]# fio ./random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [21258K/0K /s] [5190 /0  iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=466094
  read : io=131072KB, bw=20785KB/s, iops=5196 , runt=  6306msec
    clat (usec): min=57 , max=1100 , avg=188.92, stdev=21.62
     lat (usec): min=57 , max=1101 , avg=189.14, stdev=21.63
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  169], 10.00th=[  169], 20.00th=[  171],
     | 30.00th=[  175], 40.00th=[  179], 50.00th=[  199], 60.00th=[  201],
     | 70.00th=[  203], 80.00th=[  209], 90.00th=[  211], 95.00th=[  213],
     | 99.00th=[  229], 99.50th=[  241], 99.90th=[  262], 99.95th=[  286],
     | 99.99th=[  326]
    bw (KB/s)  : min=20224, max=21032, per=100.00%, avg=20788.67, stdev=204.49
    lat (usec) : 100=0.99%, 250=98.83%, 500=0.17%
    lat (msec) : 2=0.01%
  cpu          : usr=1.68%, sys=9.69%, ctx=33161, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=131072KB, aggrb=20785KB/s, minb=20785KB/s, maxb=20785KB/s, mint=6306msec, maxt=6306msec

Disk stats (read/write):
    dm-3: ios=31953/0, merge=0/0, ticks=5779/0, in_queue=5779, util=92.54%, aggrios=16384/0, aggrmerge=0/0, aggrticks=2962/0, aggrin_queue=2961, aggrutil=92.39%
    dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/3, aggrmerge=0/1, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
  sda: ios=0/3, merge=0/1, ticks=0/0, in_queue=0, util=0.00%
  sdb: ios=32768/0, merge=0/0, ticks=5924/0, in_queue=5923, util=92.39%

So now I'm trying to make sense of it all, reading the man page and the article fully, any other opinions or are you familiar and can lend some explaining to the test?