New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How to test NVMe performance under linux
Hi,
I recently got a new dedi with two samsung sm961 NVMe SSDs and I wated to test the write/ read performance.
I'm using a Software Raid 1.
According to benchmarks, this SSD should write with 2.7 GB/s and read with 1.7 GB/s.
I have tested the write performance with the following command, but I "only" got around 700MB/s
dd if=/dev/zero of=testfile1 bs=1G count=1 oflag=direct 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.48908 s, 721 MB/s
Did I do something wrong and is this even the correct way to detect the write performance?
Regards
Comments
https://www.cyberciti.biz/tips/how-fast-is-linux-sata-hard-disk.html
https://www.shellhacks.com/disk-speed-test-read-write-hdd-ssd-perfomance-linux/
1GB really is not big enough for NVMe
I think you want something more like:
Break it after a few seconds.
Thank you @AnthonySmith
dd if=/dev/md2 of=/dev/null bs=4096k iflag=direct ^C5371+0 records in 5370+0 records out 22523412480 bytes (23 GB, 21 GiB) copied, 8.63151 s, 2.6 GB/s
But if I write e.g. 40GB with my previous command, the speed is much lower.
dd if=/dev/zero of=testfile1 bs=1G count=40 oflag=direct 40+0 records in 40+0 records out 42949672960 bytes (43 GB, 40 GiB) copied, 34.3405 s, 1.3 GB/s
I still cant explain why the results vary so much.
@v3ng
OS caches. Plus writing all zeroes will strongly bend the result and not show the true performance. Note that the dd test is only good for a first rudimentary impression.
You could download my vpsbenchmark linked in my signature. Your situation is one of the reasons I wrote it. Read the short docu and you'll find that you can also run it as a disk test only.
vpsbench does both sequential and random access tests each reading and writing performance. Due to the fine granularity of timing you get solid results even with a small sample (you don't want to waste lots of lifetime of your NVMe just for benchmarking).
Usage is very simple. Just download the binary for your system (probably linux 64 bit), have a quick look at the docu and run it.
You are aware that the first test is a read test, and the second is a write test right? Read tests are generally faster.
bs=1G ... uhm
Ugh, of course..
So which command should I use for write tests? There are again pretty large differences, probably because of the filesize.
Which size is the most realistic one?
sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.835828 s, 1.3 GB/s
dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.4278 s, 752 MB/s
Realistic? 4k; but you'll only really be seeing your iops.
Is this value too high?
don't use dd at all. use fio and run tests for different usual blocksizes like 4k and 64k.
PS: if that's with a new hetzner box from their no setup-fee sale, please post a full smartctl on those nvme thingies :-)
As per your request:
FIO
hey, thanks a lot!!
You're welcome. I think you might not resist when keeping reading these posts
At least posting a test is the only useful thing that I can do with my idle server collection atm :-D
Sure, here you go
Hetzner AX160-NVMe, CentOS 7.5, Both NVMe use same model SAMSUNG MZQLW960HMJP-00003
Detail DISK for who needed
Test FIO Mixed Read/Write
Test FIO Read Only
Test FIO Write Only
What tool/command did you use to make this hardware overview + disk & bandwidth benchmarks?
wget -qO- bench.sh | bash
do you have an URL for bench.sh? it looks really nice
Guess what, it's https://bench.sh/ ; I'd be wary of piping something right into bash without further inspection as general practice, but to each their own
EDIT: btw I'd be wary of those I/O tests (dd based) as well, I'd second fio as per Falzo's suggestion
I recommend my Vpsbench tool (see my sig). Not because I wrote it but because it DOES cope with NVMe and reports sensible results. I know because I DID test NVMe VPS and dedis and it worked fine.
Note: Normally Vpsbench offers an all in one (like @jovpov's screen) but using a command line switch one also can test/benchmark only a disk. It's documented in a short but completed manual.
Thermal throttling is a huge concern for NVME disks, and can result in 50%+ performance throttle during long writes.