New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
NVMe Speed Issue :: CentOS vs Windows
Mahfuz_SS_EHL
Host Rep, Veteran
in Help
Hello,
I'm facing NVMe speed difference between CentOS & Windows. In windows, it's pulling ~3.5 GB/s Read & ~2 GB/s Write.
But, when it's CentOS 7/CentOS 8, it's pulling only 1.2~1.3GB/s write.
[root@localhost ~]# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 0.997785 s, 1.1 GB/s
The configuration is: AMD Ryzen 3700X + Asrock Rack X470D4U + 32GB Ram + NVMe Adapter (PCIe x16).
Can anyone shed some light if I'm making any mistake ?
Regards.
Comments
Hwo about debian 11?
Or maybe use yabs.sh/fio as the software as standard disk benchmark.
You must know that different software give different result as the method of measurement is different.
Filesystem?
Different size of bytes
Uh... sort of like different weights of 1 kg?
here is an idea. do the dd or yabs on windows too, using wsl.
NTFS in Windows, XFS in CentOS
Probably not. It's the same.
Centos 7-8 has too old kernels to see full potential of your ryzen maybe for the nvme too (all drivers are in the kernel) try Ubuntu 20.4.3 with hwe kernel it is 5.11 or proxmox
https://askubuntu.com/questions/248914/what-is-hardware-enablement-hwe
Yabs provides 500+700 MB, so around 1.2GB/s. I'll post the benchmark as soon as I'm free.
Which one? 4k? 64k? 512k? Or 1m?
Try some other OS like Debian 11 or Ubuntu 20.04/21.04 - Centos has old garbage kernel 4.x https://repology.org/project/linux/versions
Try ext4 rather than XFS.
while I get the joke, he is right ;-)
testing 1M blocksize in windows vs 64k in linux obviously leads to different results, if you keep in mind that bandwidth is the result of iops*blocksize ...
So, what will be the bs & count size if I want to match it to Windows ?? I thought he's talking about Allocation unit of the partition.
I haven't used crystaldiskmark in a while, but from your screenshot seems you are using a testfile of 1 GB size and the blocksize for the first sequential test is 1MB
should be closest to that (1M blocksize and a 1GB testfile)
What Falzo says is correct, you are testing with different blocksizes. Apart from that, I also observed that dd returns worse results for "small" files (1 GB is small in that case) with a fast storage backend, also under CentOS 7.
I invested some time to find the cause of this, but could not figure it out and I can't remember the exact details either. However, I believe that there is a short delay when dd is started where some pre stuff is done, which is counted as runtime. With such fast storages as with NVMe and with such small test files this has a big effect on the result.
If you let dd write a larger file (with 1M blocksize) you should get a similar result as under Windows. By larger file I mean e.g. 10 GB.
yes definitely possible, as with small files you are in a sub-second field already, so this is a good suggestion. for comparison it seems reasonable to also use a larger testfile in windows as well. might also help with caching related things, though I haven't checked, how crystaldiskmark handles that.
always difficult to compare different benchmarks anyway. maybe use fio und centos instead of dd as I have found it more reliable or at least more 'tunable' to replicate the same settings somehow.
bs=1M count=1K pulls the same as 1.2GB/s. But, if I increase the filesize then sequentially speed increases too. I have got fio. But the command I found is:
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
I think I need to change the stats here to match with CrystalMark (SEQ1M Q8T1 is interpreted as Sequential Test for 1 Mebibyte block size data with total of 8 tasks in sequence on thread 1.)
yeah you're getting closer.
with fio obviously --bs=1M would represent the correct blocksize and --size=1GB the overall testfile size. you don't need 16 jobs, with fio they run in parallel but I think Q8T1 in CDM means it breaks down it's test in 8 parts that still run one after another (1 thread), so you'd rather want only one job. yet if CPU power plays a role windows could still handle that differently - I don't know.
however deep you dive into it, IMHO your takeaway should be: do not overengineer it!
the nvme speeds won't change just because of the os. there might be slight differences depending on filesystem and such, but I'd say these are rather negligible.
everything you see in the benchmarks that comes across as big difference reflects an artificial combination of different settings, and as said before even caching could play a role...
also maybe try vpsbench if you want to see linux beat windows anyway.
...sorry @jsg , but I simply couldn't resist ;-)
Actually the motherboard was having NVMe Slots of PCIe Gen3 x2 & PCIe Gen2 x4. I wanted to use PCIe Gen3 x4. That's why I managed adapters. But, then got confused at this. Now, this is clear to me. Thanks for helping out
(a) "HWE" is mainly about stuff that actually needs newer or specific driver like graphics cards.
(b) For NVMe there is a standard driver because NVMe, unlike e.g. graphics cards, is a standard. So, unless one is on a really old kernel like 2.6 there is no need at all to worry about v.5.x vs v.4x
Makes no sense on diverse levels. For one the testing methods are totally different (e.g. read+write in 1 test vs. read test + write test), plus Windows and Unix/linux are totally different beasts on many levels.
Are you sure that CrystalMark reads/writes sync?
Anyway, I think you nailed it well by hinting at the diverse differences between benchmarks (not to even talk between Windows and Unix/linux).
G3x2 is roughly equal to G2x4. Btw in such a case you should go with G2x4 because in case the NVMe itself is G2 (unlikely but it could be) it will at least see - and use - 4 lanes.
Finally a piece of advice: use whatever OS you'll run in production for benchmarking. Unless you plan to use Windows don't waste time on Windows benchmarks. Also don't be overly concerned about fine details, because you don't know how VM users (and the software they use) will use the system anyway. A database for example has very different needs and ways to operate than say some kind of a file server.
As a provider you should just care about no test (e.g. random writing) being particularly bad plus learn valuable information for your node caching strategy.