1

We have rent some servers from Hetzner.de. A few of them have NVMe hard disks while the others have SSDs. We becnchmarked the read/write performance on 4 of our servers using the following commands:

fio
dd
hdparm

The operating system is CentOS7 and each server has two hard disks with software Raid 1. All servers are localted in Hetzner datacenter.

Disks brands:

SSD:
    Model Family:     Samsung based SSDs
    Device Model:     SAMSUNG MZ7LM240HCGR-00003
NVMe:
    Model Number:                       THNSN5512GPU7 TOSHIBA
    Serial Number:                      Z62S101OTUHV

Here are the benchmark results:

Server1(NVMe):

Base Board Information
        Manufacturer: FUJITSU
        Product Name: D3417-B1
        Version: S26361-D3417-B1

# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Run status group 0 (all jobs):
   READ: bw=45.9MiB/s (48.1MB/s), 45.9MiB/s-45.9MiB/s (48.1MB/s-48.1MB/s), io=3070MiB (3219MB), run=66884-66884msec
  WRITE: bw=15.3MiB/s (16.1MB/s), 15.3MiB/s-15.3MiB/s (16.1MB/s-16.1MB/s), io=1026MiB (1076MB), run=66884-66884msec

Disk stats (read/write):
    md127: ios=785293/276106, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=393078/273876, aggrmerge=3/9643, aggrticks=330689/2134457, aggrin_queue=2467357, aggrutil=63.84%
  nvme0n1: ios=410663/273879, merge=7/9640, ticks=257384/2054071, in_queue=2311731, util=55.06%
  nvme1n1: ios=375494/273874, merge=0/9647, ticks=403994/2214844, in_queue=2622983, util=63.84%

#dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 14.2603 s, 75.3 MB/s

# hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   29320 MB in  1.98 seconds = 14818.11 MB/sec
 Timing buffered disk reads: 2660 MB in  3.00 seconds = 886.22 MB/sec
------------------------------------------------------------
------------------------------------------------------------
Server2(NVMe):

Base Board Information
        Manufacturer: FUJITSU
        Product Name: D3417-B1
        Version: S26361-D3417-B1

# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Run status group 0 (all jobs):
   READ: io=3072.2MB, aggrb=40296KB/s, minb=40296KB/s, maxb=40296KB/s, mint=78069msec, maxt=78069msec
  WRITE: io=1023.9MB, aggrb=13429KB/s, minb=13429KB/s, maxb=13429KB/s, mint=78069msec, maxt=78069msec

Disk stats (read/write):
    md1: ios=786339/298554, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=393673/300844, aggrmerge=0/0, aggrticks=543418/2294840, aggrin_queue=2838462, aggrutil=65.25%
  nvme0n1: ios=180052/300844, merge=0/0, ticks=480768/1879827, in_queue=2360788, util=56.22%
  nvme1n1: ios=607294/300844, merge=0/0, ticks=606068/2709853, in_queue=3316136, util=65.25%

#dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 33.2734 s, 32.3 MB/s

# hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   33788 MB in  1.99 seconds = 16977.90 MB/sec
 Timing buffered disk reads: 2204 MB in  3.00 seconds = 734.34 MB/sec

------------------------------------------------------------
------------------------------------------------------------
Server3(SSD)
Base Board Information
        Manufacturer: ASUSTeK COMPUTER INC.
        Product Name: Z10PA-U8 Series
        Version: Rev 1.xx

# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Run status group 0 (all jobs):
   READ: bw=262MiB/s (275MB/s), 262MiB/s-262MiB/s (275MB/s-275MB/s), io=3070MiB (3219MB), run=11718-11718msec
  WRITE: bw=87.6MiB/s (91.8MB/s), 87.6MiB/s-87.6MiB/s (91.8MB/s-91.8MB/s), io=1026MiB (1076MB), run=11718-11718msec

Disk stats (read/write):
    md2: ios=769518/258504, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=392958/263227, aggrmerge=9/864, aggrticks=219931/33550, aggrin_queue=253441, aggrutil=99.06%
  sda: ios=402306/263220, merge=12/871, ticks=222960/35975, in_queue=258904, util=99.04%
  sdb: ios=383611/263234, merge=7/857, ticks=216902/31125, in_queue=247978, util=99.06%

#dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 5.19855 s, 207 MB/s

# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   22452 MB in  1.99 seconds = 11258.90 MB/sec
 Timing buffered disk reads: 1546 MB in  3.00 seconds = 514.90 MB/sec

------------------------------------------------------------
------------------------------------------------------------
Server4(SSD)
Base Board Information
        Manufacturer: FUJITSU
        Product Name: D3401-H2
        Version: S26361-D3401-H2

# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Run status group 0 (all jobs):
   READ: io=3073.6MB, aggrb=61065KB/s, minb=61065KB/s, maxb=61065KB/s, mint=51539msec, maxt=51539msec
  WRITE: io=1022.5MB, aggrb=20315KB/s, minb=20315KB/s, maxb=20315KB/s, mint=51539msec, maxt=51539msec

Disk stats (read/write):
    md2: ios=784514/278548, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=392439/266239, aggrmerge=1246/13570, aggrticks=822829/716748, aggrin_queue=1539532, aggrutil=91.82%
  sda: ios=421561/266337, merge=1030/13473, ticks=867321/639461, in_queue=1506738, util=91.82%
  sdb: ios=363317/266142, merge=1463/13667, ticks=778338/794035, in_queue=1572326, util=91.76%

# dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=dsync
1073741824 bytes (1.1 GB) copied, 10.6605 s, 101 MB/s

# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   33686 MB in  1.98 seconds = 16985.97 MB/sec
 Timing buffered disk reads: 1304 MB in  3.00 seconds = 434.34 MB/sec

As it can be seen from the results, the fio command shows disappointing read/write results for server1(NVMe). But it shows better results for server2(NVMe) in comparison to the SSDs. The dd command shows disappointing read/write results for both NVMe servers compared to the SSD ones. The hdparm command also shows almost the same results for read/write performance for all servers.

All tests where done on the off peak times where the average load of a server was 0.0 .

Another strange issue we are facing in our NVMe servers is high I/O load while we are restoring the backup of an account or even restoring a zip file. For example, if I restore a zip file with the size of 150MB, the average server load goes beyond 20 after it got unzipped completely and it is directly because of the I/O wait (from the 'top' command).

We wonder to know what causes the NVMes to have such a disappoiting performance in contrast with the SSDs? Do creating software or hardware raids affect the NVMes performce in which the read/write perfomance gets worst than SSDs? If it is, so why SSDs are working almost great with software or hardware raids?

  • Please post the brand and specific model of the NMVe drives. – shodanshok Jan 1 at 17:03
  • I edited my post and added hard disks brands and model – Sinai Jan 1 at 17:34
  • 1
    Can you try a fio and dd (with bs=1M) run directly on the raw devices themselves? Be aware that this will overwrite data, so don't do that if you have valuable data on the drives. – shodanshok Jan 1 at 22:00
  • @shodanshok, unfortunately we cannot do any tests on raw devices! Because the servers are production and not development. – Sinai Jan 2 at 15:12
  • If so, you can't do much to debug the issue, short of renting another, empty server and run the tests on it. – shodanshok Jan 2 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.