2

I've just installed two Samsung 970 EVO Plus drives, and configured the biggest partition as RAID 10 (far 2 layout). The problem is FIO only reports a read speed of 72 MB/s. I used the following command to build the array;

mdadm --create --verbose --level=10 --metadata=1.2 --chunk=512 --raid-devices=2 --layout=f2 /dev/md/MyRAID10Array /dev/nvme0n1p3 /dev/nvme1n1p3

This should give me pretty close to double the quoted read speed of 3500 MB/s. I've formatted this with the default options for XFS,

mkfs.xfs /dev/md127

The disks are laid out as

fdisk -l /dev/nvme0n1

Output:

Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 970 EVO Plus 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: FF148851-E4A9-41D2-96C6-418280164BC6

Device            Start        End    Sectors   Size Type
/dev/nvme0n1p1     2048     821247     819200   400M Linux RAID
/dev/nvme0n1p2   821248   67930111   67108864    32G Linux swap
/dev/nvme0n1p3 67930112 1953523711 1885593600 899.1G Linux RAID

mdstat reports

md127 : active raid10 nvme1n1p3[0] nvme0n1p33 942664704 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU] bitmap: 0/8 pages [0KB], 65536KB chunk

but when I run this through FIO,

; Random read of 128 MB of data

[random-read]
rw=randread
size=128m
directory=/backup_mount_point

I only get really bad figures;

random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1)
random-read: (groupid=0, jobs=1): err= 0: pid=214183: Sun Oct  1 06:28:06 2023
  read: IOPS=18.2k, BW=71.3MiB/s (74.7MB/s)(128MiB/1796msec)
    clat (usec): min=14, max=6880, avg=54.09, stdev=53.44
     lat (usec): min=14, max=6880, avg=54.12, stdev=53.45
    clat percentiles (usec):
     |  1.00th=[   51],  5.00th=[   51], 10.00th=[   51], 20.00th=[   52],
     | 30.00th=[   52], 40.00th=[   53], 50.00th=[   53], 60.00th=[   53],
     | 70.00th=[   54], 80.00th=[   55], 90.00th=[   57], 95.00th=[   58],
     | 99.00th=[   75], 99.50th=[   76], 99.90th=[   78], 99.95th=[   80],
     | 99.99th=[  123]
   bw (  KiB/s): min=70720, max=74920, per=100.00%, avg=73509.33, stdev=2415.69, samples=3
   iops        : min=17680, max=18730, avg=18377.33, stdev=603.92, samples=3
  lat (usec)   : 20=0.03%, 50=0.02%, 100=99.93%, 250=0.02%
  lat (msec)   : 10=0.01%
  cpu          : usr=0.39%, sys=9.25%, ctx=32778, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=71.3MiB/s (74.7MB/s), 71.3MiB/s-71.3MiB/s (74.7MB/s-74.7MB/s), io=128MiB (134MB), run=1796-1796msec

Disk stats (read/write):
    md127: ios=30225/0, merge=0/0, ticks=1407/0, in_queue=1407, util=93.46%, aggrios=16384/0, aggrmerge=0/0, aggrticks=784/0, aggrin_queue=784, aggrutil=93.19%
  nvme0n1: ios=16384/0, merge=0/0, ticks=768/0, in_queue=768, util=93.04%
  nvme1n1: ios=16384/0, merge=0/0, ticks=800/0, in_queue=800, util=93.19%

If I move to sequential (rw=read) it improves to 1803 MB/s, but it is still less than a third of what I'd expect. I'm running Arch Linux on AMD Ryzen system with 64 GB RAM. The motherboard is an MSI X570S Edge Wifi Max.

Edit; After Batistuta9's answer, I changed my tests and got a significantly faster result. Here's the FIO summary in case that's of interest;

random-read: (groupid=0, jobs=1): err= 0: pid=143332: Thu Oct  5 23:23:49 2023
  read: IOPS=15.0k, BW=58.7MiB/s (61.5MB/s)(20.0GiB/348908msec)
    clat (usec): min=48, max=2746, avg=66.21, stdev=16.90
     lat (usec): min=48, max=2746, avg=66.24, stdev=16.90
    clat percentiles (usec):
     |  1.00th=[   49],  5.00th=[   50], 10.00th=[   51], 20.00th=[   51],
     | 30.00th=[   52], 40.00th=[   54], 50.00th=[   69], 60.00th=[   72],
     | 70.00th=[   74], 80.00th=[   81], 90.00th=[   88], 95.00th=[   91],
     | 99.00th=[  120], 99.50th=[  130], 99.90th=[  149], 99.95th=[  157],
     | 99.99th=[  184]
   bw (  KiB/s): min=57064, max=61112, per=14.93%, avg=60131.86, stdev=374.32, samples=697
   iops        : min=14266, max=15278, avg=15032.97, stdev=93.58, samples=697
  lat (usec)   : 50=6.97%, 100=90.22%, 250=2.81%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.74%, sys=5.61%, ctx=5246560, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143333: Thu Oct  5 23:23:49 2023
  read: IOPS=15.2k, BW=59.3MiB/s (62.2MB/s)(20.0GiB/345469msec)
    clat (usec): min=48, max=2828, avg=65.55, stdev=16.90
     lat (usec): min=48, max=2828, avg=65.58, stdev=16.90
    clat percentiles (usec):
     |  1.00th=[   49],  5.00th=[   50], 10.00th=[   51], 20.00th=[   51],
     | 30.00th=[   52], 40.00th=[   53], 50.00th=[   60], 60.00th=[   71],
     | 70.00th=[   73], 80.00th=[   79], 90.00th=[   88], 95.00th=[   90],
     | 99.00th=[  120], 99.50th=[  130], 99.90th=[  149], 99.95th=[  157],
     | 99.99th=[  184]
   bw (  KiB/s): min=59472, max=61528, per=15.08%, avg=60729.74, stdev=344.11, samples=690
   iops        : min=14868, max=15382, avg=15182.43, stdev=86.02, samples=690
  lat (usec)   : 50=6.62%, 100=90.63%, 250=2.75%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.76%, sys=5.63%, ctx=5246568, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143334: Thu Oct  5 23:23:49 2023
  read: IOPS=14.3k, BW=55.8MiB/s (58.5MB/s)(20.0GiB/366867msec)
    clat (usec): min=48, max=2742, avg=69.63, stdev=16.68
     lat (usec): min=48, max=2742, avg=69.66, stdev=16.69
    clat percentiles (usec):
     |  1.00th=[   50],  5.00th=[   50], 10.00th=[   51], 20.00th=[   52],
     | 30.00th=[   54], 40.00th=[   70], 50.00th=[   72], 60.00th=[   73],
     | 70.00th=[   75], 80.00th=[   86], 90.00th=[   88], 95.00th=[   92],
     | 99.00th=[  123], 99.50th=[  133], 99.90th=[  151], 99.95th=[  159],
     | 99.99th=[  184]
   bw (  KiB/s): min=55232, max=58272, per=14.20%, avg=57191.21, stdev=322.36, samples=733
   iops        : min=13808, max=14568, avg=14297.80, stdev=80.58, samples=733
  lat (usec)   : 50=3.98%, 100=92.88%, 250=3.14%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.74%, sys=5.33%, ctx=5246807, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143335: Thu Oct  5 23:23:49 2023
  read: IOPS=12.6k, BW=49.2MiB/s (51.6MB/s)(20.0GiB/416494msec)
    clat (usec): min=56, max=2742, avg=79.09, stdev=12.05
     lat (usec): min=56, max=2742, avg=79.12, stdev=12.05
    clat percentiles (usec):
     |  1.00th=[   64],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],
     | 30.00th=[   73], 40.00th=[   73], 50.00th=[   74], 60.00th=[   78],
     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   90], 95.00th=[   95],
     | 99.00th=[  128], 99.50th=[  137], 99.90th=[  155], 99.95th=[  163],
     | 99.99th=[  190]
   bw (  KiB/s): min=48912, max=51584, per=12.51%, avg=50377.18, stdev=326.60, samples=832
   iops        : min=12228, max=12896, avg=12594.28, stdev=81.63, samples=832
  lat (usec)   : 100=96.05%, 250=3.95%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.64%, sys=4.86%, ctx=5247122, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143336: Thu Oct  5 23:23:49 2023
  read: IOPS=12.6k, BW=49.2MiB/s (51.6MB/s)(20.0GiB/416501msec)
    clat (usec): min=57, max=2889, avg=79.08, stdev=12.27
     lat (usec): min=57, max=2889, avg=79.12, stdev=12.27
    clat percentiles (usec):
     |  1.00th=[   64],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],
     | 30.00th=[   73], 40.00th=[   73], 50.00th=[   74], 60.00th=[   78],
     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   90], 95.00th=[   95],
     | 99.00th=[  128], 99.50th=[  137], 99.90th=[  155], 99.95th=[  163],
     | 99.99th=[  188]
   bw (  KiB/s): min=48640, max=51608, per=12.51%, avg=50376.18, stdev=314.71, samples=832
   iops        : min=12160, max=12902, avg=12594.03, stdev=78.66, samples=832
  lat (usec)   : 100=96.08%, 250=3.92%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.68%, sys=4.83%, ctx=5247298, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143337: Thu Oct  5 23:23:49 2023
  read: IOPS=12.6k, BW=49.3MiB/s (51.7MB/s)(20.0GiB/415249msec)
    clat (usec): min=14, max=3163, avg=78.85, stdev=12.35
     lat (usec): min=14, max=3163, avg=78.88, stdev=12.35
    clat percentiles (usec):
     |  1.00th=[   60],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],
     | 30.00th=[   73], 40.00th=[   73], 50.00th=[   74], 60.00th=[   77],
     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   90], 95.00th=[   95],
     | 99.00th=[  128], 99.50th=[  137], 99.90th=[  153], 99.95th=[  161],
     | 99.99th=[  190]
   bw (  KiB/s): min=49104, max=51799, per=12.54%, avg=50526.68, stdev=317.81, samples=830
   iops        : min=12276, max=12949, avg=12631.66, stdev=79.43, samples=830
  lat (usec)   : 20=0.01%, 50=0.04%, 100=96.03%, 250=3.93%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.61%, sys=4.90%, ctx=5246732, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143338: Thu Oct  5 23:23:49 2023
  read: IOPS=12.7k, BW=49.5MiB/s (51.9MB/s)(20.0GiB/413718msec)
    clat (usec): min=48, max=2834, avg=78.55, stdev=12.65
     lat (usec): min=48, max=2834, avg=78.59, stdev=12.65
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],
     | 30.00th=[   72], 40.00th=[   73], 50.00th=[   74], 60.00th=[   77],
     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   89], 95.00th=[   95],
     | 99.00th=[  127], 99.50th=[  137], 99.90th=[  153], 99.95th=[  161],
     | 99.99th=[  190]
   bw (  KiB/s): min=49776, max=51912, per=12.59%, avg=50713.30, stdev=309.76, samples=827
   iops        : min=12444, max=12978, avg=12678.31, stdev=77.41, samples=827
  lat (usec)   : 50=0.20%, 100=95.92%, 250=3.87%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.65%, sys=4.89%, ctx=5246845, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
random-read: (groupid=0, jobs=1): err= 0: pid=143339: Thu Oct  5 23:23:49 2023
  read: IOPS=13.3k, BW=52.0MiB/s (54.5MB/s)(20.0GiB/394165msec)
    clat (usec): min=47, max=2904, avg=74.83, stdev=15.26
     lat (usec): min=47, max=2904, avg=74.86, stdev=15.26
    clat percentiles (usec):
     |  1.00th=[   50],  5.00th=[   51], 10.00th=[   53], 20.00th=[   70],
     | 30.00th=[   71], 40.00th=[   72], 50.00th=[   73], 60.00th=[   75],
     | 70.00th=[   85], 80.00th=[   87], 90.00th=[   89], 95.00th=[   94],
     | 99.00th=[  126], 99.50th=[  135], 99.90th=[  153], 99.95th=[  161],
     | 99.99th=[  188]
   bw (  KiB/s): min=52096, max=54312, per=13.21%, avg=53229.10, stdev=312.71, samples=788
   iops        : min=13024, max=13578, avg=13307.26, stdev=78.16, samples=788
  lat (usec)   : 50=1.65%, 100=94.73%, 250=3.62%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.67%, sys=5.07%, ctx=5246945, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=393MiB/s (412MB/s), 49.2MiB/s-59.3MiB/s (51.6MB/s-62.2MB/s), io=160GiB (172GB), run=345469-416501msec

Disk stats (read/write):
    md127: ios=41940854/14, merge=0/0, ticks=2850618/100, in_queue=2850718, util=100.00%, aggrios=20971520/47, aggrmerge=0/2, aggrticks=1428779/139, aggrin_queue=1428932, aggrutil=100.00%
  nvme0n1: ios=21179809/47, merge=0/2, ticks=1427626/139, in_queue=1427779, util=100.00%
  nvme1n1: ios=20763231/47, merge=0/2, ticks=1429932/139, in_queue=1430086, util=100.00%
Improve this question
7
  • Why do not create ZFS tool on partitions and set as mirror directly?
    – Romeo Ninov
    Commented Oct 1, 2023 at 6:22
  • I'm using XFS not ZFS
    – James
    Commented Oct 1, 2023 at 12:44
  • I've confirmed both drives are running at pci3 x4. 8Tb/s
    – James
    Commented Oct 1, 2023 at 12:47
  • Increased the speed of the sequential reads, but the random read still at 72.
    – James
    Commented Oct 1, 2023 at 12:48
  • 1
    RAID10 on 2 devices? What advantage does that have vs. RAID1? It that an attempt to get different wear patterns on the different devices, since replicated blocks go to different offsets?
    – Peter Cordes
    Commented Oct 1, 2023 at 19:58

1 Answer 1

Reset to default
6

You got the expected result for single thread 4k block size performance benchmark with queue depth 1. Check the drive’s specification here - https://semiconductor.samsung.com/consumer-storage/internal-ssd/970evoplus/ . For such patterns the result should be up to 19000 IOPS (you got 18200 IOPS). Add the following parameters in fio benchmark file (random read) and rerun test:

numjobs=8
iodepth=64
direct=1
ioengine=libaio

I’d also suggest to increase the size of benchmarking file from 128MB to at least 20GB.

Improve this answer
2
  • 7
    And not play too much with tests: we once worn 6% drive's life with such tests. Remember, those are consumer drives.
    – Nikita Kipriyanov
    Commented Oct 1, 2023 at 19:06
  • 1
    Reads obviously won’t wear out the SSD, hammering it with 4K random writes in a big volume WILL.
    – BaronSamedi1958
    Commented Oct 24, 2023 at 13:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .