1

I have a server with Ubuntu 20.04.1 LTS with NVME ssd devices running on raid1 and the hard drive operates so slow! to open a gzip file to of 500MB to a 3.7GB takes a while.. way more then it should. this is a development server just for me, so even when I use MariaDB, loading SQL dumps take around 30 minutes that when I try to load them locally on my home computer it takes several minutes, everything is slow! even upgrading ubuntu packages takes a long time!

so I gathered some specs:

Linux Kernel: 5.4.0-42-generic
CPU:  Intel(R) Xeon(R) D-2141I CPU @ 2.20GHz
Memory: 32GB
two WDC CL SN720 SDAQNTW-512G-2000 hard drives with software raid1 (nvme ssd)

and information from some commands

# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [linear] [multipath] [raid10]
md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
      523200 blocks [2/2] [UU]

md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0]
      498530240 blocks [2/2] [UU]
      bitmap: 4/4 pages [16KB], 65536KB chunk

unused devices: <none> 

md3 is used as the root partition and I test on that.

# lsblk -io KNAME,TYPE,SIZE,MODEL,MOUNTPOINT
KNAME     TYPE    SIZE MODEL                          MOUNTPOINT
loop0     loop     55M                                /snap/core18/1880
loop1     loop   70.6M                                /snap/lxd/16894
loop2     loop   29.9M                                /snap/snapd/8542
loop3     loop   70.6M                                /snap/lxd/16922
loop4     loop   55.3M                                /snap/core18/1885
loop5     loop   29.9M                                /snap/snapd/8790
md2       raid1   511M                                /boot
md2       raid1   511M                                /boot
md3       raid1 475.4G                                /
md3       raid1 475.4G                                /
nvme0n1   disk    477G WDC CL SN720 SDAQNTW-512G-2000
nvme0n1p1 part    511M                                /boot/efi
nvme0n1p2 part    511M
nvme0n1p3 part  475.4G
nvme0n1p4 part    511M                                [SWAP]
nvme1n1   disk    477G WDC CL SN720 SDAQNTW-512G-2000
nvme1n1p1 part    511M
nvme1n1p2 part    511M
nvme1n1p3 part  475.4G
nvme1n1p4 part    511M                                [SWAP]

and

# madam -detail /dev/md3
/dev/md3:
           Version : 0.90
     Creation Time : Thu Jul 30 13:49:54 2020
        Raid Level : raid1
        Array Size : 498530240 (475.44 GiB 510.49 GB)
     Used Dev Size : 498530240 (475.44 GiB 510.49 GB)
      Raid Devices : 2
     Total Devices : 2
   Preferred Minor : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Sep  8 13:37:54 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              UUID : 9dd3cf94:cfc5c935:a4d2adc2:26fd5302
            Events : 0.13

    Number   Major   Minor   RaidDevice State
       0     259        3        0      active sync   /dev/nvme0n1p3
       1     259        8        1      active sync   /dev/nvme1n1p3

I tried testing the speed of the drive using fio with the command

fio --name=randwrite --ioengine=libaio --iodepth=64 --rw=randwrite     --bs=64k --direct=1 --size=32G --numjobs=8 --runtime=240     --group_reporting

and the results are:

Jobs: 8 (f=8): [w(8)][100.0%][w=776MiB/s][w=12.4k IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=8): err= 0: pid=1028157: Tue Sep  8 13:09:05 2020
  write: IOPS=11.1k, BW=692MiB/s (726MB/s)(162GiB/240041msec); 0 zone resets
    slat (usec): min=127, max=567387, avg=385.31, stdev=5093.87
    clat (usec): min=2, max=1044.7k, avg=45818.20, stdev=55462.71
     lat (usec): min=268, max=1045.0k, avg=46206.45, stdev=55680.36
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   22], 10.00th=[   23], 20.00th=[   26],
     | 30.00th=[   29], 40.00th=[   33], 50.00th=[   36], 60.00th=[   41],
     | 70.00th=[   46], 80.00th=[   53], 90.00th=[   64], 95.00th=[   75],
     | 99.00th=[  443], 99.50th=[  493], 99.90th=[  550], 99.95th=[  567],
     | 99.99th=[  600]
   bw (  KiB/s): min=48768, max=1394246, per=99.97%, avg=708325.21, stdev=25832.90, samples=3840
   iops        : min=  762, max=21784, avg=11066.98, stdev=403.63, samples=3840
  lat (usec)   : 4=0.01%, 10=0.01%, 50=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.08%, 4=0.18%, 10=0.80%, 20=1.91%, 50=74.09%
  lat (msec)   : 100=20.72%, 250=0.61%, 500=1.14%, 750=0.41%, 1000=0.01%
  lat (msec)   : 2000=0.01%
  cpu          : usr=9.10%, sys=41.41%, ctx=1203665, majf=0, minf=95
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,2657370,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=692MiB/s (726MB/s), 692MiB/s-692MiB/s (726MB/s-726MB/s), io=162GiB (174GB), run=240041-240041msec

Disk stats (read/write):
    md3: ios=0/3319927, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/2692572, aggrmerge=0/634101, aggrticks=0/79485190, aggrin_queue=74460696, aggrutil=94.36%
  nvme0n1: ios=0/2692573, merge=0/634101, ticks=0/83651179, in_queue=78562212, util=94.36%
  nvme1n1: ios=0/2692572, merge=0/634102, ticks=0/75319202, in_queue=70359180, util=94.04%

I tried googling and found that people said that if I'll change Intent Bitmap from Internal to none it should speed things up, but after changing it and running fio it made things a bit slower.. maybe I needed to wait a while? I don't know.

so I'm pretty much lost.. I really don't know how to continue investigating from here, so really.. any information regarding this issue would be greatly appreciated. of course I also monitored the CPU to make sure that it's related but it looks like the cpu is not used greatly at all.

thank you!

update

someone on IRC asked me if I set writethrough by accident, I tried to google about it and found this https://www.kernel.org/doc/html/latest/driver-api/md/raid5-cache.html

it talks about raid4/5/6 and I use raid1 so maybe it's not relevant, also the file sys/block/md3/md/journal_mode as stated in this document does not exist.

update 2

found a way to test cache reads and writes

# hdparm -tT /dev/md3

/dev/md3:
 Timing cached reads:   1006 MB in  1.99 seconds = 504.40 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 664 MB in  3.01 seconds = 220.88 MB/sec

I hope this information is also useful

4
  • What's the server specs?
    – Chopper3
    Sep 8, 2020 at 14:10
  • @Chopper3 - updated the post so the specs would be clearer.
    – ufk
    Sep 8, 2020 at 15:01
  • The following IRRECOVERABLY DESTROYS DATA don't do it if you can't afford for your DATA TO BE DESTROYED FOREVER. It's always nicer to remove as many layers as possible and then build things back up so it can be interesting for you to benchmark the RAID block device directly (but in case I wasn't clear this will utterly destroy any filesystems, LVM volumes or data you had on there so clear everything down first, you do so at your own risk etc).
    – Anon
    Oct 24, 2020 at 8:08
  • Seeing same issue, feels like a bug in Linux. Read/write speeds are 100x slower than they should be. Windows works fine with expected speeds.
    – jjxtra
    Aug 21, 2022 at 19:31

0

You must log in to answer this question.

Browse other questions tagged .