4

I have a 30Tb-sized hardware RAID-6 system (LSI 9280-8e) of 10 DC-S4500 Intel SSDs that is used for database purposes. The OS Debian 7.11 with 3.2 kernel. The filesystem is XFS mounted with nobarrier option.

Seeing somewhat sluggish comparing to my expectations performance in random I/O, I started to investigate what's going on by running fio benchmarks. And to my surprise when I just used fio on 1Tb file in random-read settings with (iodepth=32 and ioengine=libaio) I get ~ 3000 IOPS which is much lower than what I was expecting.

random-read: (groupid=0, jobs=1): err= 0: pid=128531
  read : io=233364KB, bw=19149KB/s, iops=4787 , runt= 12187msec
  ...
  cpu          : usr=1.94%, sys=5.81%, ctx=58484, majf=0, minf=53
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=58341/w=0/d=0, short=r=0/w=0/d=0

However if I use direct=1 option (i.e. bypassing linux's buffer cache), I get ~ 40000 IOPS, which what I'd like to see.

random-read: (groupid=0, jobs=1): err= 0: pid=130252
  read : io=2063.7MB, bw=182028KB/s, iops=45507 , runt= 11609msec
....
  cpu          : usr=6.93%, sys=23.29%, ctx=56503, majf=0, minf=54
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=528291/w=0/d=0, short=r=0/w=0/d=0

I seem to have all the right settings for the SSD partition in the form of the scheduler, read-ahead and rotational setting.

root@XX:~# cat /sys/block/sdd/queue/scheduler
[noop] deadline cfq 
root@XX:~# cat /sys/block/sdd/queue/rotational
0
root@XX:~# blockdev --getra /dev/sdd
0

Am I still missing something that lowers the buffered performance so much ? Or is it expected to see such a difference between DIRECT vs buffered ?

I also looked at iostat output during two runs This is when direct=1 was used:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdd               0.00     0.00 48110.00    0.00 192544.00     0.00     8.00    27.83    0.58    0.58    0.00   0.02  99.60

This is a buffered run

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdd               0.00     0.00 4863.00    0.00 19780.00     0.00     8.13     0.89    0.18    0.18    0.00   0.18  85.60

So it looks like the key difference is the queue size (avgqu-sz), which is small when using buffered I/O. I find it weird given that nr_requests and queue_depth are all high:

root@XX:~# cat /sys/block/sdd/queue/nr_requests
128
root@XX:~# cat /sys/block/sda/device/queue_depth
256

Any advice here ?

  • Hardware RAID? Software RAID? RAID configuration details? File system details? – Andrew Henle Jun 29 '18 at 19:26
  • 1
    I've added a bit more info about RAID and filesystem to the post. It is LSI hardware raid with XFS filesystem. However I'd argue the problem is not with the hardware, given that the hardware clearly delivers okay performance (with direct IO). - – sega_sai Jun 29 '18 at 19:43
  • maybe "direct" IO uses mmap? mmap is usually faster than other ways of IO – porton Jun 29 '18 at 20:04
  • @porton maybe "direct" IO uses mmap? mmap is usually faster than other ways of IO mmap in general is no faster nor any slower than any other IO type. mmap can also be quite a bit slower in some cases because it requires the creation of page mappings that aren't necessary for other types of IO. And direct IO certainly does not use mmap - direct IO means the page cache is bypassed. Direct IO is good for streaming files where the same bytes are never reread. mmap is actually likely the slowest IO method for that particular use case. – Andrew Henle Jun 29 '18 at 20:13
  • @sega_sai It sure looks like the page cache is really getting in the way, which is strange. How much free RAM do you have? Maybe try dropping all caches before trying your buffered IO tests? – Andrew Henle Jun 29 '18 at 20:18
8

Debian 7.11 with 3.2 kernel

Upgrade if at all possible. Not only do you get kernel improvements, but Wheezy is end of life.


Yes, you see higher utilization and queue depth when direct=1. The fio manual calls out this case in particular (emphasis mine):

iodepth=int

Number of I/O units to keep in flight against the file. Note that increasing iodepth beyond 1 will not affect synchronous ioengines (except for small degrees when verify_async is in use). Even async engines may impose OS restrictions causing the desired depth not to be achieved. This may happen on Linux when using libaio and not setting direct=1, since buffered I/O is not async on that OS. Keep an eye on the I/O depth distribution in the fio output to verify that the achieved depth is as expected

So libaio requires O_DIRECT for asynchronous, an important implementation detail to know. Someone asked if not direct with libaio was a good idea:

Is it valid to set direct=0 when using libaio?

You can do it but I wouldn't recommend it. With today's Linux kernels, libaio submission is likely to become blocking (and thus no longer asynchronous) without O_DIRECT which can limit the amount of parallel I/O achieved. There's a strong argument that fio examples should NOT encourage such a combination of options...

what does "queued" behavior mean in the man doc?

If you mean the sentence "Note that Linux may only support queued behavior with non-buffered I/O" (over in http://fio.readthedocs.io/en/latest/fio_doc.html#i-o-engine ) I think it's trying to say:

"Rather than blocking on the submission syscall until the I/O has gone down AND come back from the lowest disk device (blocking behaviour), when using direct=1 with libaio you can submit an I/O and have it asynchronously queued by the kernel allowing the submission syscall to return straight away and opening the opportunity for you to queue other submissions before the I/O completes".

Also try a control test with ioengine=psync and direct=0. Even synchronous writes with a cache can do a lot of IOPS.

All of this sidesteps the real question: what was the problem on the database workload you were running? Problem symptoms, software versions, configuration, performance metrics (iostat). The DBMS's implementation of I/O may be wildly different from what you simulated, the system calls used, multiple files and jobs doing I/O, any number of things. This is worth its own question if you want to investigate further.

| improve this answer | |
  • Thanks, The problem was in indeed in the libaio engine. When I disabled it and used the default engine and numjobs=32 I've got ~ 40000 IOPS without using direct=1 – sega_sai Jul 1 '18 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.