0

I have dedicated web server with CentOS.

Recently, I get very high server load, probably caused due to IO bottleneck. iostat -mx 3 shows:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.15    0.00   19.97   67.89    0.00    0.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              15.33     6.67 32.67  0.67     2.58     0.03   160.00     0.27    8.20   7.03  23.43
sda1             15.33     6.67 32.67  0.67     2.58     0.03   160.00     0.27    8.20   7.03  23.43
sdb             134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03
sdb1            134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03
sdc               0.00    17.33  0.33 21.00     0.00     0.15    14.50     0.08    3.84   0.31   0.67
sdc1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc2              0.00    14.33  0.33  9.33     0.00     0.09    19.86     0.04    4.28   0.48   0.47
sdc3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc5              0.00     0.67  0.00  4.33     0.00     0.02     9.23     0.02    4.85   0.46   0.20
sdc6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc7              0.00     2.33  0.00  7.33     0.00     0.04    10.55     0.02    2.68   0.18   0.13
sdc8              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

what can be done to improve performance? My web server reads lots of big files from that HD drive. does 13MB/s for HDD sata2 is a reasonable value? I would expect it to be higher.

top output (head only):

Cpu(s):  7.5%us,  5.5%sy,  0.0%ni,  4.1%id, 71.5%wa,  3.0%hi,  8.5%si,  0.0%st
Mem:   2048144k total,  2036732k used,    11412k free,    23732k buffers
Swap:  8385888k total,      192k used,  8385696k free,  1375064k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15369 apache    15   0  116m 9.8m 3564 S  3.3  0.5   0:01.37 httpd
23491 apache    15   0  117m  10m 3644 S  3.3  0.5   0:00.55 httpd
21694 mysql     25   0  308m  49m 4752 S  3.0  2.5   1068:49 mysqld
25735 apache    15   0  117m  10m 3660 S  2.7  0.5   0:00.53 httpd
25909 apache    15   0  116m 9964 3548 S  1.0  0.5   0:00.15 httpd
14814 apache    15   0  117m  10m 3684 S  0.7  0.5   0:00.53 httpd
15708 apache    15   0  117m 9.9m 3564 S  0.7  0.5   0:00.66 httpd
21464 apache    15   0  117m  10m 3664 S  0.7  0.5   0:01.32 httpd
22997 apache    15   0  116m 9.8m 3656 R  0.7  0.5   0:04.16 httpd
23484 apache    15   0  117m  10m 3668 S  0.7  0.5   0:00.28 httpd

thanks

6
  • Are you swapping? You might want to include a snapshot of top.
    – mdpc
    Nov 27, 2012 at 18:27
  • top output: top - 21:02:52 up 60 days, 15:43, 2 users, load average: 78.72, 72.56, 70.79 Tasks: 313 total, 3 running, 310 sleeping, 0 stopped, 0 zombie Cpu(s): 9.8%us, 6.8%sy, 0.0%ni, 0.0%id, 70.8%wa, 2.7%hi, 10.0%si, 0.0%st Mem: 2048144k total, 2034540k used, 13604k free, 14428k buffers Swap: 8385888k total, 192k used, 8385696k free, 1430824k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21694 mysql 25 0 308m 49m 4752 S 5.0 2.5 1067:53 mysqld 6897 apache 15 0 117m 9.8m 3528 S 4.6 0.5 0:00.38 httpd
    – Niros
    Nov 27, 2012 at 19:06
  • @Niros, please update the question instead so we can benefit from the correct formatting. Cheers Nov 27, 2012 at 19:10
  • If your Centos install has iotop available I'd have a look at its output, too. Also, is there anything in dmesg' output regarding the HDD, and its performance? Bus resets?
    – tink
    Nov 27, 2012 at 21:17
  • Exactly what hard drives are you using, and with what controller? Nov 27, 2012 at 23:08

4 Answers 4

1

What iostat is basically showing here is your disk saturated with IOPS.

man iostat:

• rrqm/s — The number of read requests merged per second that were queued to the device.

• r/s — The number (after merges) of read requests completed per second for the device.

HDDs IOPS performance reference data, Wikipedia

Compare the numbers.

1

I know its a pretty old thread but I am facing similar issue

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util

sdb             134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03

So in my case whenever avgqu-sz reaches equal to lun queue depth(/sys/block/sdb/device/queue_depth),disk saturation take place and in my case it's set to 128.I just increase it to 192 and now getting better performance.I know it's not a better solution but I can't afford to change disk currently so this is the only solution I have.

2
  • That's a perfectly correct solution to enlarge the queue, if your disk is capable of managing more events in queue.
    – wazoox
    Oct 31, 2018 at 17:06
  • What if the device is a SSD? In my case there is only /sys/block/nvme0n1/device/queue_count which is 9. And I fialed to change its value with permission denied error althrough with sudo.
    – ToughMind
    Dec 1, 2022 at 2:53
0

I would strongly urge you to consider replacing sdb with an SSD or SSD-backed RAID array.

0

13.34 MB/s is quite slow for server or desktop class hard disk but the throughput may be hampered by lots of random access. If this disk is presented as RAID 1 or RAID 5 from the hardware then this may explain some of your loss in throughput

You can test your real throughput using bonnie++ while the webserver is offline. A plain SATA II desktop drive should achieve somewhere between 40-70MB/s.

You may achieve good random access by either using RAID 0 (BAD redundancy) or ideally RAID 10 with > 4 disks

Update: You should also add as much RAM as you can afford. The memory figures show that the server is already caching a massive amount of data in RAM.

You may find the cost of Amazon S3 and/or EC2 is actually more cost effective when you consider how much more equipment you may have to buy.

6
  • I have the follwoing HDD: wdc.com/global/products/specs/?driveID=899&language=1 I don't have raid controller. What should I do in order to achive 40 MB/s? with hdparm -t I get: Timing buffered disk reads: 272 MB in 3.00 seconds = 90.63 MB/sec
    – Niros
    Nov 27, 2012 at 19:31
  • bonnie++ is a much better benchmark of IO. Nonetheless, I think you need to spread your load across spindles to help your random access times. This can only be achieved with RAID Nov 27, 2012 at 19:50
  • What about defrag? maybe the files are fragmented? In addition, is there anything that I can do to improve performance with my current hardware? changing block size might help?
    – Niros
    Nov 27, 2012 at 20:46
  • Fragmentation is very unlikely and is not something that concerns most Linux engineers. Judging by the amount of httpd servers you've got running, you can't avoid all those clients trying to access different parts of the filesystem. If you don't want to spend the money then perhaps you should reduce the number of clients. Nov 27, 2012 at 21:01
  • what about converting disk from ext3 to ext4? what about mounting the disk with noatime? what about changing blcok size? thx
    – Niros
    Nov 28, 2012 at 19:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .