Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have dedicated web server with CentOS.

Recently, I get very high server load, probably caused due to IO bottleneck. iostat -mx 3 shows:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.15    0.00   19.97   67.89    0.00    0.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda              15.33     6.67 32.67  0.67     2.58     0.03   160.00     0.27    8.20   7.03  23.43
sda1             15.33     6.67 32.67  0.67     2.58     0.03   160.00     0.27    8.20   7.03  23.43
sdb             134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03
sdb1            134.67     0.00 145.33 10.67    13.34     0.19   177.59   131.25  963.31   6.41 100.03
sdc               0.00    17.33  0.33 21.00     0.00     0.15    14.50     0.08    3.84   0.31   0.67
sdc1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc2              0.00    14.33  0.33  9.33     0.00     0.09    19.86     0.04    4.28   0.48   0.47
sdc3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc5              0.00     0.67  0.00  4.33     0.00     0.02     9.23     0.02    4.85   0.46   0.20
sdc6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc7              0.00     2.33  0.00  7.33     0.00     0.04    10.55     0.02    2.68   0.18   0.13
sdc8              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

what can be done to improve performance? My web server reads lots of big files from that HD drive. does 13MB/s for HDD sata2 is a reasonable value? I would expect it to be higher.

top output (head only):

Cpu(s):  7.5%us,  5.5%sy,  0.0%ni,  4.1%id, 71.5%wa,  3.0%hi,  8.5%si,  0.0%st
Mem:   2048144k total,  2036732k used,    11412k free,    23732k buffers
Swap:  8385888k total,      192k used,  8385696k free,  1375064k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15369 apache    15   0  116m 9.8m 3564 S  3.3  0.5   0:01.37 httpd
23491 apache    15   0  117m  10m 3644 S  3.3  0.5   0:00.55 httpd
21694 mysql     25   0  308m  49m 4752 S  3.0  2.5   1068:49 mysqld
25735 apache    15   0  117m  10m 3660 S  2.7  0.5   0:00.53 httpd
25909 apache    15   0  116m 9964 3548 S  1.0  0.5   0:00.15 httpd
14814 apache    15   0  117m  10m 3684 S  0.7  0.5   0:00.53 httpd
15708 apache    15   0  117m 9.9m 3564 S  0.7  0.5   0:00.66 httpd
21464 apache    15   0  117m  10m 3664 S  0.7  0.5   0:01.32 httpd
22997 apache    15   0  116m 9.8m 3656 R  0.7  0.5   0:04.16 httpd
23484 apache    15   0  117m  10m 3668 S  0.7  0.5   0:00.28 httpd

thanks

share|improve this question
    
Are you swapping? You might want to include a snapshot of top. –  mdpc Nov 27 '12 at 18:27
    
top output: top - 21:02:52 up 60 days, 15:43, 2 users, load average: 78.72, 72.56, 70.79 Tasks: 313 total, 3 running, 310 sleeping, 0 stopped, 0 zombie Cpu(s): 9.8%us, 6.8%sy, 0.0%ni, 0.0%id, 70.8%wa, 2.7%hi, 10.0%si, 0.0%st Mem: 2048144k total, 2034540k used, 13604k free, 14428k buffers Swap: 8385888k total, 192k used, 8385696k free, 1430824k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21694 mysql 25 0 308m 49m 4752 S 5.0 2.5 1067:53 mysqld 6897 apache 15 0 117m 9.8m 3528 S 4.6 0.5 0:00.38 httpd –  Niros Nov 27 '12 at 19:06
    
@Niros, please update the question instead so we can benefit from the correct formatting. Cheers –  Alastair McCormack Nov 27 '12 at 19:10
    
If your Centos install has iotop available I'd have a look at its output, too. Also, is there anything in dmesg' output regarding the HDD, and its performance? Bus resets? –  tink Nov 27 '12 at 21:17
    
Exactly what hard drives are you using, and with what controller? –  Michael Hampton Nov 27 '12 at 23:08

3 Answers 3

I know its a pretty old thread but I am facing similar issue

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util

sdb 134.67 0.00 145.33 10.67 13.34 0.19 177.59 131.25 963.31 6.41 100.03

So in my case whenever avgqu-sz reaches equal to lun queue depth(/sys/block/sdb/device/queue_depth),disk saturation take place and in my case it's set to 128.I just increase it to 192 and now getting better performance.I know it's not a better solution but I can't afford to change disk currently so this is the only solution I have.

share|improve this answer

13.34 MB/s is quite slow for server or desktop class hard disk but the throughput may be hampered by lots of random access. If this disk is presented as RAID 1 or RAID 5 from the hardware then this may explain some of your loss in throughput

You can test your real throughput using bonnie++ while the webserver is offline. A plain SATA II desktop drive should achieve somewhere between 40-70MB/s.

You may achieve good random access by either using RAID 0 (BAD redundancy) or ideally RAID 10 with > 4 disks

Update: You should also add as much RAM as you can afford. The memory figures show that the server is already caching a massive amount of data in RAM.

You may find the cost of Amazon S3 and/or EC2 is actually more cost effective when you consider how much more equipment you may have to buy.

share|improve this answer
    
I have the follwoing HDD: wdc.com/global/products/specs/?driveID=899&language=1 I don't have raid controller. What should I do in order to achive 40 MB/s? with hdparm -t I get: Timing buffered disk reads: 272 MB in 3.00 seconds = 90.63 MB/sec –  Niros Nov 27 '12 at 19:31
    
bonnie++ is a much better benchmark of IO. Nonetheless, I think you need to spread your load across spindles to help your random access times. This can only be achieved with RAID –  Alastair McCormack Nov 27 '12 at 19:50
    
What about defrag? maybe the files are fragmented? In addition, is there anything that I can do to improve performance with my current hardware? changing block size might help? –  Niros Nov 27 '12 at 20:46
    
Fragmentation is very unlikely and is not something that concerns most Linux engineers. Judging by the amount of httpd servers you've got running, you can't avoid all those clients trying to access different parts of the filesystem. If you don't want to spend the money then perhaps you should reduce the number of clients. –  Alastair McCormack Nov 27 '12 at 21:01
    
what about converting disk from ext3 to ext4? what about mounting the disk with noatime? what about changing blcok size? thx –  Niros Nov 28 '12 at 19:54

I would strongly urge you to consider replacing sdb with an SSD or SSD-backed RAID array.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.