Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a mongo server on ec2 and its storing some collection which is having 12+ millions documents. Their are 7 other instances trying to fetch some documents from the mongo server. Also their are some documents which are to be deleted. when I logged into the server its saying server load > 2.0 . The out put of iostat -xdk 1 20 command is pasted below

ubuntu@ip-10-28-170-37:~$ iostat -xdk 1 20
Linux 3.2.0-40-virtual (ip-10-28-170-37)    09/03/2013  _x86_64_    (2 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.03    0.25    0.11     3.90     3.73    43.28     0.01   33.32    4.93   99.35   1.26   0.04
xvdb              0.00     1.19    0.01    0.72     0.25     7.64    21.69     0.00    1.25    2.13    1.24   0.42   0.03
xvdf              0.00    21.94  171.95   99.20  4613.53  1132.67    42.38     0.25    0.91    0.68    1.30   1.34  36.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    43.00  694.00   31.00 22544.00   780.00    64.34     4.65    6.40    6.34    7.74   1.37  99.60

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    35.00  699.00   26.00 22352.00   684.00    63.55     4.64    6.41    6.31    9.08   1.32  96.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    39.00  666.00   29.00 19700.00   800.00    58.99     5.48    7.88    7.81    9.38   1.44 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    23.00  668.00   22.00 19716.00   656.00    59.05     5.29    7.67    7.63    8.73   1.40  96.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     3.00    1.00    1.00     4.00    16.00    20.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    39.00  675.00   28.00 21040.00   788.00    62.10     5.32    7.56    7.44   10.43   1.39  98.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     4.00    0.00    3.00     0.00    28.00    18.67     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    20.00  406.00  221.00 12536.00  2044.00    46.51    69.67   72.59   10.52  186.61   1.57  98.40

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     5.00    0.00  510.00     0.00  3996.00    15.67   144.60  290.53    0.00  290.53   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     0.00    0.00  510.00     0.00  3508.00    13.76   144.06  282.85    0.00  282.85   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     0.00    0.00  510.00     0.00  4076.00    15.98   138.05  282.61    0.00  282.61   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     6.00   44.00  466.00  1272.00  3808.00    19.92   143.29  268.35   79.00  286.22   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     6.00    0.00    4.00     0.00    40.00    20.00     0.00    1.00    0.00    1.00   1.00   0.40
xvdf              0.00     0.00   17.00  493.00   524.00  3484.00    15.72   145.42  279.91   86.82  286.56   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     0.00   24.00  486.00   544.00  2292.00    11.12   146.01  290.81   73.17  301.56   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     5.00    0.00  510.00     0.00  3264.00    12.80   144.30  284.31    0.00  284.31   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     0.00   43.00  467.00  1104.00  2980.00    16.02   149.16  290.46  114.79  306.64   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00     0.00    0.00  510.00     0.00  2948.00    11.56   144.12  281.62    0.00  281.62   1.96 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     5.00    0.00    4.00     0.00    36.00    18.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    11.00  195.00  366.00  5828.00  2860.00    30.97    84.44  189.44   13.62  283.11   1.78  99.60

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     2.00    0.00    1.00     0.00    12.00    24.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    46.00  660.00   32.00 20032.00   820.00    60.27     5.38    7.92    7.76   11.38   1.41  97.60

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    34.00  683.00   25.00 22400.00   652.00    65.12     4.54    6.41    6.33    8.64   1.38  98.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdf              0.00    37.00  684.00   30.00 21784.00   764.00    63.16     5.07    7.05    6.92   10.13   1.39  99.20

ubuntu@ip-10-28-170-37:~$ 

I am seeing %util reaching close to 100%. What does load > 2.0 implies and will moving data to a bigger instance could solve my problem. or it can be done with out increasing my instance size. (Current instance size is m1.large)

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

Your load should be lower than the amount of CPU-cores your server has. So a sixteen core machine should have load below 16.0. Load 1.00 means 1 CPU core is used 100%.

Iostat shows the disk usage. So moving over to a bigger instance will only solve the issue when there are more iops available on that instance. I think the AWS High IO instances probably will help more.

Maybe this article can help you abit; http://blog.joshsoftware.com/2012/11/22/amazon-ec2-mongodb-configuration-great-performance/

share|improve this answer
    
Should also look at: info.10gen.com/rs/10gen/images/AWS_NoSQL_MongoDB.pdf –  Adam C Sep 3 '13 at 22:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.