9

I am using Ubuntu server 12.04 , having trouble finding cause of load, I have seen change in response time of server from past week

after reading Linux Troubleshooting, Part I: High Load

It seems like there is no issue with CPU and RAM, and this load may be related to I/O-bound load by using top command I got following output

Load and memory usage

Here it is 97.6%wa , RAM is free and no swap used .

Following is output of command iostat which sows that there is 89% iowait

ubuntu@ip-my-sys-ubuntu:~$ iostat
Linux 3.2.0-58-virtual (ip-172-31-6-203)    02/19/2015  _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.05    0.01    3.64   89.50    3.76    0.03

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvdap1           69.91         3.81       964.37     978925  247942876

I also used iotop which after fix interval shows 99 %I/O, Disk writes I observer as 1266 KB/s

enter image description here

and

enter image description here

Is is bad? as response time is lowered. what is causing this?

EDITS which are asked by others

iftop O/P

                  12.5kb             25.0kb            37.5kb             50.0kb       62.5kb
└─────────────────┴──────────────────┴─────────────────┴──────────────────┴──────────────────
ip-12-1-1-111.ap-southeast-1.  => 115.231.218.130                      0b   2.04kb   522b
                                 <=                                      0b   1.53kb   393b
ip-112-1-1-111.ap-southeast-1.  => 62.snat-111-91-22.hns.net.in      1.52kb  1.52kb  1.72kb
                                 <=                                    208b    208b    262b
ip-112-1-1-111.ap-southeast-1.  => static-mum-120.63.141.177.mtnl.      0b    480b    240b
                                 <=                                      0b    350b    175b
ip-112-1-1-111.ap-southeast-1.  => ip-112-11-1-1.ap-southeast-1.co      0b    118b    178b
                                 <=                                      0b    210b    292b
ip-112-1-1-111.ap-southeast-1.  => static-mum-120.63.194.119.mtnl.      0b      0b    240b
                                 <=                                      0b      0b    175b

TX:             cum:    123kB   peak:   3.72kb               rates:   1.67kb  2.02kb  1.78kb
RX:                    51.5kB           4.88kb                        1.19kb   989b    918b
TOTAL:                  174kB           8.60kb                        2.86kb  2.98kb  2.68kb

output of iostat -x -k 5 2

ubuntu@ip-111-11-1-111:~$ iostat -x -k 5 2
Linux 3.2.0-58-virtual (ip-111-11-1-111)        03/04/2015      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.75    0.01    4.74   22.72    4.06   64.71

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00   263.80    0.42  109.42     7.28  1572.36    28.76     1.92   17.52   17.57   17.52   2.31  25.39

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.97    0.00    4.77   76.34    9.92    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00    35.69    0.00   85.88     0.00   438.93    10.22   137.55 1612.71    0.00 1612.71  11.11  95.42

@shodanshok point 2

enter image description here

iotop -a

enter image description here

| improve this question | | | | |
  • 1
    99% IOwait with 0 disk read and write does not look good. Here serverfault.com/questions/426181/… it is mentioned, that I/O could be related not only to disk activity, baut also to the network. Could you check it with, for example, iftop (and other tools as well)? – Andrey Sapegin Feb 19 '15 at 10:47
  • @AndreySapegin added iftop – Straw Hat Feb 19 '15 at 11:51
  • I think problem was with Disc on which AWS Instance was deployed.. I created AMI of current instance and launched new Instance using that.. Now there is no extra load on I/O – Straw Hat Feb 23 '15 at 10:50
  • @StrawHat does that mean you think there was something wrong with the disc on your first instance? – sbrattla Feb 24 '15 at 19:56
  • @sbrattla No I think. after few days same problem popped out – Straw Hat Feb 27 '15 at 11:51
2
+50

Tune your mysql service for avoid touching to disk and watch out in your postfix queue, you may have a lot of emails into an I/O sensitive queue (i.e. deferred, small itens with random read behavior).

Your email system have been used as relay for spammers.

Take a look at postfix documentation and restrict relay access to your MTA.

| improve this answer | | | | |
  • moving mysql to RDS instance will work ? – Straw Hat Mar 4 '15 at 12:56
  • 1
    Sort of, the main problem is because of high number of itens into a postfix queue eating your iops, you can see with qshape deferred command. – fgbreel Mar 4 '15 at 13:00
  • postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_mailbox_limit_maps=proxy:mysql:/etc/zpanel/configs/postfix/mysql-virtual_mailbox_limit_maps.cf – Straw Hat Mar 4 '15 at 13:17
  • postconf: warning: /etc/postfix/master.cf: unused parameter: smtpd_bind_address=127.0.0.1 got these errors qshape deferred – Straw Hat Mar 4 '15 at 13:18
  • 1
    I think your postfix can be misconfigured, but for your currently problem, take a look how many emails you have at /var/lib/postfix/deferred. Move them to hold queue for further investigation or cleanup. – fgbreel Mar 4 '15 at 16:48
1

Edited after additional information gathered using iostat and iotop
Your disk is 100% loaded as it running out of available IOPS: as per iostat, you have a constant 50+ IOPS (85 w/s - 35 merged w/s). EC2 instances, especially cheap one, have a strong cap on sustained IOPS (in the range of 30-50 IOPS).

As per new iotop output, both mysql and bounce are eating significant amount of IOPS. However, iotop's output seems not complete, or badly sorted at least. Can you re-run "iotop -a" sorting one time by IOPS and another time by disk write?

Original answer
My bet: the "bounce" process is issuing many synchronized writes that choke the virtual disk device offered by Amazon (by the way, what profile are you using? EC2 disks have quite strict rules for sustained vs burst I/O).

Anyway, identify what is burning I/O bandwidth can be somewhat difficult at times. While iotop is a very good tool, sometime it don't give you the information required. We need to go deeper. So, follow these advice:

  1. First, we need to identify the type of I/O being processed and the affected block device.
    Please run the following command: iostat -x -k 5 2. Please report both results sets.
  2. Then, we need to identify the processes waiting for I/O.
    When can use "top" for that: launch it, press shift+f (F), then w, then enter, then shift+r (R). The first processes will be the one in D or D+ state (ie: waiting for disk/network). Please report back the list.
  3. Use iotop to show the accumulated I/O values for processes.
    Run iotop -a for about a minute and paste here the output.
| improve this answer | | | | |
1

A little late, but I had the same problem on a similar machine and found out that the problem was a bunch of corrupted MySQL tables. As some of these tables had a lot of data, it produced a lot of I/O waiting time.

Look at /var/log/mysql/error.log or use mysqlcheckto find and repair corrupted data.

| improve this answer | | | | |
0

As stated above, it is quite likely that your EC2 instance comes with an IO cap or maybe it is backed on an Amazon EBS Standard volume which simply doesn't deliver very much IO wise. Have a look that this page - it describes the different volume types Amazon offers.

Even if you do have the slow kind of volume, you should still be able to write reasonably fast to it, but if your load is random by nature, as it seems it may be (SQL stuff), you may want to upgrade the IOPS capacity, since that usually puts the upper bound on SQL performance.

So - from your numbers, it would seem you might run out of IOPS using standard storage. Buying faster storage is not that expensive. Have a look at this.

| improve this answer | | | | |
-3

The disk maybe be in non-DMA mode. Please check DMA status of the drive. (hdparm command)

If it's not that, something else may generate a lot of interrupts. Anyone remember those from good old DOS era ?

| improve this answer | | | | |
  • EC2 is a virtualization platform and uses virtual disks. DMA in not the culprit here. Anyway, an IRQ storm pose a toll on CPU, not disk. – shodanshok Mar 4 '15 at 20:20
  • Yes and IRQ means interrupts. – Overmind Mar 5 '15 at 6:49
  • EC2 is as far removed from that sort of problem as possible I would say. I/O is bounded by instance type - and in the end by some really pricey SAN solution that has plenty of capacity. – MrMajestyk Mar 5 '15 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.