27

this is a standard apache web server on AWS Linux AMI + EBS. We are noticing high load average (+8) and iotop -a shows:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 2.37 M/s

  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND             
 3730 be/4 root          0.00 B      0.00 B  0.00 % 91.98 % [kworker/u8:1]
  774 be/3 root          0.00 B   1636.00 K  0.00 % 15.77 % [jbd2/xvda1-8]
 3215 be/4 apache        0.00 B     40.39 M  0.00 %  0.88 % httpd
 3270 be/4 apache        0.00 B     38.20 M  0.00 %  0.93 % httpd
 2770 be/4 apache        0.00 B     46.86 M  0.00 %  0.71 % httpd

When apache is down, kworker and jbd2 is also down.

Server is not swapping as we have plenty of RAM available. I've seen this issue related to Database servers, but nothing only isolated to Apache.

Any idea on how to diagnose this further and prevent it?

UPDATE 1: perf report (perf record -g -a sleep 10)

Samples: 114K of event 'cpu-clock', Event count (approx.): 28728500000
-  83.58%          swapper  [kernel.kallsyms]         [k] xen_hypercall_sched_op                                          ◆
   + xen_hypercall_sched_op                                                                                               ▒
   + default_idle                                                                                                         ▒
   + arch_cpu_idle                                                                                                        ▒
   - cpu_startup_entry                                                                                                    ▒
        70.16% cpu_bringup_and_idle                                                                                       ▒
      - 29.84% rest_init                                                                                                  ▒
           start_kernel                                                                                                   ▒
           x86_64_start_reservations                                                                                      ▒
           xen_start_kernel                                                                                               ▒
+   1.73%            httpd  [kernel.kallsyms]         [k] __d_lookup_rcu                                                  ▒
+   1.08%            httpd  [kernel.kallsyms]         [k] xen_hypercall_xen_version                                       ▒
+   0.38%            httpd  [vdso]                    [.] 0x0000000000000d7c                                              ▒
+   0.36%            httpd  libphp5.so                [.] zend_hash_find                                                  ▒
+   0.33%            httpd  libphp5.so                [.] _zend_hash_add_or_update                                        ▒
+   0.25%            httpd  libc-2.17.so              [.] __memcpy_ssse3                                                  ▒
+   0.24%            httpd  libphp5.so                [.] _zval_ptr_dtor                                                  ▒
+   0.24%            httpd  [kernel.kallsyms]         [k] __audit_syscall_entry                                           ▒
+   0.22%            httpd  [kernel.kallsyms]         [k] pvclock_clocksource_read                                        ▒
Improve this question
9
  • 3
    You may want to use perf to find out what kworker is doing as a troubleshooting step.
    – David Schwartz
    Jan 13, 2015 at 18:40
  • kworker's behaviour is technically interesting, but I wonder why Apache threads are writing megabytes to the disk. Assuming that explains the 2MB/s, isn't that high for a web server? Then one could identify the files being written, e.g. strace -p (and maybe lsof) and see if that shows anything interesting.
    – sourcejedi
    Jan 13, 2015 at 19:18
  • 1
    Is it swapping by any chance?
    – Grizly
    Jan 14, 2015 at 22:22
  • 1
    Try to enable sendfile on apache to take advantage of zero copy.
    – fgbreel
    Jun 1, 2015 at 16:19
  • 1
    @user2383712 This issue maybe related your cloud "neighbor" can you contact aws about this issue, if not try to shutdown you aws instance to change it's hypervisor, i had this problem in the past.
    – Alin Andrei
    Apr 11, 2016 at 6:26

2 Answers 2

Reset to default
7

100% IO doesn't mean it's using all your IO operations. It means it's doing nothing but waiting on IO. Therefore, high %IO with low/zero disk bandwidth can be normal.

man iotop:

[...] It also displays the percentage of time the thread/process spent while swapping in and while waiting on I/O.

It may be a different issue if your kworker is waiting on IO forever, but I don't know. Maybe it's supposed to be waiting on a pipe or something. I see kworker doing the same on my server sometimes, and it doesn't seem to be a problem. (I also panicked the first time I saw it.)

Improve this answer
2
  • 2
    This is also in a shared environment, where they all access the same storage arrays. This is a sign of a busy disk (of which the VM may not know anything about because it's effectively isolated). On dedicated hardware, it would be more likely to be a failing disk with lots of retry. On network mounted access, it can mean a bad link as well as NAS / target side congestion.
    – Spooler
    Sep 25, 2016 at 10:25
  • This also happens when writing to slower storage in general, I notice kworker freaking out in iotop when writing an ISO or IMG file to an SD Card for example. Sometimes it's the SD card reader, other times it's the card itself. I once thought that it couldn't be the reader, because the system should know the maximum capacity of that reader, but it doesn't seem to matter...
    – Tmanok
    Mar 3, 2021 at 19:53
0

Had the problem of the drive being written to every 5 secs I used the command below and found that it was x2goserver keeps running every 5 sec and trigger kworker. Note that google chrome will write to the drive if its open.

sudo apt remove x2goserver

sudo pidstat -dvl 5

04:52:35 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command

04:52:40 PM     0      2318    539.42      0.01      0.00       0  /usr/bin/perl /usr/sbin/x2gocleansessions
04:52:40 PM     0   1920632      0.00    251.20      0.00       0  kworker/u64:3-events_unbound



04:52:40 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
04:52:45 PM     0      2318    809.12      0.01      0.00       0  /usr/bin/perl /usr/sbin/x2gocleansessions



04:52:45 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
04:52:50 PM     0      2318    539.42      0.01      0.00       0  /usr/bin/perl /usr/sbin/x2gocleansessions
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.