0

This is a mail server which is running Apache2 and Postfix as services. It's been up for 135 days and nothing has been changed to the services since then. The usual load average was about 0.xx but it spiked up to around 1.00 consistently a week ago. I've checked through all the solutions regarding this issue online that I can find and investigate thoroughly on the usual culprits such as IO queue, CPU utilization, and etc. All looks normal to me.

This server is running on top of some ESXI VMWare server as VM. Some other engineer might have performed a migration at the hypervisor level. The migrations objective was to update the host servers firmware version. After that this VM was migrated back to the upgraded host. The migration is seamless , meaning the operation of this VM is not interrupted in the process.

uname -a

Linux mail.net 4.9.0-040900rc2-generic #201610232131 SMP Mon Oct 24 01:32:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Stepping:              2
CPU MHz:               2399.569
BogoMIPS:              4799.99
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0

top


top - 17:12:23 up 135 days,  6:37,  1 user,  load average: 1.00, 1.00, 1.00
Tasks: 154 total,   1 running, 153 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  12303728 total, 10122732 used,  2180996 free,   215672 buffers
KiB Swap:  1048572 total,        0 used,  1048572 free.  1321424 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    1 root      20   0   35604   4024   2660 S  0.0  0.0   0:13.76 init
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.92 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   2:18.67 ksoftirqd/0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S  0.0  0.0   4:44.58 rcu_sched
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 lru-add-drain
   11 root      rt   0       0      0      0 S  0.0  0.0   0:18.56 watchdog/0
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0
   13 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   14 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
   15 root      20   0       0      0      0 S  0.0  0.0   0:05.61 khungtaskd
   16 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
   17 root       0 -20       0      0      0 S  0.0  0.0   0:00.24 writeback
   18 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
   19 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
   20 root      39  19       0      0      0 S  0.0  0.0   0:43.27 khugepaged
   21 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
   22 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd

netstat -natp

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:873             0.0.0.0:*               LISTEN      1546/rsync
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      15493/mysqld
tcp        0      0 127.0.0.1:10031         0.0.0.0:*               LISTEN      1524/perl
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1058/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      1409/master
tcp        0      0 0.0.0.0:10080           0.0.0.0:*               LISTEN      1066/xinetd
tcp        0      0 203.92.11.241:25        43.245.16.27:1304       TIME_WAIT   -
tcp        0      0 203.92.11.241:25        43.245.16.27:1307       ESTABLISHED 29082/smtpd
tcp        0      0 127.0.0.1:10031         127.0.0.1:52966         TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52956         TIME_WAIT   -
tcp        0      0 127.0.0.1:52984         127.0.0.1:10031         CLOSE_WAIT  29082/smtpd
tcp        0    384 203.92.11.241:22        203.92.90.66:56408      ESTABLISHED 28753/sshd: xxx [
tcp        0      0 203.92.11.241:25        43.245.16.27:1305       TIME_WAIT   -
tcp        0      0 203.92.11.241:25        203.92.81.184:8509      TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52962         TIME_WAIT   -
tcp        1      0 127.0.0.1:52982         127.0.0.1:10031         CLOSE_WAIT  29107/smtpd
tcp        0      0 203.92.11.241:25        43.245.17.28:52703      TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52958         TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52978         TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52988         TIME_WAIT   -
tcp        0      0 203.92.11.241:47072     203.92.86.79:389        ESTABLISHED 1079/sssd_be
tcp        0      0 127.0.0.1:10031         127.0.0.1:52982         FIN_WAIT2   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52952         TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52984         FIN_WAIT2   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52974         TIME_WAIT   -
tcp        0      0 203.92.11.241:25        43.245.17.28:61916      TIME_WAIT   -
tcp        0      0 127.0.0.1:10031         127.0.0.1:52968         TIME_WAIT   -
tcp6       0      0 :::873                  :::*                    LISTEN      1546/rsync
tcp6       0      0 :::80                   :::*                    LISTEN      1788/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      1058/sshd
tcp6       0      0 :::25                   :::*                    LISTEN      1409/master

iostat

Linux 4.9.0-040900rc2-generic (mail1.lga.net.sg)    03/10/19    _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.14    0.02    0.12    0.19    0.00   99.53

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               1.26         0.07        24.53     816686  286303390
dm-0              2.04         0.07        24.53     796673  286303376
dm-1              0.00         0.00         0.00        896          0

vmstat 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 2180892 215672 1321628    0    0     0    25    0    3  0  0 100  0  0
 0  0      0 2180768 215672 1321428    0    0     0     0   64  140  0  0 100  0  0
 0  0      0 2180768 215672 1321432    0    0     0   115   71  157  0  0 95  5  0
 0  0      0 2180768 215672 1321432    0    0     0    12   59  125  0  0 100  0  0
 0  0      0 2180768 215672 1321432    0    0     0     2   62  132  0  0 100  0  0
 0  0      0 2180768 215672 1321432    0    0     0     6   62  133  0  0 100  0  0
 0  0      0 2185960 215672 1321432    0    0     0     0   60  131  0  0 100  0  0
 0  0      0 2179896 215672 1321432    0    0     0    46  123  267  1  0 99  0  0
 0  0      0 2180392 215672 1321432    0    0     0    10   66  142  0  0 100  0  0
 0  0      0 2180392 215672 1321432    0    0     0   132   71  151  0  0 93  7  0
 0  0      0 2180392 215672 1321432    0    0     0     4   64  134  0  0 99  0  0
 0  0      0 2180392 215672 1321432    0    0     0    13   65  133  0  0 100  0  0
 0  0      0 2181548 215672 1321432    0    0     0     0   64  137  0  0 100  0  0
  • Depending on the firmware update that was done, it might affect the CPU performance, if said firmware was to mitigate some of the CPU exploits. – GregL Oct 4 '19 at 0:45
  • The only solution I can think of is just reboot the VM, using VMware after taking snapshot. – Weng Hui Oct 4 '19 at 1:42
  • That might help, but not if it’s what I’m thinking it is. – GregL Oct 4 '19 at 1:44
  • 1
    Issue resolved by a simple rebooting. Seems like it really was due to the hypervisor firmware update. – Weng Hui Oct 4 '19 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.