We upgraded to php7 (this was only change in server), this is causing high load on CPU.
Cacti graph
sar output
04:40:01 AM CPU %user %nice %system %iowait %steal %idle
04:50:01 AM all 2.85 0.00 2.13 0.01 0.00 95.01
04:50:01 AM 0 4.53 0.00 8.41 0.03 0.00 87.03
04:50:01 AM 1 24.64 0.00 3.93 0.01 0.00 71.43
04:50:01 AM 2 9.01 0.00 4.28 0.00 0.00 86.71
04:50:01 AM 3 3.89 0.00 2.23 0.00 0.00 93.87
04:50:01 AM 4 0.49 0.00 0.20 0.00 0.00 99.31
04:50:01 AM 5 0.38 0.00 0.79 0.00 0.00 98.83
04:50:01 AM 6 5.15 0.00 13.87 0.04 0.00 80.94
04:50:01 AM 7 1.18 0.00 5.05 0.00 0.00 93.76
04:50:01 AM 8 0.61 0.00 1.94 0.00 0.00 97.45
04:50:01 AM 9 0.37 0.00 1.33 0.00 0.00 98.30
04:50:01 AM 10 0.05 0.00 0.30 0.00 0.00 99.64
04:50:01 AM 11 0.11 0.00 0.03 0.00 0.00 99.85
04:50:01 AM 12 13.35 0.00 2.62 0.02 0.00 84.00
04:50:01 AM 13 0.98 0.00 0.24 0.00 0.00 98.78
04:50:01 AM 14 0.59 0.00 0.36 0.00 0.00 99.05
04:50:01 AM 15 0.24 0.00 0.13 0.00 0.00 99.63
04:50:01 AM 16 0.24 0.00 0.14 0.00 0.00 99.62
04:50:01 AM 17 0.03 0.00 0.03 0.00 0.00 99.94
04:50:01 AM 18 0.99 0.00 3.73 0.02 0.00 95.27
04:50:01 AM 19 0.37 0.00 0.54 0.00 0.00 99.09
04:50:01 AM 20 0.62 0.00 0.56 0.00 0.00 98.82
04:50:01 AM 21 0.24 0.00 0.28 0.00 0.00 99.48
04:50:01 AM 22 0.29 0.00 0.21 0.00 0.00 99.49
04:50:01 AM 23 0.07 0.00 0.07 0.00 0.00 99.87
04:50:01 AM CPU %user %nice %system %iowait %steal %idle
05:00:01 AM all 5.66 0.00 2.19 0.01 0.00 92.13
05:00:01 AM 0 7.73 0.00 7.75 0.05 0.00 84.47
05:00:01 AM 1 49.79 0.00 8.65 0.00 0.00 41.55
05:00:01 AM 2 25.74 0.00 5.77 0.00 0.00 68.50
05:00:01 AM 3 4.95 0.00 1.42 0.00 0.00 93.63
05:00:01 AM 4 1.89 0.00 1.69 0.00 0.00 96.42
05:00:01 AM 5 0.38 0.00 0.97 0.00 0.00 98.65
05:00:01 AM 6 9.79 0.00 3.96 0.10 0.00 86.15
05:00:01 AM 7 2.09 0.00 5.39 0.02 0.00 92.50
05:00:01 AM 8 0.31 0.00 0.24 0.01 0.00 99.44
05:00:01 AM 9 0.76 0.00 1.08 0.00 0.00 98.16
05:00:01 AM 10 0.02 0.00 0.13 0.01 0.00 99.84
05:00:01 AM 11 0.33 0.00 1.33 0.00 0.00 98.34
05:00:01 AM 12 25.21 0.00 3.81 0.02 0.00 70.96
05:00:01 AM 13 0.90 0.00 0.66 0.00 0.00 98.44
05:00:01 AM 14 0.67 0.00 0.76 0.05 0.00 98.52
05:00:01 AM 15 0.31 0.00 0.27 0.00 0.00 99.42
05:00:01 AM 16 0.28 0.00 0.26 0.00 0.00 99.46
05:00:01 AM 17 0.12 0.00 0.18 0.00 0.00 99.69
05:00:01 AM 18 1.24 0.00 5.03 0.04 0.00 93.69
05:00:01 AM 19 0.89 0.00 0.87 0.03 0.00 98.20
05:00:01 AM 20 1.74 0.00 1.23 0.00 0.00 97.02
05:00:01 AM 21 0.44 0.00 0.42 0.00 0.00 99.14
05:00:01 AM 22 0.38 0.00 0.45 0.00 0.00 99.17
05:00:01 AM 23 0.09 0.00 0.47 0.00 0.00 99.44
05:00:01 AM CPU %user %nice %system %iowait %steal %idle
05:10:01 AM all 6.16 0.00 1.64 0.02 0.00 92.18
05:10:01 AM 0 6.03 0.00 5.99 0.09 0.00 87.90
05:10:01 AM 1 48.29 0.00 7.89 0.00 0.00 43.81
05:10:01 AM 2 30.24 0.00 5.09 0.00 0.00 64.67
05:10:01 AM 3 9.02 0.00 1.20 0.00 0.00 89.77
05:10:01 AM 4 2.13 0.00 0.36 0.00 0.00 97.51
05:10:01 AM 5 0.54 0.00 0.16 0.00 0.00 99.30
05:10:01 AM 6 10.47 0.00 5.23 0.15 0.00 84.15
05:10:01 AM 7 1.90 0.00 1.13 0.02 0.00 96.95
05:10:01 AM 8 0.66 0.00 1.85 0.01 0.00 97.47
05:10:01 AM 9 0.26 0.00 0.13 0.00 0.00 99.61
05:10:01 AM 10 0.02 0.00 0.08 0.00 0.00 99.90
05:10:01 AM 11 0.01 0.00 0.08 0.00 0.00 99.92
05:10:01 AM 12 28.45 0.00 3.41 0.06 0.00 68.09
05:10:01 AM 13 1.60 0.00 0.73 0.00 0.00 97.67
05:10:01 AM 14 0.55 0.00 0.67 0.00 0.00 98.78
05:10:01 AM 15 0.66 0.00 0.24 0.00 0.00 99.09
05:10:01 AM 16 0.44 0.00 0.18 0.00 0.00 99.39
05:10:01 AM 17 0.19 0.00 0.12 0.00 0.00 99.69
05:10:01 AM 18 0.39 0.00 0.53 0.03 0.00 99.05
05:10:01 AM 19 2.51 0.00 1.42 0.00 0.00 96.07
05:10:01 AM 20 1.11 0.00 1.15 0.00 0.00 97.74
05:10:01 AM 21 1.99 0.00 1.25 0.00 0.00 96.76
05:10:01 AM 22 0.35 0.00 0.35 0.00 0.00 99.29
05:10:01 AM 23 0.22 0.00 0.34 0.01 0.00 99.44
05:10:01 AM CPU %user %nice %system %iowait %steal %idle
05:20:01 AM all 4.02 0.00 12.27 0.01 0.00 83.69
05:20:01 AM 0 7.35 0.00 18.25 0.02 0.00 74.38
05:20:01 AM 1 11.90 0.00 31.41 0.05 0.00 56.64
05:20:01 AM 2 6.57 0.00 25.74 0.00 0.00 67.68
05:20:01 AM 3 3.52 0.00 19.25 0.00 0.00 77.23
05:20:01 AM 4 2.58 0.00 16.29 0.00 0.00 81.14
05:20:01 AM 5 2.12 0.00 12.28 0.02 0.00 85.58
05:20:01 AM 6 12.51 0.00 25.54 0.06 0.00 61.88
05:20:01 AM 7 4.24 0.00 16.93 0.00 0.00 78.82
05:20:01 AM 8 3.75 0.00 8.85 0.01 0.00 87.40
05:20:01 AM 9 2.45 0.00 5.40 0.01 0.00 92.15
05:20:01 AM 10 2.79 0.00 4.57 0.00 0.00 92.64
05:20:01 AM 11 1.00 0.00 5.14 0.00 0.00 93.86
05:20:01 AM 12 5.14 0.00 19.32 0.02 0.00 75.52
05:20:01 AM 13 2.28 0.00 16.37 0.00 0.00 81.35
05:20:01 AM 14 1.26 0.00 11.01 0.00 0.00 87.73
05:20:01 AM 15 0.80 0.00 8.60 0.00 0.00 90.60
05:20:01 AM 16 0.93 0.00 5.01 0.00 0.00 94.06
05:20:01 AM 17 1.41 0.00 4.92 0.00 0.00 93.67
05:20:01 AM 18 6.21 0.00 12.08 0.04 0.00 81.67
05:20:01 AM 19 4.66 0.00 7.26 0.00 0.00 88.08
05:20:01 AM 20 0.70 0.00 4.83 0.00 0.00 94.47
05:20:01 AM 21 5.03 0.00 5.96 0.00 0.00 89.00
05:20:01 AM 22 7.01 0.00 5.47 0.00 0.00 87.52
05:20:01 AM 23 0.30 0.00 4.15 0.00 0.00 95.55
05:20:01 AM CPU %user %nice %system %iowait %steal %idle
05:30:01 AM all 4.34 0.00 11.07 0.01 0.00 84.58
05:30:01 AM 0 5.33 0.00 16.49 0.02 0.00 78.17
05:30:01 AM 1 17.19 0.00 33.85 0.03 0.00 48.93
05:30:01 AM 2 11.77 0.00 25.47 0.00 0.00 62.76
05:30:01 AM 3 5.08 0.00 21.68 0.00 0.00 73.23
05:30:01 AM 4 6.13 0.00 14.52 0.00 0.00 79.35
05:30:01 AM 5 2.06 0.00 11.81 0.00 0.00 86.13
05:30:01 AM 6 16.30 0.00 26.25 0.03 0.00 57.42
05:30:01 AM 7 3.80 0.00 17.13 0.01 0.00 79.06
05:30:01 AM 8 4.56 0.00 7.31 0.00 0.00 88.13
05:30:01 AM 9 0.11 0.00 2.77 0.01 0.00 97.11
05:30:01 AM 10 0.51 0.00 2.72 0.00 0.00 96.77
05:30:01 AM 11 4.69 0.00 2.10 0.00 0.00 93.21
05:30:01 AM 12 8.04 0.00 20.36 0.02 0.00 71.58
05:30:01 AM 13 1.30 0.00 12.20 0.00 0.00 86.50
05:30:01 AM 14 3.16 0.00 9.36 0.00 0.00 87.47
05:30:01 AM 15 0.51 0.00 4.93 0.01 0.00 94.55
05:30:01 AM 16 0.47 0.00 4.21 0.00 0.00 95.32
05:30:01 AM 17 0.43 0.00 3.02 0.00 0.00 96.55
05:30:01 AM 18 6.61 0.00 14.71 0.02 0.00 78.66
05:30:01 AM 19 3.60 0.00 6.18 0.00 0.00 90.22
05:30:01 AM 20 1.95 0.00 3.81 0.00 0.00 94.24
05:30:01 AM 21 0.46 0.00 2.55 0.00 0.00 96.99
05:30:01 AM 22 0.12 0.00 1.47 0.00 0.00 98.41
05:30:01 AM 23 0.13 0.00 0.88 0.00 0.00 98.99
05:30:01 AM CPU %user %nice %system %iowait %steal %idle
05:40:02 AM all 3.23 0.00 14.96 0.02 0.00 81.80
05:40:02 AM 0 5.70 0.00 21.45 0.18 0.00 72.67
05:40:02 AM 1 18.01 0.00 39.59 0.01 0.00 42.39
05:40:02 AM 2 9.73 0.00 33.71 0.01 0.00 56.55
05:40:02 AM 3 6.22 0.00 27.88 0.01 0.00 65.90
05:40:02 AM 4 2.83 0.00 20.78 0.00 0.00 76.39
05:40:02 AM 5 2.48 0.00 14.93 0.01 0.00 82.57
05:40:02 AM 6 5.30 0.00 35.07 0.05 0.00 59.57
05:40:02 AM 7 1.29 0.00 22.27 0.00 0.00 76.44
05:40:02 AM 8 0.77 0.00 9.50 0.00 0.00 89.73
05:40:02 AM 9 1.67 0.00 6.24 0.00 0.00 92.09
05:40:02 AM 10 0.88 0.00 4.54 0.00 0.00 94.59
05:40:02 AM 11 1.10 0.00 4.21 0.00 0.00 94.69
05:40:02 AM 12 9.25 0.00 26.51 0.01 0.00 64.23
05:40:02 AM 13 2.30 0.00 15.82 0.00 0.00 81.87
05:40:02 AM 14 1.31 0.00 11.75 0.00 0.00 86.94
05:40:02 AM 15 0.66 0.00 10.17 0.00 0.00 89.18
05:40:02 AM 16 1.17 0.00 5.83 0.00 0.00 93.01
05:40:02 AM 17 0.60 0.00 4.94 0.00 0.00 94.46
05:40:02 AM 18 2.62 0.00 20.10 0.12 0.00 77.16
05:40:02 AM 19 0.83 0.00 9.10 0.00 0.00 90.08
05:40:02 AM 20 0.75 0.00 5.42 0.00 0.00 93.82
05:40:02 AM 21 1.04 0.00 3.14 0.00 0.00 95.82
05:40:02 AM 22 0.28 0.00 2.73 0.00 0.00 96.99
05:40:02 AM 23 0.67 0.00 3.34 0.00 0.00 95.99
05:40:02 AM CPU %user %nice %system %iowait %steal %idle
05:50:01 AM all 2.28 0.00 11.49 0.04 0.00 86.19
05:50:01 AM 0 3.58 0.00 16.41 0.03 0.00 79.98
05:50:01 AM 1 14.45 0.00 32.73 0.01 0.00 52.82
05:50:01 AM 2 7.32 0.00 25.51 0.02 0.00 67.15
05:50:01 AM 3 3.46 0.00 20.23 0.00 0.00 76.31
05:50:01 AM 4 2.44 0.00 12.95 0.01 0.00 84.60
05:50:01 AM 5 1.33 0.00 12.12 0.00 0.00 86.55
05:50:01 AM 6 4.11 0.00 29.63 0.46 0.00 65.80
05:50:01 AM 7 1.67 0.00 14.28 0.06 0.00 83.99
05:50:01 AM 8 0.58 0.00 4.48 0.01 0.00 94.92
05:50:01 AM 9 0.24 0.00 4.33 0.02 0.00 95.41
05:50:01 AM 10 0.07 0.00 1.87 0.00 0.00 98.05
05:50:01 AM 11 0.08 0.00 1.36 0.00 0.00 98.55
05:50:01 AM 12 7.46 0.00 19.27 0.01 0.00 73.27
05:50:01 AM 13 1.08 0.00 13.72 0.00 0.00 85.20
05:50:01 AM 14 0.71 0.00 10.76 0.02 0.00 88.51
05:50:01 AM 15 0.66 0.00 7.16 0.01 0.00 92.17
05:50:01 AM 16 0.36 0.00 4.58 0.00 0.00 95.06
05:50:01 AM 17 0.49 0.00 3.64 0.00 0.00 95.87
05:50:01 AM 18 0.90 0.00 19.63 0.11 0.00 79.36
05:50:01 AM 19 0.76 0.00 6.19 0.03 0.00 93.02
05:50:01 AM 20 1.11 0.00 4.42 0.06 0.00 94.41
05:50:01 AM 21 0.74 0.00 3.29 0.02 0.00 95.95
05:50:01 AM 22 0.37 0.00 1.72 0.01 0.00 97.90
05:50:01 AM 23 0.78 0.00 5.67 0.04 0.00 93.50
05:50:01 AM CPU %user %nice %system %iowait %steal %idle
06:00:01 AM all 1.93 0.00 11.03 0.01 0.00 87.03
06:00:01 AM 0 3.39 0.00 17.41 0.02 0.00 79.18
06:00:01 AM 1 14.49 0.00 35.42 0.00 0.00 50.09
06:00:01 AM 2 5.96 0.00 27.15 0.00 0.00 66.89
06:00:01 AM 3 3.31 0.00 18.51 0.02 0.00 78.16
06:00:01 AM 4 1.45 0.00 11.41 0.00 0.00 87.15
06:00:01 AM 5 0.50 0.00 7.73 0.00 0.00 91.77
06:00:01 AM 6 2.20 0.00 35.46 0.04 0.00 62.30
06:00:01 AM 7 1.59 0.00 15.17 0.01 0.00 83.24
06:00:01 AM 8 0.23 0.00 3.56 0.01 0.00 96.21
06:00:01 AM 9 0.14 0.00 2.01 0.00 0.00 97.85
06:00:01 AM 10 0.13 0.00 1.37 0.00 0.00 98.51
06:00:01 AM 11 0.04 0.00 0.73 0.00 0.00 99.23
06:00:01 AM 12 7.45 0.00 22.89 0.01 0.00 69.66
06:00:01 AM 13 1.45 0.00 13.04 0.00 0.00 85.51
06:00:01 AM 14 0.82 0.00 8.06 0.00 0.00 91.12
06:00:01 AM 15 0.31 0.00 4.05 0.00 0.00 95.63
06:00:01 AM 16 0.29 0.00 2.70 0.00 0.00 97.02
06:00:01 AM 17 0.13 0.00 0.97 0.00 0.00 98.90
06:00:01 AM 18 1.15 0.00 21.66 0.08 0.00 77.10
06:00:01 AM 19 0.43 0.00 7.24 0.00 0.00 92.33
06:00:01 AM 20 0.39 0.00 4.84 0.00 0.00 94.77
06:00:01 AM 21 0.25 0.00 1.94 0.00 0.00 97.81
06:00:01 AM 22 0.15 0.00 0.64 0.00 0.00 99.21
06:00:01 AM 23 0.13 0.00 0.91 0.00 0.00 98.96
06:00:01 AM CPU %user %nice %system %iowait %steal %idle
06:10:01 AM all 2.78 0.00 18.12 0.01 0.00 79.09
06:10:01 AM 0 4.13 0.00 25.91 0.07 0.00 69.89
06:10:01 AM 1 14.89 0.00 41.23 0.01 0.00 43.88
06:10:01 AM 2 8.86 0.00 35.86 0.01 0.00 55.28
06:10:01 AM 3 4.97 0.00 31.55 0.00 0.00 63.48
06:10:01 AM 4 2.86 0.00 25.14 0.00 0.00 72.00
06:10:01 AM 5 2.20 0.00 19.33 0.00 0.00 78.48
06:10:01 AM 6 5.45 0.00 36.19 0.02 0.00 58.34
06:10:01 AM 7 1.96 0.00 26.27 0.04 0.00 71.72
06:10:01 AM 8 1.26 0.00 12.71 0.03 0.00 86.01
06:10:01 AM 9 0.67 0.00 9.15 0.01 0.00 90.18
06:10:01 AM 10 0.55 0.00 5.23 0.00 0.00 94.22
06:10:01 AM 11 0.74 0.00 5.65 0.00 0.00 93.61
06:10:01 AM 12 6.99 0.00 28.78 0.01 0.00 64.23
06:10:01 AM 13 1.61 0.00 19.93 0.00 0.00 78.46
06:10:01 AM 14 1.53 0.00 16.81 0.00 0.00 81.66
06:10:01 AM 15 1.30 0.00 13.27 0.00 0.00 85.43
06:10:01 AM 16 1.07 0.00 10.22 0.00 0.00 88.71
06:10:01 AM 17 0.93 0.00 8.63 0.00 0.00 90.44
06:10:01 AM 18 0.93 0.00 24.76 0.03 0.00 74.28
06:10:01 AM 19 0.90 0.00 13.35 0.01 0.00 85.74
06:10:01 AM 20 0.85 0.00 9.04 0.00 0.00 90.11
06:10:01 AM 21 1.00 0.00 6.38 0.02 0.00 92.60
06:10:01 AM 22 0.54 0.00 6.25 0.00 0.00 93.21
06:10:01 AM 23 0.46 0.00 3.31 0.01 0.00 96.23
If you notice, System time increased after 5:20 am.
Strace output of php process
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
41.96 0.001331 3 438 poll
30.14 0.000956 2 548 munmap
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
52.21 0.001264 6 222 munmap
11.61 0.000281 1 247 poll
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
44.56 0.001159 5 225 munmap
37.99 0.000988 3 320 poll
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.000539 67 8 munmap
0.00 0.000000 0 119 write
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
51.37 0.001034 6 161 munmap
21.56 0.000434 1 293 poll
How to troubleshoot high load caused by php7
UPDATE 1
- This problem does not occurs when total RAM on system is low (<32GB) but it is occurring on system with high RAM (256GB).
- Restarting PHP sometimes fixes this problem for few days.
