0

We upgraded to php7 (this was only change in server), this is causing high load on CPU.

Cacti graph

enter image description here

sar output  
04:40:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:50:01 AM     all      2.85      0.00      2.13      0.01      0.00     95.01
04:50:01 AM       0      4.53      0.00      8.41      0.03      0.00     87.03
04:50:01 AM       1     24.64      0.00      3.93      0.01      0.00     71.43
04:50:01 AM       2      9.01      0.00      4.28      0.00      0.00     86.71
04:50:01 AM       3      3.89      0.00      2.23      0.00      0.00     93.87
04:50:01 AM       4      0.49      0.00      0.20      0.00      0.00     99.31
04:50:01 AM       5      0.38      0.00      0.79      0.00      0.00     98.83
04:50:01 AM       6      5.15      0.00     13.87      0.04      0.00     80.94
04:50:01 AM       7      1.18      0.00      5.05      0.00      0.00     93.76
04:50:01 AM       8      0.61      0.00      1.94      0.00      0.00     97.45
04:50:01 AM       9      0.37      0.00      1.33      0.00      0.00     98.30
04:50:01 AM      10      0.05      0.00      0.30      0.00      0.00     99.64
04:50:01 AM      11      0.11      0.00      0.03      0.00      0.00     99.85
04:50:01 AM      12     13.35      0.00      2.62      0.02      0.00     84.00
04:50:01 AM      13      0.98      0.00      0.24      0.00      0.00     98.78
04:50:01 AM      14      0.59      0.00      0.36      0.00      0.00     99.05
04:50:01 AM      15      0.24      0.00      0.13      0.00      0.00     99.63
04:50:01 AM      16      0.24      0.00      0.14      0.00      0.00     99.62
04:50:01 AM      17      0.03      0.00      0.03      0.00      0.00     99.94
04:50:01 AM      18      0.99      0.00      3.73      0.02      0.00     95.27
04:50:01 AM      19      0.37      0.00      0.54      0.00      0.00     99.09
04:50:01 AM      20      0.62      0.00      0.56      0.00      0.00     98.82
04:50:01 AM      21      0.24      0.00      0.28      0.00      0.00     99.48
04:50:01 AM      22      0.29      0.00      0.21      0.00      0.00     99.49
04:50:01 AM      23      0.07      0.00      0.07      0.00      0.00     99.87

04:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:00:01 AM     all      5.66      0.00      2.19      0.01      0.00     92.13
05:00:01 AM       0      7.73      0.00      7.75      0.05      0.00     84.47
05:00:01 AM       1     49.79      0.00      8.65      0.00      0.00     41.55
05:00:01 AM       2     25.74      0.00      5.77      0.00      0.00     68.50
05:00:01 AM       3      4.95      0.00      1.42      0.00      0.00     93.63
05:00:01 AM       4      1.89      0.00      1.69      0.00      0.00     96.42
05:00:01 AM       5      0.38      0.00      0.97      0.00      0.00     98.65
05:00:01 AM       6      9.79      0.00      3.96      0.10      0.00     86.15
05:00:01 AM       7      2.09      0.00      5.39      0.02      0.00     92.50
05:00:01 AM       8      0.31      0.00      0.24      0.01      0.00     99.44
05:00:01 AM       9      0.76      0.00      1.08      0.00      0.00     98.16
05:00:01 AM      10      0.02      0.00      0.13      0.01      0.00     99.84
05:00:01 AM      11      0.33      0.00      1.33      0.00      0.00     98.34
05:00:01 AM      12     25.21      0.00      3.81      0.02      0.00     70.96
05:00:01 AM      13      0.90      0.00      0.66      0.00      0.00     98.44
05:00:01 AM      14      0.67      0.00      0.76      0.05      0.00     98.52
05:00:01 AM      15      0.31      0.00      0.27      0.00      0.00     99.42
05:00:01 AM      16      0.28      0.00      0.26      0.00      0.00     99.46
05:00:01 AM      17      0.12      0.00      0.18      0.00      0.00     99.69
05:00:01 AM      18      1.24      0.00      5.03      0.04      0.00     93.69
05:00:01 AM      19      0.89      0.00      0.87      0.03      0.00     98.20
05:00:01 AM      20      1.74      0.00      1.23      0.00      0.00     97.02
05:00:01 AM      21      0.44      0.00      0.42      0.00      0.00     99.14
05:00:01 AM      22      0.38      0.00      0.45      0.00      0.00     99.17
05:00:01 AM      23      0.09      0.00      0.47      0.00      0.00     99.44

05:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:10:01 AM     all      6.16      0.00      1.64      0.02      0.00     92.18
05:10:01 AM       0      6.03      0.00      5.99      0.09      0.00     87.90
05:10:01 AM       1     48.29      0.00      7.89      0.00      0.00     43.81
05:10:01 AM       2     30.24      0.00      5.09      0.00      0.00     64.67
05:10:01 AM       3      9.02      0.00      1.20      0.00      0.00     89.77
05:10:01 AM       4      2.13      0.00      0.36      0.00      0.00     97.51
05:10:01 AM       5      0.54      0.00      0.16      0.00      0.00     99.30
05:10:01 AM       6     10.47      0.00      5.23      0.15      0.00     84.15
05:10:01 AM       7      1.90      0.00      1.13      0.02      0.00     96.95
05:10:01 AM       8      0.66      0.00      1.85      0.01      0.00     97.47
05:10:01 AM       9      0.26      0.00      0.13      0.00      0.00     99.61
05:10:01 AM      10      0.02      0.00      0.08      0.00      0.00     99.90
05:10:01 AM      11      0.01      0.00      0.08      0.00      0.00     99.92
05:10:01 AM      12     28.45      0.00      3.41      0.06      0.00     68.09
05:10:01 AM      13      1.60      0.00      0.73      0.00      0.00     97.67
05:10:01 AM      14      0.55      0.00      0.67      0.00      0.00     98.78
05:10:01 AM      15      0.66      0.00      0.24      0.00      0.00     99.09
05:10:01 AM      16      0.44      0.00      0.18      0.00      0.00     99.39
05:10:01 AM      17      0.19      0.00      0.12      0.00      0.00     99.69
05:10:01 AM      18      0.39      0.00      0.53      0.03      0.00     99.05
05:10:01 AM      19      2.51      0.00      1.42      0.00      0.00     96.07
05:10:01 AM      20      1.11      0.00      1.15      0.00      0.00     97.74
05:10:01 AM      21      1.99      0.00      1.25      0.00      0.00     96.76
05:10:01 AM      22      0.35      0.00      0.35      0.00      0.00     99.29
05:10:01 AM      23      0.22      0.00      0.34      0.01      0.00     99.44

05:10:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:20:01 AM     all      4.02      0.00     12.27      0.01      0.00     83.69
05:20:01 AM       0      7.35      0.00     18.25      0.02      0.00     74.38
05:20:01 AM       1     11.90      0.00     31.41      0.05      0.00     56.64
05:20:01 AM       2      6.57      0.00     25.74      0.00      0.00     67.68
05:20:01 AM       3      3.52      0.00     19.25      0.00      0.00     77.23
05:20:01 AM       4      2.58      0.00     16.29      0.00      0.00     81.14
05:20:01 AM       5      2.12      0.00     12.28      0.02      0.00     85.58
05:20:01 AM       6     12.51      0.00     25.54      0.06      0.00     61.88
05:20:01 AM       7      4.24      0.00     16.93      0.00      0.00     78.82
05:20:01 AM       8      3.75      0.00      8.85      0.01      0.00     87.40
05:20:01 AM       9      2.45      0.00      5.40      0.01      0.00     92.15
05:20:01 AM      10      2.79      0.00      4.57      0.00      0.00     92.64
05:20:01 AM      11      1.00      0.00      5.14      0.00      0.00     93.86
05:20:01 AM      12      5.14      0.00     19.32      0.02      0.00     75.52
05:20:01 AM      13      2.28      0.00     16.37      0.00      0.00     81.35
05:20:01 AM      14      1.26      0.00     11.01      0.00      0.00     87.73
05:20:01 AM      15      0.80      0.00      8.60      0.00      0.00     90.60
05:20:01 AM      16      0.93      0.00      5.01      0.00      0.00     94.06
05:20:01 AM      17      1.41      0.00      4.92      0.00      0.00     93.67
05:20:01 AM      18      6.21      0.00     12.08      0.04      0.00     81.67
05:20:01 AM      19      4.66      0.00      7.26      0.00      0.00     88.08
05:20:01 AM      20      0.70      0.00      4.83      0.00      0.00     94.47
05:20:01 AM      21      5.03      0.00      5.96      0.00      0.00     89.00
05:20:01 AM      22      7.01      0.00      5.47      0.00      0.00     87.52
05:20:01 AM      23      0.30      0.00      4.15      0.00      0.00     95.55

05:20:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:30:01 AM     all      4.34      0.00     11.07      0.01      0.00     84.58
05:30:01 AM       0      5.33      0.00     16.49      0.02      0.00     78.17
05:30:01 AM       1     17.19      0.00     33.85      0.03      0.00     48.93
05:30:01 AM       2     11.77      0.00     25.47      0.00      0.00     62.76
05:30:01 AM       3      5.08      0.00     21.68      0.00      0.00     73.23
05:30:01 AM       4      6.13      0.00     14.52      0.00      0.00     79.35
05:30:01 AM       5      2.06      0.00     11.81      0.00      0.00     86.13
05:30:01 AM       6     16.30      0.00     26.25      0.03      0.00     57.42
05:30:01 AM       7      3.80      0.00     17.13      0.01      0.00     79.06
05:30:01 AM       8      4.56      0.00      7.31      0.00      0.00     88.13
05:30:01 AM       9      0.11      0.00      2.77      0.01      0.00     97.11
05:30:01 AM      10      0.51      0.00      2.72      0.00      0.00     96.77
05:30:01 AM      11      4.69      0.00      2.10      0.00      0.00     93.21
05:30:01 AM      12      8.04      0.00     20.36      0.02      0.00     71.58
05:30:01 AM      13      1.30      0.00     12.20      0.00      0.00     86.50
05:30:01 AM      14      3.16      0.00      9.36      0.00      0.00     87.47
05:30:01 AM      15      0.51      0.00      4.93      0.01      0.00     94.55
05:30:01 AM      16      0.47      0.00      4.21      0.00      0.00     95.32
05:30:01 AM      17      0.43      0.00      3.02      0.00      0.00     96.55
05:30:01 AM      18      6.61      0.00     14.71      0.02      0.00     78.66
05:30:01 AM      19      3.60      0.00      6.18      0.00      0.00     90.22
05:30:01 AM      20      1.95      0.00      3.81      0.00      0.00     94.24
05:30:01 AM      21      0.46      0.00      2.55      0.00      0.00     96.99
05:30:01 AM      22      0.12      0.00      1.47      0.00      0.00     98.41
05:30:01 AM      23      0.13      0.00      0.88      0.00      0.00     98.99

05:30:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:40:02 AM     all      3.23      0.00     14.96      0.02      0.00     81.80
05:40:02 AM       0      5.70      0.00     21.45      0.18      0.00     72.67
05:40:02 AM       1     18.01      0.00     39.59      0.01      0.00     42.39
05:40:02 AM       2      9.73      0.00     33.71      0.01      0.00     56.55
05:40:02 AM       3      6.22      0.00     27.88      0.01      0.00     65.90
05:40:02 AM       4      2.83      0.00     20.78      0.00      0.00     76.39
05:40:02 AM       5      2.48      0.00     14.93      0.01      0.00     82.57
05:40:02 AM       6      5.30      0.00     35.07      0.05      0.00     59.57
05:40:02 AM       7      1.29      0.00     22.27      0.00      0.00     76.44
05:40:02 AM       8      0.77      0.00      9.50      0.00      0.00     89.73
05:40:02 AM       9      1.67      0.00      6.24      0.00      0.00     92.09
05:40:02 AM      10      0.88      0.00      4.54      0.00      0.00     94.59
05:40:02 AM      11      1.10      0.00      4.21      0.00      0.00     94.69
05:40:02 AM      12      9.25      0.00     26.51      0.01      0.00     64.23
05:40:02 AM      13      2.30      0.00     15.82      0.00      0.00     81.87
05:40:02 AM      14      1.31      0.00     11.75      0.00      0.00     86.94
05:40:02 AM      15      0.66      0.00     10.17      0.00      0.00     89.18
05:40:02 AM      16      1.17      0.00      5.83      0.00      0.00     93.01
05:40:02 AM      17      0.60      0.00      4.94      0.00      0.00     94.46
05:40:02 AM      18      2.62      0.00     20.10      0.12      0.00     77.16
05:40:02 AM      19      0.83      0.00      9.10      0.00      0.00     90.08
05:40:02 AM      20      0.75      0.00      5.42      0.00      0.00     93.82
05:40:02 AM      21      1.04      0.00      3.14      0.00      0.00     95.82
05:40:02 AM      22      0.28      0.00      2.73      0.00      0.00     96.99
05:40:02 AM      23      0.67      0.00      3.34      0.00      0.00     95.99

05:40:02 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:50:01 AM     all      2.28      0.00     11.49      0.04      0.00     86.19
05:50:01 AM       0      3.58      0.00     16.41      0.03      0.00     79.98
05:50:01 AM       1     14.45      0.00     32.73      0.01      0.00     52.82
05:50:01 AM       2      7.32      0.00     25.51      0.02      0.00     67.15
05:50:01 AM       3      3.46      0.00     20.23      0.00      0.00     76.31
05:50:01 AM       4      2.44      0.00     12.95      0.01      0.00     84.60
05:50:01 AM       5      1.33      0.00     12.12      0.00      0.00     86.55
05:50:01 AM       6      4.11      0.00     29.63      0.46      0.00     65.80
05:50:01 AM       7      1.67      0.00     14.28      0.06      0.00     83.99
05:50:01 AM       8      0.58      0.00      4.48      0.01      0.00     94.92
05:50:01 AM       9      0.24      0.00      4.33      0.02      0.00     95.41
05:50:01 AM      10      0.07      0.00      1.87      0.00      0.00     98.05
05:50:01 AM      11      0.08      0.00      1.36      0.00      0.00     98.55
05:50:01 AM      12      7.46      0.00     19.27      0.01      0.00     73.27
05:50:01 AM      13      1.08      0.00     13.72      0.00      0.00     85.20
05:50:01 AM      14      0.71      0.00     10.76      0.02      0.00     88.51
05:50:01 AM      15      0.66      0.00      7.16      0.01      0.00     92.17
05:50:01 AM      16      0.36      0.00      4.58      0.00      0.00     95.06
05:50:01 AM      17      0.49      0.00      3.64      0.00      0.00     95.87
05:50:01 AM      18      0.90      0.00     19.63      0.11      0.00     79.36
05:50:01 AM      19      0.76      0.00      6.19      0.03      0.00     93.02
05:50:01 AM      20      1.11      0.00      4.42      0.06      0.00     94.41
05:50:01 AM      21      0.74      0.00      3.29      0.02      0.00     95.95
05:50:01 AM      22      0.37      0.00      1.72      0.01      0.00     97.90
05:50:01 AM      23      0.78      0.00      5.67      0.04      0.00     93.50

05:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
06:00:01 AM     all      1.93      0.00     11.03      0.01      0.00     87.03
06:00:01 AM       0      3.39      0.00     17.41      0.02      0.00     79.18
06:00:01 AM       1     14.49      0.00     35.42      0.00      0.00     50.09
06:00:01 AM       2      5.96      0.00     27.15      0.00      0.00     66.89
06:00:01 AM       3      3.31      0.00     18.51      0.02      0.00     78.16
06:00:01 AM       4      1.45      0.00     11.41      0.00      0.00     87.15
06:00:01 AM       5      0.50      0.00      7.73      0.00      0.00     91.77
06:00:01 AM       6      2.20      0.00     35.46      0.04      0.00     62.30
06:00:01 AM       7      1.59      0.00     15.17      0.01      0.00     83.24
06:00:01 AM       8      0.23      0.00      3.56      0.01      0.00     96.21
06:00:01 AM       9      0.14      0.00      2.01      0.00      0.00     97.85
06:00:01 AM      10      0.13      0.00      1.37      0.00      0.00     98.51
06:00:01 AM      11      0.04      0.00      0.73      0.00      0.00     99.23
06:00:01 AM      12      7.45      0.00     22.89      0.01      0.00     69.66
06:00:01 AM      13      1.45      0.00     13.04      0.00      0.00     85.51
06:00:01 AM      14      0.82      0.00      8.06      0.00      0.00     91.12
06:00:01 AM      15      0.31      0.00      4.05      0.00      0.00     95.63
06:00:01 AM      16      0.29      0.00      2.70      0.00      0.00     97.02
06:00:01 AM      17      0.13      0.00      0.97      0.00      0.00     98.90
06:00:01 AM      18      1.15      0.00     21.66      0.08      0.00     77.10
06:00:01 AM      19      0.43      0.00      7.24      0.00      0.00     92.33
06:00:01 AM      20      0.39      0.00      4.84      0.00      0.00     94.77
06:00:01 AM      21      0.25      0.00      1.94      0.00      0.00     97.81
06:00:01 AM      22      0.15      0.00      0.64      0.00      0.00     99.21
06:00:01 AM      23      0.13      0.00      0.91      0.00      0.00     98.96

06:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
06:10:01 AM     all      2.78      0.00     18.12      0.01      0.00     79.09
06:10:01 AM       0      4.13      0.00     25.91      0.07      0.00     69.89
06:10:01 AM       1     14.89      0.00     41.23      0.01      0.00     43.88
06:10:01 AM       2      8.86      0.00     35.86      0.01      0.00     55.28
06:10:01 AM       3      4.97      0.00     31.55      0.00      0.00     63.48
06:10:01 AM       4      2.86      0.00     25.14      0.00      0.00     72.00
06:10:01 AM       5      2.20      0.00     19.33      0.00      0.00     78.48
06:10:01 AM       6      5.45      0.00     36.19      0.02      0.00     58.34
06:10:01 AM       7      1.96      0.00     26.27      0.04      0.00     71.72
06:10:01 AM       8      1.26      0.00     12.71      0.03      0.00     86.01
06:10:01 AM       9      0.67      0.00      9.15      0.01      0.00     90.18
06:10:01 AM      10      0.55      0.00      5.23      0.00      0.00     94.22
06:10:01 AM      11      0.74      0.00      5.65      0.00      0.00     93.61
06:10:01 AM      12      6.99      0.00     28.78      0.01      0.00     64.23
06:10:01 AM      13      1.61      0.00     19.93      0.00      0.00     78.46
06:10:01 AM      14      1.53      0.00     16.81      0.00      0.00     81.66
06:10:01 AM      15      1.30      0.00     13.27      0.00      0.00     85.43
06:10:01 AM      16      1.07      0.00     10.22      0.00      0.00     88.71
06:10:01 AM      17      0.93      0.00      8.63      0.00      0.00     90.44
06:10:01 AM      18      0.93      0.00     24.76      0.03      0.00     74.28
06:10:01 AM      19      0.90      0.00     13.35      0.01      0.00     85.74
06:10:01 AM      20      0.85      0.00      9.04      0.00      0.00     90.11
06:10:01 AM      21      1.00      0.00      6.38      0.02      0.00     92.60
06:10:01 AM      22      0.54      0.00      6.25      0.00      0.00     93.21
06:10:01 AM      23      0.46      0.00      3.31      0.01      0.00     96.23

If you notice, System time increased after 5:20 am.

Strace output of php process
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 41.96    0.001331           3       438           poll
 30.14    0.000956           2       548           munmap

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 52.21    0.001264           6       222           munmap
 11.61    0.000281           1       247           poll

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 44.56    0.001159           5       225           munmap
 37.99    0.000988           3       320           poll

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000539          67         8           munmap
  0.00    0.000000           0       119           write

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 51.37    0.001034           6       161           munmap
 21.56    0.000434           1       293           poll

How to troubleshoot high load caused by php7

UPDATE 1

  1. This problem does not occurs when total RAM on system is low (<32GB) but it is occurring on system with high RAM (256GB).
  2. Restarting PHP sometimes fixes this problem for few days.
1

All of the time appears to be going on munmap. Which is unmapping memory. PHP7 has the capability to use Transparent Huge Pages in the Kernel.

Transparent huge pages have caused issues in the past with different kernels and software. https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html shows how this can be an issue and how to turn it off.

I have had problems with Huge Pages and PostgreSQL on RHEL 6 using this kernel. Turning off huge pages resolved the issues.

You have also submitted https://bugs.php.net/bug.php?id=72322 directly to the PHP project to get answers there as well.

0

There is nowhere near enough information in the post to provide a definitive answer to what's causing this.

How are you changing the amount of RAM?

You've tagged this as CentOS 6 - but I don't think PHP7 is available from the standard build repo - where did it come from? Where did the previous version of PHP come from?

A random guess would be that you've not sized the cache correctly - but you didn't say what you upgraded from (APC has very different characteristics than opcache) but it could also be caused by a switch from mysql to mysqli (if you are using a mysql client at all and your codebase does very sloppy result set management).

Your first step in troubleshooting this is identifying exactly what has changed.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.