0

This morning my server went down for about an hour I'm using ubuntu 16.04LTS on AWS EC2.

some info: it's a t2.medium instance, I recently upgraded from ubuntu 14.04 to 16.04, updated lots of packages, this is a production instance, I'm using PHP Version 5.6.40-10+ubuntu16.04.1+deb.sury.org+1 and for some sites I'm using the latest php via fast-cgi php modules, my device memory /xvda1 is at 49% storage remaining.

From syslog:

Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Starting Clean php session files...
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Stopping User Manager for UID 110...
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Reached target Shutdown.
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Starting Exit the Session...
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Default.
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Basic System.
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Timers.
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Sockets.
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Paths.
    Oct 11 05:09:10 ip-10-0-x-xxx CRON[28971]: (root) CMD (  [ -

x/usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Oct 11 05:09:10 ip-10-0-x-xxx CRON[28972]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Received SIGRTMIN+24 from PID 28967 (kill).
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Stopped User Manager for UID 110.
    Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Removed slice User Slice of solr.
    Oct 11 05:10:32 ip-10-0-x-xxx kernel: [488908.756151] apache2 invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
    Oct 11 05:10:32 ip-10-0-x-xxx kernel: [488908.756155] apache2 cpuset=/ mems_allowed=0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756160] CPU: 1 PID: 29346 Comm: apache2 Not tainted 4.4.0-157-generic #185-Ubuntu
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756162] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756163]  0000000000000286 3f04de0b6dc470df ffff88005a2739e8 ffffffff8140c9a1
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756166]  ffff88005a273ba0 ffff8800e84e4600 ffff88005a273a58 ffffffff81219bde
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756169]  ffff8800eaf61c00 ffff88005a273ba0 ffff88005a273ab0 ffff8800e86a5400
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756171] Call Trace:
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756180]  [<ffffffff8140c9a1>] dump_stack+0x63/0x82
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756184]  [<ffffffff81219bde>] dump_header+0x5a/0x1c3
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756189]  [<ffffffff8119db2b>] oom_kill_process+0x20b/0x3d0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756191]  [<ffffffff8119df38>] out_of_memory+0x1f8/0x460
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756196]  [<ffffffff811a3f73>] __alloc_pages_slowpath.constprop.88+0x943/0xaf0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756199]  [<ffffffff811a43a8>] __alloc_pages_nodemask+0x288/0x2a0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756203]  [<ffffffff811efc6c>] alloc_pages_current+0x8c/0x110
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756205]  [<ffffffff81199c5b>] __page_cache_alloc+0xab/0xc0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756208]  [<ffffffff8119c580>] filemap_fault+0x160/0x440
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756211]  [<ffffffff812b3016>] ext4_filemap_fault+0x36/0x50
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756215]  [<ffffffff811c9d37>] __do_fault+0x77/0x110
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756218]  [<ffffffff811cdcdc>] handle_mm_fault+0x125c/0x1b80
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756222]  [<ffffffff81860041>] ? __schedule+0x301/0x810
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756224]  [<ffffffff8186004d>] ? __schedule+0x30d/0x810
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756225]  [<ffffffff81860041>] ? __schedule+0x301/0x810
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756227]  [<ffffffff8186004d>] ? __schedule+0x30d/0x810
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756229]  [<ffffffff81860041>] ? __schedule+0x301/0x810
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756231]  [<ffffffff81860041>] ? __schedule+0x301/0x810
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756235]  [<ffffffff8106edd4>] __do_page_fault+0x1a4/0x410
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756237]  [<ffffffff8106f062>] do_page_fault+0x22/0x30
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756241]  [<ffffffff81867ae8>] page_fault+0x28/0x30
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756242] Mem-Info:
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] active_anon:919500 inactive_anon:18067 isolated_anon:0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247]  active_file:620 inactive_file:764 isolated_file:0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247]  unevictable:0 dirty:0 writeback:0 unstable:0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247]  slab_reclaimable:8517 slab_unreclaimable:10839
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247]  mapped:25419 shmem:36238 pagetables:21622 bounce:0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247]  free:21367 free_pcp:140 free_cma:0
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756251] Node 0 DMA free:15820kB min:268kB low:332kB high:400kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756256] lowmem_reserve[]: 0 3704 3893 3893 3893
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756260] Node 0 DMA32 free:64756kB min:64040kB low:80048kB high:96060kB active_anon:3513132kB inactive_anon:69560kB active_file:2472kB inactive_file:3028kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3834888kB mlocked:0kB dirty:0kB writeback:0kB mapped:95428kB shmem:137276kB slab_reclaimable:32652kB slab_unreclaimable:38928kB kernel_stack:4192kB pagetables:83352kB unstable:0kB bounce:0kB free_pcp:560kB local_pcp:400kB free_cma:0kB writeback_tmp:0kB pages_scanned:37356 all_unreclaimable? yes
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756265] lowmem_reserve[]: 0 0 189 189 189
    Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756268] Node 0 Normal free:4892kB min:3268kB low:4084kB high:4900kB active_anon:164868kB inactive_anon:2708kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:193808kB mlocked:0kB dirty:0kB writeback:0kB mapped:6240kB shmem:7676kB slab_reclaimable:1416kB slab_unreclaimable:4428kB kernel_stack:720kB pagetables:3136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Then I get an huge list of this:

    Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174008] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174009] 37420 total pagecache pages
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174011] 0 pages in swap cache
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174012] Swap cache stats: add 0, delete 0, find 0/0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174013] Free swap  = 0kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174014] Total swap = 0kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174015] 1048477 pages RAM
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174016] 0 pages HighMem/MovableOnly
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174017] 37328 pages reserved
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174018] 0 pages cma reserved
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174019] 0 pages hwpoisoned
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174020] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174025] [  275]     0   275    10189      689      21       3        0             0 systemd-journal
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174028] [  303]     0   303    11224      309      23       3        0         -1000 systemd-udevd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174030] [  532]     0   532     7240      179      18       3        0             0 systemd-logind
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174032] [  534]     0   534     7470       52      20       3        0             0 cgmanager
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174034] [  536]     0   536     6512       49      18       3        0             0 atd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174037] [  544]     0   544     6933       71      18       3        0             0 cron
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174039] [  550]   102   550    10762      176      25       3        0          -900 dbus-daemon
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174041] [  600]     0   600     1099       16       7       3        0             0 acpid
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174043] [  621]     0   621    68655      244      36       3        0             0 accounts-daemon
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174045] [  626]   101   626    65157      313      29       3        0             0 rsyslogd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174047] [  670]     0   670     4869       65      15       3        0             0 irqbalance
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174050] [  689]     0   689    13193     2548      29       3        0             0 munin-node
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174052] [  691]     0   691     4031      216      11       3        0             0 dhclient
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174054] [  698]     0   698    69861      342      39       3        0             0 polkitd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174056] [  735]     0   735     9667      180      22       3        0             0 monit
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174058] [ 1228]     0  1228   133721     6781      85       5        0          -500 dockerd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174060] [ 1248]     0  1248   109978     1466     139       4        0             0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174063] [ 1253]     0  1253    43332     2030      52       3        0             0 unattended-upgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174065] [ 1255]     0  1255    91709     3642     139       3        0             0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174067] [ 1321]   107  1321    28735      291      53       3        0             0 opendkim
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174069] [ 1324]     0  1324    16379      177      35       3        0         -1000 sshd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174072] [ 1327]   107  1327    88165      827      62       3        0             0 opendkim
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174074] [ 1352]     0  1352    43196      733      52       5        0             0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174076] [ 1353]    33  1353    43196      733      54       5        0             0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174078] [ 1354]    33  1354    43196      733      54       5        0             0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174080] [ 1355]    33  1355    43196      733      54       5        0             0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174087] [ 1362]     0  1362     3664       33      12       3        0             0 agetty
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174089] [ 1371]    33  1371   109978     1327     125       4        0             0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174091] [ 1372]    33  1372   109978     1327     125       4        0             0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174093] [ 1373]    33  1373   112297     9698     150       3        0             0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174095] [ 1374]    33  1374   112315    10265     151       3        0             0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174098] [ 1492]     0  1492    93681     2281      40       5        0          -500 docker-containe
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174100] [ 1506]     0  1506    16352      120      24       3        0             0 master
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174102] [ 1974]   111  1974    27509      166      26       3        0             0 ntpd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174105] [27112]    33 27112   111777     8536     150       3        0             0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174107] [30935]     0 30935   146764     2750      54       4        0             0 fail2ban-server
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174109] [ 6574]     0  6574   122194     3926     179       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174111] [27782]    33 27782    44694     2331      70       3        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174114] [27811]   108 27811    23037      213      45       3        0             0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174116] [27812]   108 27812    23037      212      45       3        0             0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174118] [27813]   108 27813    23037      211      45       3        0             0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174120] [27814]   108 27814    23037      211      45       3        0             0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174122] [27815]   108 27815    23037      211      45       3        0             0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174124] [27816]   108 27816    23037      218      45       3        0             0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174127] [27291]   106 27291    16911      131      25       3        0             0 qmgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174129] [27413]   106 27413    20177      184      30       3        0             0 tlsmgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174131] [26417]   106 26417    16869      112      25       3        0             0 pickup
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174133] [27850]    33 27850   163843    26103     224       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174136] [27856]    33 27856   161347    23458     215       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174138] [27895]    33 27895   162572    24791     221       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174140] [28374]    33 28374   163657    33629     236       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174142] [28377]    33 28377   142967    31279     231       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174144] [28379]    33 28379   144883    32799     235       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174146] [28423]    33 28423   162756    24751     222       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174149] [28429]    33 28429   161587    31448     232       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174151] [28434]    33 28434   161233    23432     219       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174153] [28463]    33 28463   163658    33523     236       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174155] [28464]    33 28464   122401     2826     177       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174157] [28467]    33 28467   143221    31432     231       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174159] [28468]    33 28468   122473     2965     177       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174161] [28469]    33 28469   129756    26267     220       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174164] [28482]    33 28482   161283    22980     219       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174166] [28483]    33 28483   163618    25643     223       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174168] [28484]    33 28484   163367    25141     223       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174170] [28486]    33 28486   143209    31129     231       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174172] [28488]    33 28488   153425    24734     220       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174174] [28492]    33 28492   161351    23526     215       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174176] [28500]    33 28500   163574    25769     221       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174178] [28501]    33 28501   163577    26837     224       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174180] [28503]    33 28503   142863    23211     214       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174182] [28865]    33 28865   161160    23608     218       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174184] [28873]    33 28873   161345    23207     215       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174186] [28890]    33 28890   163621    25344     223       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174188] [28891]    33 28891   122250     2748     166       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174190] [28892]    33 28892   162593    24527     217       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174193] [28894]    33 28894   127346     7830     186       4        0             0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174195] [28899]    33 28899   163557    25511     223       4        0             0 apache2
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.936086] INFO: task apache2:28423 blocked for more than 120 seconds.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.943206]       Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.948135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955958] apache2         D ffff8800e674fc58     0 28423   6574 0x00000000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955963]  ffff8800e674fc58 0000000000000001 ffff880021adc600 ffff88001dac8000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955966]  ffff8800e6750000 ffff880034195984 ffff88001dac8000 00000000ffffffff
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955969]  ffff880034195988 ffff8800e674fc70 ffffffff81860585 ffff880034195980
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955971] Call Trace:
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955981]  [<ffffffff81860585>] schedule+0x35/0x80
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955989]  [<ffffffff818608de>] schedule_preempt_disabled+0xe/0x10
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955992]  [<ffffffff81862777>] __mutex_lock_slowpath+0xb7/0x130
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955994]  [<ffffffff8186280f>] mutex_lock+0x1f/0x30
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955998]  [<ffffffff8122bbf2>] path_openat+0x512/0x1360
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956001]  [<ffffffff8122e611>] do_filp_open+0x91/0x100
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956004]  [<ffffffff8123c2e7>] ? __alloc_fd+0xc7/0x190
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956007]  [<ffffffff8121c8c8>] do_sys_open+0x138/0x2a0
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956009]  [<ffffffff8121ca4e>] SyS_open+0x1e/0x20
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956012]  [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956026] INFO: task apache2:29254 blocked for more than 120 seconds.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.962952]       Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.977383] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992125] apache2         D ffff880013db3bf8     0 29254   6574 0x00000000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992131]  ffff880013db3bf8 0000000000000000 ffffffff81e13500 ffff8800e8701c00
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992135]  ffff880013db4000 ffff880034195984 ffff8800e8701c00 00000000ffffffff
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992137]  ffff880034195988 ffff880013db3c10 ffffffff81860585 ffff880034195980
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992140] Call Trace:

Possible reasons: Cron is running sessionclean at the same time as systemd starts clean php session files? It's simply not cleaning php sessions correctly? My RAM is too low?

    free -m #see memory info
              total        used        free      shared  buff/cache   available
Mem:           3949        1710        1397         141         842        1793
Swap:             0           0           0

Also since I updated the OS my sites are not using php5 anymore but that's an cron job from before the update that migrated over, maybe that could be an issue? Thanks!

1

You might want to double-check your php version. I have php7.0 as the bundled version in my Ubuntu 16.04 installs. IIRC, PHP5 has been deprecated/left-for-dead

Also, this suggests you're running out of RAM:

apache2 invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0

Possibly add more RAM, disk swap, or at least a swap file and see if things improve.

Total swap = 0kB

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.