2

One of my server (Ubuntu 12.04, 16CPU, 32GB RAM) runs the app section of my website (Nginx 1.1.19, PHP5-FPM, Php 5.3.10, Symfony2 Web Framework)

Suddenly, users started getting HTTP 5** Errors. When I went on the server, there was huge disk IO and the culprit was [flush].

When I went through the kernel logs using dmesg, it showed the reason for [flush] was php5-fpm invoked oom-killer

Kernel Log entry from the time of Incident:

Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295736] php5-fpm invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295740] php5-fpm cpuset=/ mems_allowed=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295743] Pid: 13310, comm: php5-fpm Not tainted 3.2.0-69-virtual #103-Ubuntu
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295744] Call Trace:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295751]  [<ffffffff8111b971>] dump_header+0x91/0xe0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295754]  [<ffffffff8111bd25>] oom_kill_process+0x85/0xb0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295757]  [<ffffffff8111c0ca>] out_of_memory+0xfa/0x220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295761]  [<ffffffff81121aa3>] __alloc_pages_nodemask+0x8c3/0x8e0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295764]  [<ffffffff8115a853>] alloc_pages_vma+0xb3/0x190
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295768]  [<ffffffff8113ae9c>] do_anonymous_page.isra.38+0x7c/0x2f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295772]  [<ffffffff8113e921>] handle_pte_fault+0x1e1/0x200
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295777]  [<ffffffff8165d9be>] ? _raw_spin_lock+0xe/0x20
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295780]  [<ffffffff8113c089>] ? __pte_alloc+0xa9/0x160
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295782]  [<ffffffff8113fb19>] handle_mm_fault+0x269/0x370
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295785]  [<ffffffff81661504>] do_page_fault+0x184/0x550
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295789]  [<ffffffff81013728>] ? __switch_to+0x138/0x360
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295793]  [<ffffffff8105668d>] ? set_next_entity+0xad/0xd0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295796]  [<ffffffff8105682a>] ? finish_task_switch+0x4a/0xf0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295799]  [<ffffffff8165b6ec>] ? __schedule+0x3cc/0x6f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295802]  [<ffffffff8165e135>] page_fault+0x25/0x30
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295804] Mem-Info:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295805] Node 0 DMA per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295808] CPU    0: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295809] CPU    1: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295810] CPU    2: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295812] CPU    3: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295813] CPU    4: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295814] CPU    5: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295816] CPU    6: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295817] CPU    7: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295819] CPU    8: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295820] CPU    9: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295822] CPU   10: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295823] CPU   11: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295824] CPU   12: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295826] CPU   13: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295827] CPU   14: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295828] CPU   15: hi:    0, btch:   1 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295830] Node 0 DMA32 per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295831] CPU    0: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295833] CPU    1: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295834] CPU    2: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295836] CPU    3: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295837] CPU    4: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295838] CPU    5: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295840] CPU    6: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295841] CPU    7: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295842] CPU    8: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295844] CPU    9: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295845] CPU   10: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295846] CPU   11: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295848] CPU   12: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295850] CPU   13: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295851] CPU   14: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295853] CPU   15: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295854] Node 0 Normal per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295856] CPU    0: hi:  186, btch:  31 usd:  15
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295857] CPU    1: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295859] CPU    2: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295860] CPU    3: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295861] CPU    4: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295863] CPU    5: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295864] CPU    6: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295865] CPU    7: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295867] CPU    8: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295868] CPU    9: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295870] CPU   10: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295871] CPU   11: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295872] CPU   12: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295874] CPU   13: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295875] CPU   14: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295877] CPU   15: hi:  186, btch:  31 usd:   0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295880] active_anon:7408050 inactive_anon:436 isolated_anon:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295881]  active_file:21159 inactive_file:33077 isolated_file:419
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882]  unevictable:0 dirty:36283 writeback:0 unstable:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882]  free:47243 slab_reclaimable:20065 slab_unreclaimable:15299
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295883]  mapped:15592 shmem:138 pagetables:28362 bounce:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295885] Node 0 DMA free:15908kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295892] lowmem_reserve[]: 0 3760 30220 30220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295895] Node 0 DMA32 free:114216kB min:8404kB low:10504kB high:12604kB active_anon:3651120kB inactive_anon:284kB active_file:7848kB inactive_file:38464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3850496kB mlocked:0kB dirty:38664kB writeback:0kB mapped:272kB shmem:0kB slab_reclaimable:8444kB slab_unreclaimable:7164kB kernel_stack:208kB pagetables:10056kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116761 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295902] lowmem_reserve[]: 0 0 26460 26460
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295905] Node 0 Normal free:58848kB min:59144kB low:73928kB high:88716kB active_anon:25981080kB inactive_anon:1460kB active_file:76788kB inactive_file:93844kB unevictable:0kB isolated(anon):0kB isolated(file):1676kB present:27095040kB mlocked:0kB dirty:106468kB writeback:0kB mapped:62096kB shmem:552kB slab_reclaimable:71816kB slab_unreclaimable:54032kB kernel_stack:2352kB pagetables:103392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:388409 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295912] lowmem_reserve[]: 0 0 0 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295915] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15908kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295922] Node 0 DMA32: 268*4kB 544*8kB 367*16kB 391*32kB 271*64kB 165*128kB 103*256kB 32*512kB 5*1024kB 0*2048kB 1*4096kB = 114240kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295929] Node 0 Normal: 13697*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 58884kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295937] 54555 total pagecache pages
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295938] 0 pages in swap cache
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295939] Swap cache stats: add 0, delete 0, find 0/0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295940] Free swap  = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295941] Total swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359778] 7864304 pages RAM
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359780] 144336 pages reserved
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359781] 654931 pages shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359782] 7618451 pages non-shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359783] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359798] [  445]     0   445     4800      221   4       0             0 rpcbind
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359801] [  452]   108   452     5376      279   2       0             0 rpc.statd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359804] [  482]     0   482     4308       94   0       0             0 upstart-udev-br
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359807] [  494]     0   494     5431      327   1     -17         -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359809] [  608]     0   608     5430      231   1     -17         -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359812] [  624]     0   624     5430      233   3     -17         -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359815] [  706]     0   706     3797       49   3       0             0 upstart-socket-
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359818] [  882]     0   882     1816      213   6       0             0 dhclient3
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359821] [  993]     0   993    12509      499   2     -17         -1000 sshd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359824] [ 1018]   101  1018    63430      480   4       0             0 rsyslogd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359827] [ 1040]   102  1040     5954      155  13       0             0 dbus-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359830] [ 1048]     0  1048     7445      153   9       0             0 rpc.idmapd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359832] [ 1151]     0  1151     3626      190   3       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359835] [ 1157]     0  1157     3626      189   1       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359837] [ 1162]     0  1162     3626      189   5       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359840] [ 1163]     0  1163     3626      188   1       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359842] [ 1165]     0  1165     3626      188   6       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359845] [ 1170]     0  1170     4778      211   4       0             0 cron
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359847] [ 1171]     0  1171     4227       89  12       0             0 atd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359850] [ 1172]     0  1172     1082      149  10       0             0 acpid
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359852] [ 1178]     0  1178     3995      173   0       0             0 irqbalance
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359855] [ 1179]   103  1179    46916      650  10       0             0 whoopsie
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359858] [ 1195]   106  1195    80805      236   5       0             0 memcached
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359861] [ 1228]   109  1228     4199       99   0       0             0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359863] [ 1230]   109  1230    62851     1327   0       0             0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359866] [ 1340]     0  1340     4164      105   9       0             0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359869] [ 1341]     0  1341    61114     1948  13       0             0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359871] [ 1381]     0  1381     3310      100   6       0             0 mdadm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359874] [ 1444]     0  1444     3626      189   0       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359877] [ 1445]     0  1445     3188      177   3       0             0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359879] [ 1890]     0  1890   369259     2517   3       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359882] [ 1916]     0  1916    20056     1040   8       0             0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359885] [ 1917]    33  1917    20473     1994   8       0             0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359887] [ 1918]    33  1918    20399     1927   0       0             0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359890] [ 1919]    33  1919    20454     2020   7       0             0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359893] [ 1920]    33  1920    20201     1775   5       0             0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359896] [ 6544]    33  6544   624267    32944   6       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359898] [ 7293]    33  7293   618265    25932   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359901] [ 7381]    33  7381   618555    26851  12       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359904] [ 8280]    33  8280   618008    27712   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359906] [ 8737]    33  8737   619699    27793   6       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359909] [ 8908]    33  8908   618037    26037   6       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359912] [ 9093]    33  9093   618471    25848   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359914] [ 9479]    33  9479   688805    75927   6       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359917] [ 9503]    33  9503  5636713  4312476  15       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359919] [10392]    33 10392   621196    28967  13       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359922] [10509]    33 10509   618070    26222   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359925] [10510]    33 10510   622084    30147   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359927] [10540]    33 10540   620672    29171  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359930] [10799]    33 10799   670237    78421   4       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359932] [11066]    33 11066   616125    23836   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359935] [11250]    33 11250   619613    27729   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359937] [11254]    33 11254   619605    27515   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359940] [11426]    33 11426   620759    28353   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359943] [11428]    33 11428   618500    24942   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359945] [11429]    33 11429   832300   131639  12       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359948] [11444]    33 11444   390293    26564  12       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359951] [12119]    33 12119   389579    26295   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359953] [12542]    33 12542   390336    27483  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359956] [12577]    33 12577  1770706  1177669   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359958] [13310]    33 13310  1559195  1166076  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359961] [13481]    33 13481   729336   121410   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359963] [14074]    33 14074   613987    21729  15       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359966] [14686]    33 14686   405943    20594   6       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359968] [14771]    33 14771  1341333    17036  10       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359971] [14814]    33 14814   375034    16151   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359973] [14888]    33 14888   374759    15618   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359976] [14917]    33 14917   382071    16366  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359979] [14919]    33 14919   373976    14146   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359981] [14982]    33 14982   373012    11466   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359984] [14983]    33 14983   372940    10379   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359986] [14984]    33 14984   374522    12917  14       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359989] [14985]    33 14985   372898     9890   7       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359991] [14986]    33 14986   373733    11996  15       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359994] [14987]    33 14987   373203    12061   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359996] [14988]    33 14988   373368    12404  13       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359999] [14989]    33 14989   372407     8838   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360002] [14990]    33 14990   373491    11722   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360004] [14991]    33 14991   371987     8545   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360007] [14992]    33 14992   372184     9768  12       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360010] [14993]    33 14993   375221    12528   6       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360012] [15011]    33 15011   372181     8558  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360015] [15012]    33 15012   374863    13426   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360018] [15013]    33 15013   372497    10278   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360020] [15014]    33 15014   372489     9905   8       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360023] [15015]    33 15015   373200    11024   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360026] [15017]    33 15017   372621     9203   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360028] [15018]    33 15018   371993     9152  12       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360031] [15019]    33 15019   374082    10949   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360033] [15020]    33 15020   373137    10112  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360036] [15021]    33 15021   377055    14606   0       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360038] [15027]    33 15027   372243     9148   9       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360041] [15028]    33 15028   373802    11787  12       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360044] [15029]    33 15029   372094     7847  11       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360046] [15030]    33 15030   372093     7475   2       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360049] [15031]    33 15031   372606     9650  15       0             0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360071] Out of memory: Kill process 9503 (php5-fpm) score 564 or sacrifice child
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.363179] Killed process 9503 (php5-fpm) total-vm:22546852kB, anon-rss:17203684kB, file-rss:46220kB

Seems like the issue is with php5-fpm overcommiting the memory for some cases.

How do I ensure that such cases, doesn't create HIGH DISK IO / Server Unstable?

0

You need to make sure that you don't run too many PHP-FPM processes on your server. Look at pm.max_children, pm.min_spare_servers and pm.max_spare_servers settings in your PHP-FPM pool configuration.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.