3

Since this morning one of our webserver's memory getting filled up by php which eventually going to cause the server to start swapping, slow down and become unavailable (an ssh login takes like 5 minutes then sometimes it dies in completely)...

I wonder how would you go on with solving these type of issues.

Jul 23 06:16:31 websrv2 kernel: [10409040.981882] php-cgi invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Jul 23 06:16:31 websrv2 kernel: [10409040.981888] php-cgi cpuset=/ mems_allowed=0
Jul 23 06:16:31 websrv2 kernel: [10409040.981892] Pid: 27697, comm: php-cgi Not tainted 2.6.32-5-amd64 #1
Jul 23 06:16:31 websrv2 kernel: [10409040.981894] Call Trace:
Jul 23 06:16:31 websrv2 kernel: [10409040.981903]  [<ffffffff810b6714>] ? oom_kill_process+0x7f/0x23f
Jul 23 06:16:31 websrv2 kernel: [10409040.981908]  [<ffffffff8106bdee>] ? timekeeping_get_ns+0xe/0x2e
Jul 23 06:16:31 websrv2 kernel: [10409040.981912]  [<ffffffff810b6c38>] ? __out_of_memory+0x12a/0x141
Jul 23 06:16:31 websrv2 kernel: [10409040.981916]  [<ffffffff810b6d8f>] ? out_of_memory+0x140/0x172
Jul 23 06:16:31 websrv2 kernel: [10409040.981921]  [<ffffffff810baaf4>] ? __alloc_pages_nodemask+0x4ec/0x5fc
Jul 23 06:16:31 websrv2 kernel: [10409040.981927]  [<ffffffff810d92d8>] ? read_swap_cache_async+0x5d/0xf3
Jul 23 06:16:31 websrv2 kernel: [10409040.981931]  [<ffffffff810d93c5>] ? swapin_readahead+0x57/0x98
Jul 23 06:16:31 websrv2 kernel: [10409040.981937]  [<ffffffff8100c18d>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
Jul 23 06:16:31 websrv2 kernel: [10409040.981941]  [<ffffffff810cd245>] ? handle_mm_fault+0x47f/0x80f
Jul 23 06:16:31 websrv2 kernel: [10409040.981947]  [<ffffffff813001a6>] ? do_page_fault+0x2e0/0x2fc
Jul 23 06:16:31 websrv2 kernel: [10409040.981952]  [<ffffffff812fe045>] ? page_fault+0x25/0x30
Jul 23 06:16:31 websrv2 kernel: [10409040.981954] Mem-Info:

I try to go the way to sort the processes by memory usage:

ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS

I see bunch of /usr/bin/php-cgi consuming memory. How can I limit the overall memory consumption by PHP so the server never gets pushed to the limit to use up all memory/swap and become unavailable?

More system info:

ii  php5                                5.3.3-7+squeeze18            server-side, HTML-embedded scripting language (metapackage)
ii  apache2                             2.2.16-6+squeeze11           Apache HTTP Server metapackage
  • 6
    You're much better off fixing the memory leak in the application than trying to limit PHP's memory usage at the OS level. – John Jul 23 '14 at 12:50
  • Not my code, of course I can let the client know, it's not sure when they will do anything about it. PHP's memory_limit only limit it to a single script. It's better to do a workaround than let the server die again. – posttek Jul 23 '14 at 13:04
  • And for using ulimit, you cannot cap memory usage at the user level, ulimit can do that but for a single process. Even with using per user limits in /etc/security/limits.conf, a user can use all memory by running multiple processes. – posttek Jul 23 '14 at 13:19
  • 1
    For all of these reasons and more, you should either fix the memory leak, force the client to fix the memory leak, or refuse to host the application until the memory leak is fixed. Seriously. Fix the application. – John Jul 23 '14 at 13:22
  • How many parallel PHP processes is apache allowed to spawn? What is the average RSS size of a PHP worker? – Felix Frank Jul 23 '14 at 13:39
1

you can limit the memory usage of a user (the user which runs the php-cgi processes) with cgroups.

for example to limit the memory usage of a user you can do the following:

# file: /etc/cgconfig.conf
group php {
    memory {
        memory.limit_in_bytes = "2G";
    }
}

And then you have to add your php-cgi user to this cgroup-group:

# file: /etc/cgrules.conf
phpcgiuser   memory   php

If you want to limit the memory usage per process you can do this with ulimit (100 MB per process).

# file: /etc/security/limits.conf
phpcgiuser  hard  as   102400

Edit: Nevertheless, you should analyze the software what it causing the memory leak.

| improve this answer | |
0

If this is the result of regular use, you should check those PHP scripts for memory leak, infinite loops, and such scary issues.

| improve this answer | |
0

What OS you are on at all? What I'm doing on all my webservers with all the web applications I serve, is:

  • I create seperate pools for each individual web applications.
  • I limit the amount of memory they can absorb in php.ini (memory_limit = 2048M )
  • I've modified all the pools so that NginX user, the one running the web server is actually running the PHP pool as well.

This way, I've never experienced that you now, because every time I had an application, trying to overflow the RAM with itself simply get's kicked out while not having any effect on the rest of the web applications.

However I also strongly recommend finding the culprit of this massive overflow, it could be an attack or a badly written code (I assume the second one).

But what I recommend the most is UPGRADE YOUR PHP VERSION !!!! PHP 5.5 is already dead, no support coming for it. If it is impossible to upgrade to 7.1, then at least upgrade to 5.6.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.