About once a week, but sometimes even a couple of times a day after running fine for days, my EC2 instances become unresponsive. Munin's memory graphs tell a pretty straightforward story: memory allocated to "apps" starts growing and doesn't stop until swap is fully used and the instance is effectively brought down to its knees. Another custom graph shows that the constantly growing process is apache2.
I run a standard prefork Apache setup with mod_php and a few PHP scripts. As you can see in the graph below, something happens that triggers apache2 processes to start consuming more and more memory. The first green spike I caught in time and restarted Apache before things got out of hand. The second spike got a bit farther and the instance had to be rebooted outright.
What I'm wondering is how to best debug this. Short of setting up PHP with FastCGI and getting it running in its own processes, what is a good way to find out whether it's Apache or a combination of PHP and my code that's causing the excessive memory usage? What steps would you guys take to track this issue down?
UPDATE: I was able to track the leak down after getting strace involved, as Matt suggested below.
After finding an apache2 process that was gradually and continuously growing in memory, I added a few more error_log() calls to my PHP script that printed out the total amount of RSS used at various points in its execution (using the output of ps). That however turned out to be misleading -- while it appeared that RSS jumped only after my script was done executing, later debugging revealed that wasn't indeed the case. Be careful!
Fortunately, all those error_log() calls turned out to be useful in the end. When I fired up strace (strace -p <pid> -tt -o trace.log -s 256
), I saw that for each request, the process was allocating around 400k of memory (look for the 'brk' system call and subtract the first call's parameter from the last call's -- a few usually come in one after another). I then searched for the most recent 'write' system call that contained my error_log() message, which told me at what point in the script the memory was being allocated. With a few more strategically placed error_log() calls to pinpoint the location more accurately, I finally found the culprit.
The memory was leaking when we called curl_exec() from our PHP script. Some curl code related to handling an SSL connection is doing something wrong -- the leak went away when I switched to HTTP. Curl's changelog references a few SSL memory leaks that were fixed in 7.19.5 (we were on 7.18.2) so I'll try that next.
In the meantime, I'm running with a very low MaxRequestsPerChild that's keeping Apache within reasonable bounds. Thanks everyone!