18

About once a week, but sometimes even a couple of times a day after running fine for days, my EC2 instances become unresponsive. Munin's memory graphs tell a pretty straightforward story: memory allocated to "apps" starts growing and doesn't stop until swap is fully used and the instance is effectively brought down to its knees. Another custom graph shows that the constantly growing process is apache2.

I run a standard prefork Apache setup with mod_php and a few PHP scripts. As you can see in the graph below, something happens that triggers apache2 processes to start consuming more and more memory. The first green spike I caught in time and restarted Apache before things got out of hand. The second spike got a bit farther and the instance had to be rebooted outright.

Munin Memory Graph

What I'm wondering is how to best debug this. Short of setting up PHP with FastCGI and getting it running in its own processes, what is a good way to find out whether it's Apache or a combination of PHP and my code that's causing the excessive memory usage? What steps would you guys take to track this issue down?


UPDATE: I was able to track the leak down after getting strace involved, as Matt suggested below.

After finding an apache2 process that was gradually and continuously growing in memory, I added a few more error_log() calls to my PHP script that printed out the total amount of RSS used at various points in its execution (using the output of ps). That however turned out to be misleading -- while it appeared that RSS jumped only after my script was done executing, later debugging revealed that wasn't indeed the case. Be careful!

Fortunately, all those error_log() calls turned out to be useful in the end. When I fired up strace (strace -p <pid> -tt -o trace.log -s 256), I saw that for each request, the process was allocating around 400k of memory (look for the 'brk' system call and subtract the first call's parameter from the last call's -- a few usually come in one after another). I then searched for the most recent 'write' system call that contained my error_log() message, which told me at what point in the script the memory was being allocated. With a few more strategically placed error_log() calls to pinpoint the location more accurately, I finally found the culprit.

The memory was leaking when we called curl_exec() from our PHP script. Some curl code related to handling an SSL connection is doing something wrong -- the leak went away when I switched to HTTP. Curl's changelog references a few SSL memory leaks that were fixed in 7.19.5 (we were on 7.18.2) so I'll try that next.

In the meantime, I'm running with a very low MaxRequestsPerChild that's keeping Apache within reasonable bounds. Thanks everyone!

3
  • How does the number of apache child processes vary over the same period?
    – SimonJ
    Nov 28, 2009 at 13:30
  • @SimonJ Simon, great question, the number stays pretty much the same, plus minus a few processes. It hovers around 60 when the servers are having trouble as well as when they're at rest. I'll set up a Munin graph to be 100% sure though.
    – ondrej
    Nov 28, 2009 at 21:34
  • Not a solution, but if one of the applications is known to eat RAM like crazy, then it is better to keep swap off: when kernel detects lack of RAM, it will kill the largest memory hogs (apache). With swap enabled, kernel will kill some processes much later, because swap is much slower than RAM. No swap - faster recovery, smaller downtime. (I have only tried disabling swap in a similar case on a machine with 8GiB RAM, so YMMW.)
    – chronos
    Nov 29, 2009 at 14:31

4 Answers 4

6

Tracking down WHAT is causing the problem can be a pain in the ass. The first thing I'd do if I had a problem like that is reduce MaxRequestsPerChild to an aggresively low number (~100-200) and see if that makes a difference. If it does, then you probably have code that is leaking memory in a loop somewhere and you'll want to run a code audit.

Another thing to look at is Apache's fullstatus, see if you can find out what particular request is causing the memory leak. Get the PIDs on your suspected processes and run an strace on them.

1
  • Thanks Matt. 'ps aux | grep apache2' tells me that out of the 60 or so processes active, around a dozen are using a lot more memory than they should (>100MB in RSS). I've looked at the output of /proc/<pid>/smaps and found that each one has exactly one anonymous mapping that's taking up 95%+ of the space. I'm now trying to figure out what and when allocated this huge chunk of memory. I'll look into strace -- thanks for the tip.
    – ondrej
    Nov 28, 2009 at 21:48
2

Friday @ exactly 11pm? Does that correspond to a backup time? Does your system have the I/O available to serve processes and backups at that time? Does you trending software also trend # procs or even apache scoreboard, how about disk I/O?

The first thing I would do would be to calculate how much mem each proc takes, then set a reasonable limit for MaxRequests in apache so that $procmem * $procs cannot exceed available ram. I suspect your instance needs to be rebooted because OOM kicks off a witch hunt that is likely (often) not very fruitful. You need to ensure your box can handle these heavy times by staying within its bounds and not go to swap and certainly not OOM. This is harder if you have cronjobs going, and extremely difficult if said cronjobs unilatterally run without making sure it's safe to run (i.e. the every 5 minute script fails to check if the last 5min one is still running).

Now that you've ensured that even if things go wildy wrong, you won't need to reboot your box, things will start going a lot better for you. You'll be able to login during these heavy times and get a good idea of what's going on using top, dstat, free -m, iostat, etc.

Matt's method may be worth trying, but should only be used as a tool for troubleshooting, I do not recommend keeping it that way because it will make the overall problem much harder to find the next time you're looking for it. That said, it will only really tease out issues with apache/modules and not anything in your code. I think you'll agree the chances are good it's not some sort of memory leak in apache module (assuming you're using a reputable distro).

0

The first question to ask is what is the application running through Apache?

Is it one you wrote, or a third-party app?

What other components / packages does it reference?

Are you up-to-date on your packages?

Anything specific in your httpd.conf files related to performance?

0

If your problem is caused by the PHP application and if you wrote the software yourself then I recommend you to use a profiler like e.g. PHP Quick Profiler. If you have a lot of database transactions going on then a software like e.g. Kontrollbase could help you finding the problem there.

1
  • Raffael, thanks. Yes, the PHP app is mine and it doesn't hit any SQL databases. I'll give PHP Quick Profiler a shot and report back.
    – ondrej
    Nov 28, 2009 at 21:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .