Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a virtual Ubuntu 10.04 with Apache2 server (apache2-mpm-prefork 2.2.14-5ubuntu8.10), running some PHP and MySQL sites. Its load is not high (Usually 0.2). However, it's memory usage is hugely fluctuating, and I can't find out what it is. Free memory goes from about 1 GB to only a few dozen. I'd like to know how to track it down...

Especially every night, it climbs from about 500 MB used to more than 1 GB used (of 1.2 total) over a period of several hours, and then it drops again. Over that period, the munin graph of apache access shows no traffic of importance (the nightly dip), no CPU, no load. And even during the day, apache accesses are about 1 access per second.

Using apache2ctl fullstatus and top, I can see that the processes that start increasing have a varying task. A lot of them are just NULL, some of them handle the /server-status Munin causes, some just serve an image. And what they're doing keeps changing, so it's not one task that is running away.

I suspected a memory leak, so I tried setting MaxRequestsPerChild from thousands to 200, and it doesn't help at all. Reducing the amount of spare servers reduced the total memory requirement, but the issue remains.

Current memory is:

             total       used       free     shared    buffers     cached
Mem:          1246       1040        206          0        125        437
-/+ buffers/cache:        478        768
Swap:         4095         35       4060

But every night, and sometimes during the day, free - buffers/cache goes from 768 to about 50. When I use top, I can see it's all apache processes cumulatively, but I can't see what and why...

Currently, top says:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                          
16298 www-data  20   0  300m  69m 5904 S    0  5.6   0:01.54 apache2                                                                                                                                           
17290 www-data  20   0  300m  68m 4672 S    0  5.5   0:01.53 apache2                                                                                                                                           
16297 www-data  20   0  279m  49m 6332 S    0  4.0   0:01.11 apache2                                                                                                                                           
  536 mysql     20   0  266m  42m 3592 S    0  3.4  26:10.26 mysqld                                                                                                                                            
17216 www-data  20   0  267m  38m 6236 S    0  3.1   0:00.78 apache2                                                                                                                                           
17359 www-data  20   0  268m  36m 4580 S    0  3.0   0:00.87 apache2                                                                                                                                           
 1066 root      20   0  230m  13m 4592 S    0  1.1   3:14.15 apache2                                                                                                                                           
17445 www-data  20   0  230m  10m 1612 S    0  0.8   0:00.01 apache2                                                                                                                                           
17447 www-data  20   0  230m  10m 1192 S    0  0.8   0:00.01 apache2                                                                                                                                           
  564 root      20   0 12624 4596 4532 S    0  0.4   6:55.72 ipwatchd                                                                                                                                          
  589 root      20   0 44144 3528 1112 S    0  0.3   1:17.06 munin-node

So, how can I find out what is causing it? How can I see what apache is keeping in memory?

edit: I've written a script that logs the pmap output when per-process memory usage is more than 10%. All I can see is that there is an anonymous allocation of 150 MB per process. So how do I find out what it is?

I really don't understand that it's at night when this memory usage slowly starts increasing over the course of hours. It's when the server does almost nothing that this happens.

share|improve this question
To troubleshoot apache memory problem, you can use pmap and server-status – user130370 Jan 2 '13 at 14:01
Hmm. When I see a line like: "00007f468c30b000 103896K rw--- [ anon ]", how that that help me...? – Halfgaar Jan 2 '13 at 16:22
@Halfgaar: If you see its apache process then you should take the pid and dig into the proc/pid folder you might get some info. As which script is executing. I have seen most of the time it is your script which causes this thing. Yes it is hard to debug such issue but worth digging into /proc/pid folder for such solutions. Hope you might find this helpful. Also you should explore top option to show the command which is executing currently. – PratapSingh Jan 2 '13 at 16:32
"Something" takes 104M inside apache process (cf ). This is likely to be php. – user130370 Jan 2 '13 at 16:46
I added more info about pmap about. I really don't see any way to find out what's going on. – Halfgaar Jan 8 '13 at 8:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.