Apache strace to hunt down a memory leak

Question

We have a server with a memory issue: the server keeps allocating itself memory and doesn't release it. We're running Apache. I set MaxReqsPerClient to a really low value just so the threads don't hold a lot of memory, but has anyone seen calls like this? Am I wrong in thinking that it's probably Drupal pulling too much data back from the cache in DB?

read(52, "h_index\";a:2:{s:6:\"weight\";i:1;s"..., 6171) = 1368
read(52, "\";a:2:{s:6:\"author\";a:3:{s:5:\"la"..., 4803) = 1368
read(52, ":\"description\";s:19:\"Term name t"..., 3435) = 1368
read(52, "abel\";s:4:\"Name\";s:11:\"descripti"..., 2067) = 1368
read(52, "ions\";a:2:{s:4:\"form\";a:3:{s:4:\""..., 16384) = 708
brk(0x2ab554396000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f653000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f753000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f853000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55f953000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55fa53000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55fb53000
brk(0x2ab554356000)                     = 0x2ab5542f5000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ab55fc53000
poll([{fd=52, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(52, "d\0\0\0\3SELECT cid, data, created, "..., 104) = 104
read(52, "\1\0\0\1\5E\0\0\2\3def\23drupal_database_nam"..., 16384) = 1368
read(52, ";s:11:\"granularity\";a:5:{s:4:\"ye"..., 34783) = 1368
read(52, ":4:\"date\";}s:9:\"datestamp\";a:9:{"..., 33415) = 1368
read(52, "\";i:0;s:15:\"display_default\";i:0"..., 32047) = 1368
read(52, "e as an integer value.\";s:8:\"set"..., 30679) = 1368
read(52, "label' pairs, i.e. 'Fraction': 0"..., 29311) = 1368

top (the procs just keep growing in memory..):

12845 apache    15   0  581m 246m  37m S  0.0  4.1   0:17.39 httpd                                   
12846 apache    15   0  571m 235m  37m S  0.0  4.0   0:12.13 httpd                                   
12833 apache    15   0  420m 117m  37m S  0.0  2.0   0:06.04 httpd                                   
12851 apache    15   0  412m 113m  37m S  0.0  1.9   0:05.32 httpd                                   
13871 apache    15   0  409m 109m  37m S  0.0  1.8   0:04.90 httpd                                   
12844 apache    15   0  407m 108m  37m S  0.0  1.8   0:04.50 httpd                                   
13870 apache    15   0  407m 108m  37m S  0.3  1.8   0:03.50 httpd                                   
14903 apache    15   0  402m 103m  37m S  0.3  1.7   0:01.29 httpd                                   
14850 apache    15   0  397m 100m  37m S  0.0  1.7   0:02.08 httpd                                   
14907 apache    15   0  390m  93m  36m S  0.0  1.6   0:01.32 httpd                                   
13872 apache    15   0  386m  91m  37m S  0.0  1.5   0:03.13 httpd                                   
12843 apache    15   0  373m  81m  37m S  0.0  1.4   0:02.51 httpd                                   
14901 apache    15   0  370m  75m  33m S  0.0  1.3   0:00.78 httpd                                   
14904 apache    15   0  335m  29m  15m S  0.0  0.5   0:00.26 httpd  

Answer 1

Over what time period is this occurring? How big does memory usage get?

If it's a short-enough time, or if you don't running Apache in the foreground for a bit, you could use valgrind to find memory leaks... (It's available via YUM)

Something like:

valgrind --leak-check=full /usr/sbin/httpd 

The output could potentially show something like:

==945== LEAK SUMMARY:
==945== definitely lost: 15,595 bytes in 120 blocks
==945== indirectly lost: 1,917,248 bytes in 1,242 blocks
==945== possibly lost: 0 bytes in 0 blocks
==945== still reachable: 11,743 bytes in 89 blocks
==945== suppressed: 0 bytes in 0 blocks
==945== Reachable blocks (those to which a pointer was found) are not shown.
==945== To see them, rerun with: --leak-check=full --show-reachable=yes

You can view the memory allocation of a process with the pmap utility.

For Apache, I'd take a look at the PID. Obtain the PID value with: cat /var/run/httpd.pid or just run:

pmap -x $(cat /var/run/httpd.pid)

But in my limited Apache experience, leaks always seem to be tied to modules and scripting languages. Could be caching, too...

Answer 2

I ended up just lowering MaxRequestsPerChild in httpd.conf so my threads expire before they get too big. They were previously getting up to 800M and there was some serious swapping, and now they get to at most 250M before they die.