3

I'm tuning up a Apache 2.2 server (it's shared with MySQL + Courier + Mongo + Postgre). The server is bare metal (no virtualizataion) with 2 quad-core Xeon (total 16 core) and 12GB ECC RAM (upgrading to 24GB in the next days). MySql has innodb_buffer set to 2G and is doing ok. Mongo and Postgre are barely touched - just for development.

The problem is that Apache is eating all memory available, eventually using all the swap (and therefore crashing MySql as it's using more memory). We have Apache 2.2 with worker MPM and mod_security. Whenever we restart it, we have aprox 8G of free RAM.

I'm trying to understand a few things:

  • What's the relation between the processes shown in top with the worker servers? I was expecting to see at most 10 processes (as the ServerLimit in worker settings - not taking in account the main process).
  • Is the amount of RES memory used for each THREAD or for each PROCESS?
  • In the mod_status output below, there are lot of READING requests that stays there for a long time. I've tried enabling ExtendedStatus to discover what clients are there but I couldn't find any useful info - any suggestions?
  • with the actual memory usage, what amount of RAM would I need to serve 400 concurrent connections? (as the mod status shows 250 already depleted)
  • what about raising ThreadsPerChild instead of raising MaxClients/ServerLimit?
<IfModule worker.c>
StartServers         2
MaxClients         250
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
ServerLimit 10
</IfModule>

TOP shows (only for apache):

top - 16:30:21 up 46 days, 23:12,  2 users,  load average: 0.94, 0.97, 1.31
Tasks: 460 total,   1 running, 459 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.8%us,  0.5%sy,  0.0%ni, 96.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12187448k total, 10686748k used,  1500700k free,    67104k buffers
Swap:  1048568k total,   275904k used,   772664k free,  2371208k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
24252 wwwrun    20   0 2216m 161m 7484 S 20.2  1.4   0:09.25 httpd
24750 wwwrun    20   0 1264m  51m 6196 S  9.9  0.4   0:00.93 httpd
23996 wwwrun    20   0 2136m 279m 7292 S  6.0  2.4   0:16.76 httpd
23326 wwwrun    20   0 2216m 276m 7376 S  2.7  2.3   0:16.81 httpd
23582 wwwrun    20   0 2208m 293m 7132 S  2.3  2.5   0:27.30 httpd
23688 wwwrun    20   0 1904m 208m 7304 S  1.0  1.8   0:13.62 httpd
19507 wwwrun    20   0 2182m 229m 8096 S  0.7  1.9   0:18.65 httpd
23616 wwwrun    20   0 1908m 239m 7092 S  0.7  2.0   0:18.48 httpd
18304 wwwrun    20   0 2160m 334m  11m S  0.3  2.8   0:47.38 httpd
23637 wwwrun    20   0 1830m 231m 7556 S  0.3  1.9   0:20.56 httpd
24457 wwwrun    20   0 2196m 174m 6568 S  0.3  1.5   0:04.11 httpd
15694 wwwrun    20   0 2199m 295m 7504 S  0.0  2.5   0:20.54 httpd
15783 wwwrun    20   0 2172m 244m  10m S  0.0  2.1   0:17.45 httpd
16577 wwwrun    20   0 2128m 354m 7436 S  0.0  3.0   0:28.21 httpd
18290 wwwrun    20   0  277m 9880  964 S  0.0  0.1   0:00.05 httpd
18379 wwwrun    20   0 2208m 211m 6864 S  0.0  1.8   0:08.57 httpd
18480 wwwrun    20   0 2096m 305m 7540 S  0.0  2.6   0:17.72 httpd
18791 wwwrun    20   0 1920m 251m 7244 S  0.0  2.1   0:20.24 httpd
19348 wwwrun    20   0 2060m 310m 7388 S  0.0  2.6   0:22.07 httpd
19619 wwwrun    20   0 2206m 235m 7340 S  0.0  2.0   0:15.30 httpd
19999 wwwrun    20   0 2178m 144m 7132 S  0.0  1.2   0:05.50 httpd
20697 wwwrun    20   0 2002m 193m 7276 S  0.0  1.6   0:08.12 httpd
20838 wwwrun    20   0 1890m 127m 7260 S  0.0  1.1   0:07.44 httpd
21407 wwwrun    20   0 1988m 290m 7708 S  0.0  2.4   0:39.33 httpd
22252 wwwrun    20   0 2216m 295m 7732 S  0.0  2.5   0:27.04 httpd
22723 wwwrun    20   0 2020m 187m 7444 S  0.0  1.6   0:39.80 httpd
22753 wwwrun    20   0 1850m 206m 7448 S  0.0  1.7   0:10.62 httpd
23174 wwwrun    20   0 1990m 229m 7332 S  0.0  1.9   0:07.71 httpd
23383 wwwrun    20   0 1504m  76m 6372 S  0.0  0.6   0:01.25 httpd
23720 wwwrun    20   0 1906m 225m 7080 S  0.0  1.9   0:20.12 httpd
24778 wwwrun    20   0  970m  30m 5604 S  0.0  0.3   0:00.80 httpd

Finally, mod_status output:

   Server Version: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips Domain Server PHP/5.3.24 with Suhosin-Patch mod_perl/2.0.5
          Perl/v5.10.1

   Server Built: May 15 2013 10:17:30
     ______________________________________________________________________________________________________________________________

   Current Time: Monday, 12-May-2014 16:41:08 BRT
   Restart Time: Monday, 12-May-2014 15:36:37 BRT
   Parent Server Generation: 1
   Server uptime: 1 hour 4 minutes 31 seconds
   Total accesses: 36446 - Total Traffic: 1.6 GB
   CPU Usage: u213.3 s13.65 cu0 cs0 - 5.86% CPU load
   9.42 requests/sec - 442.8 kB/second - 47.0 kB/request
   249 requests currently being processed, 0 idle workers

WWRRKRKRRKRRRRWRRWWWRWCRR.......................................
RWWWRRWRRRRKKRKRWRRRRRWWR.......................................
WRWRCRRWRRWWRKRRWWRRWRWWW.......................................
WRWKKWWWRWKRRWRCRRRWWRWWW.......................................
WRWRWCRKCWWWRWWRRWWCRWRWW.......................................
WRWKRWRRKKRKRRRRCWWKWWWRW.......................................
RWWKWRRWKRRRWRRWWCRRRRRRW.......................................
WWWRRWRKWWWKRWWWWRWWWKRRW.......................................
RRRRRRRK.WKRRWRWWKRRWRRKR.......................................
KCRRRRWRWWKRWWWRRKWRWRWRK.......................................

Update 1:

I've tried disabling mod_security2, fiddled with worker settings, but eventually Apache would consume almost 20G of RAM (I've added a 13G swapfile to prevent the server crashing).

Finally I disable KeepAlive, and kept MaxRequestsPerChild to 100 - now Apache is using 2-3G, and the requests are being cleaned, as status output below:

   Current Time: Tuesday, 13-May-2014 17:07:48 BRT
   Restart Time: Tuesday, 13-May-2014 16:49:14 BRT
   Parent Server Generation: 0
   Server uptime: 18 minutes 33 seconds
   Total accesses: 6637 - Total Traffic: 133.2 MB
   CPU Usage: u34.43 s3.05 cu0 cs0 - 3.37% CPU load
   5.96 requests/sec - 122.5 kB/second - 20.5 kB/request
   14 requests currently being processed, 86 idle workers

..................R.............................................
________________RR__R_R__RC_RWC___________________..............
R_WC__R___R_______________________________________..............
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

2 Answers 2

2

KeepaliveTimeout would help reduce the hanging "read" workers.

MPM uses multiple processes with multiple threads to handle incoming traffic, so what you're seeing is normal Apache operation.

This default tuning may work better for your needs:

ServerLimit 12
StartServers 2
MaxClients 500
MinSpareThreads 25
MaxSpareThreads 50
ThreadsPerChild 50

This would cause a total of 10 processes to spawn (MaxClients / ThreadsPerChild) which would significantly reduce the memory footprint of your server. You can afford more threads across less processes because of CPU availability.

Note that changing these variables requires Apache to be fully stopped then restarted.

2
  • Thanks, I'll try your suggestion and take a look at KeepaliveTimeout too. I still want to know the relation between the apache "servers" (ServerLimit) and the processes shown in top -- I mean, there were 20+ apache processes in top but it was configured to spawn 10 servers max. May 13, 2014 at 12:37
  • I see you're using mod_perl which may be responsible. The status page shows 10 processes, so all the others are likely different parts of Apache and not handling requests directly.
    – Nathan C
    May 13, 2014 at 12:48
2

What's the relation between the processes shown in top with the worker servers? I was expecting to see at most 10 processes (as the ServerLimit in worker settings - not taking in account the main process).

That doesn't look like the top of an Apache server running the worker MPM. I'm running two servers, one with the worker MPM and another with the prefork MPM. With worker, the top lists the Apache processes as httpd.worker, and the number of processes match the number of active servers (lines of dots with some non-dot entries) visible via mod_status. With prefork, the processes are listed as httpd, and the number roughly matches the number of idle/active workers (each non-dot entry).

So given this mod_status output:

..................R.............................................
..............._RR__R_R__RC_RWC__...............................
R_WC__R___R_....................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

top on a server running prefork should show roughly 31 httpd processes, and top on a server running worker should show 3 httpd.worker processes.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .