I'm tuning up a Apache 2.2 server (it's shared with MySQL + Courier + Mongo + Postgre). The server is bare metal (no virtualizataion) with 2 quad-core Xeon (total 16 core) and 12GB ECC RAM (upgrading to 24GB in the next days). MySql has innodb_buffer set to 2G and is doing ok. Mongo and Postgre are barely touched - just for development.
The problem is that Apache is eating all memory available, eventually using all the swap (and therefore crashing MySql as it's using more memory). We have Apache 2.2 with worker MPM and mod_security. Whenever we restart it, we have aprox 8G of free RAM.
I'm trying to understand a few things:
- What's the relation between the processes shown in
top
with the worker servers? I was expecting to see at most 10 processes (as the ServerLimit in worker settings - not taking in account the main process). - Is the amount of RES memory used for each THREAD or for each PROCESS?
- In the mod_status output below, there are lot of READING requests that stays there for a long time. I've tried enabling ExtendedStatus to discover what clients are there but I couldn't find any useful info - any suggestions?
- with the actual memory usage, what amount of RAM would I need to serve 400 concurrent connections? (as the mod status shows 250 already depleted)
- what about raising ThreadsPerChild instead of raising MaxClients/ServerLimit?
<IfModule worker.c> StartServers 2 MaxClients 250 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 ServerLimit 10 </IfModule>
TOP shows (only for apache):
top - 16:30:21 up 46 days, 23:12, 2 users, load average: 0.94, 0.97, 1.31 Tasks: 460 total, 1 running, 459 sleeping, 0 stopped, 0 zombie Cpu(s): 2.8%us, 0.5%sy, 0.0%ni, 96.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 12187448k total, 10686748k used, 1500700k free, 67104k buffers Swap: 1048568k total, 275904k used, 772664k free, 2371208k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24252 wwwrun 20 0 2216m 161m 7484 S 20.2 1.4 0:09.25 httpd 24750 wwwrun 20 0 1264m 51m 6196 S 9.9 0.4 0:00.93 httpd 23996 wwwrun 20 0 2136m 279m 7292 S 6.0 2.4 0:16.76 httpd 23326 wwwrun 20 0 2216m 276m 7376 S 2.7 2.3 0:16.81 httpd 23582 wwwrun 20 0 2208m 293m 7132 S 2.3 2.5 0:27.30 httpd 23688 wwwrun 20 0 1904m 208m 7304 S 1.0 1.8 0:13.62 httpd 19507 wwwrun 20 0 2182m 229m 8096 S 0.7 1.9 0:18.65 httpd 23616 wwwrun 20 0 1908m 239m 7092 S 0.7 2.0 0:18.48 httpd 18304 wwwrun 20 0 2160m 334m 11m S 0.3 2.8 0:47.38 httpd 23637 wwwrun 20 0 1830m 231m 7556 S 0.3 1.9 0:20.56 httpd 24457 wwwrun 20 0 2196m 174m 6568 S 0.3 1.5 0:04.11 httpd 15694 wwwrun 20 0 2199m 295m 7504 S 0.0 2.5 0:20.54 httpd 15783 wwwrun 20 0 2172m 244m 10m S 0.0 2.1 0:17.45 httpd 16577 wwwrun 20 0 2128m 354m 7436 S 0.0 3.0 0:28.21 httpd 18290 wwwrun 20 0 277m 9880 964 S 0.0 0.1 0:00.05 httpd 18379 wwwrun 20 0 2208m 211m 6864 S 0.0 1.8 0:08.57 httpd 18480 wwwrun 20 0 2096m 305m 7540 S 0.0 2.6 0:17.72 httpd 18791 wwwrun 20 0 1920m 251m 7244 S 0.0 2.1 0:20.24 httpd 19348 wwwrun 20 0 2060m 310m 7388 S 0.0 2.6 0:22.07 httpd 19619 wwwrun 20 0 2206m 235m 7340 S 0.0 2.0 0:15.30 httpd 19999 wwwrun 20 0 2178m 144m 7132 S 0.0 1.2 0:05.50 httpd 20697 wwwrun 20 0 2002m 193m 7276 S 0.0 1.6 0:08.12 httpd 20838 wwwrun 20 0 1890m 127m 7260 S 0.0 1.1 0:07.44 httpd 21407 wwwrun 20 0 1988m 290m 7708 S 0.0 2.4 0:39.33 httpd 22252 wwwrun 20 0 2216m 295m 7732 S 0.0 2.5 0:27.04 httpd 22723 wwwrun 20 0 2020m 187m 7444 S 0.0 1.6 0:39.80 httpd 22753 wwwrun 20 0 1850m 206m 7448 S 0.0 1.7 0:10.62 httpd 23174 wwwrun 20 0 1990m 229m 7332 S 0.0 1.9 0:07.71 httpd 23383 wwwrun 20 0 1504m 76m 6372 S 0.0 0.6 0:01.25 httpd 23720 wwwrun 20 0 1906m 225m 7080 S 0.0 1.9 0:20.12 httpd 24778 wwwrun 20 0 970m 30m 5604 S 0.0 0.3 0:00.80 httpd
Finally, mod_status output:
Server Version: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips Domain Server PHP/5.3.24 with Suhosin-Patch mod_perl/2.0.5 Perl/v5.10.1 Server Built: May 15 2013 10:17:30 ______________________________________________________________________________________________________________________________ Current Time: Monday, 12-May-2014 16:41:08 BRT Restart Time: Monday, 12-May-2014 15:36:37 BRT Parent Server Generation: 1 Server uptime: 1 hour 4 minutes 31 seconds Total accesses: 36446 - Total Traffic: 1.6 GB CPU Usage: u213.3 s13.65 cu0 cs0 - 5.86% CPU load 9.42 requests/sec - 442.8 kB/second - 47.0 kB/request 249 requests currently being processed, 0 idle workers WWRRKRKRRKRRRRWRRWWWRWCRR....................................... RWWWRRWRRRRKKRKRWRRRRRWWR....................................... WRWRCRRWRRWWRKRRWWRRWRWWW....................................... WRWKKWWWRWKRRWRCRRRWWRWWW....................................... WRWRWCRKCWWWRWWRRWWCRWRWW....................................... WRWKRWRRKKRKRRRRCWWKWWWRW....................................... RWWKWRRWKRRRWRRWWCRRRRRRW....................................... WWWRRWRKWWWKRWWWWRWWWKRRW....................................... RRRRRRRK.WKRRWRWWKRRWRRKR....................................... KCRRRRWRWWKRWWWRRKWRWRWRK.......................................
Update 1:
I've tried disabling mod_security2, fiddled with worker settings, but eventually Apache would consume almost 20G of RAM (I've added a 13G swapfile to prevent the server crashing).
Finally I disable KeepAlive, and kept MaxRequestsPerChild to 100 - now Apache is using 2-3G, and the requests are being cleaned, as status output below:
Current Time: Tuesday, 13-May-2014 17:07:48 BRT Restart Time: Tuesday, 13-May-2014 16:49:14 BRT Parent Server Generation: 0 Server uptime: 18 minutes 33 seconds Total accesses: 6637 - Total Traffic: 133.2 MB CPU Usage: u34.43 s3.05 cu0 cs0 - 3.37% CPU load 5.96 requests/sec - 122.5 kB/second - 20.5 kB/request 14 requests currently being processed, 86 idle workers ..................R............................................. ________________RR__R_R__RC_RWC___________________.............. R_WC__R___R_______________________________________.............. ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................