Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have Magento application deployed on Apache with mod_php and mysql. I have observed that sometime apache server starts consuming high memory which causes memory swapping and results in high load on servers. whenever there is high load on apache server, the apache processes which are causing the high load were in sleep mode at mysql end and CLOSE_WAIT state at client side. Any help is appreciated to resolve this issue.

share|improve this question
    
How high is your Load Average? It's a best practice to limit apache maxclients, so that server will not run out of memory and avoid swapping. –  DukeLion May 17 '12 at 9:54
1  
might also be worth to decrease MaxRequestsPerChild - but not too much. This will enable apache to kill child processes more often hence releasing memory it might be leaking. –  gyre May 17 '12 at 9:58
    
sometime LA jumps to 30-40. –  Raj May 17 '12 at 11:20
    
one thing I am not understanding, if client side connection state is CLOSE_WAIT still same apache process at the mysql end executes sql queries. I have straced such processes and found this weird behaviour. –  Raj May 17 '12 at 11:23

3 Answers 3

up vote 2 down vote accepted

@raj you'll probably find that the processes are in an S state as they are waiting on another process (in a D state) bound by I/O ie. For the machine to stop swapping.

So the key is to address two key issues, standard memory allocation and memory commitment.

When running Magento, it is fairly easy to massively over commit on RAM simply by setting PHP Max memory limits too high. Under normal circumstances, it won't have any ill effects, but the moment the app tries to use that memory, it will almost immediately go into swap (if you've run out of RAM that is).

Then, you'll likely find that Apache isn't the main concern for memory usage, but rather MySQL is more likely the culprit.

You haven't mentioned your server specification or any specific details for your store, so it would be impossible to accurately advise. But simply revising your Apache and MySQL configuration would be an easy place to start. Drop the number of Apache threads, MySQL connections, allocated memory commitments. Then start graphing with a tool like Munin to actually be able to correlate traffic, threads, CPU usage and memory usage. Without doing so, you are working blind.

But I would advise to stop poking around netstat as you are a long way off needing to diagnose TCP states; you need to start from the beginning.

share|improve this answer
    
@sonnassi Server details are as below. Two loadbalancers (HaProxy), 4 webservers with 8GB of Ram and one DB server with 8GB of Ram. all servers are hosted on Rackspace. Do u need any further information. –  Raj Jul 27 '12 at 11:33
    
@Raj I have near the same set up, 2GB webserver and 556MB DB server on RS (bare CentOS using the private IP for web to DB com) Have you sorted out why Magento is leaking? –  jeremy.bass Jan 10 '13 at 4:47

Go read some good books on the subject of HTTP tuning and performance management (really - you're unlikely to get the answer here).

which causes memory swapping

Then your server is badly configured - a webserver should never start swapping. Apply the usual apache tuning ritual and reduce php max memory.

whenever there is high load on apache server, the apache processes which are causing the high load were in sleep mode at mysql end

Eh? A sleeping process by definition does not contribute to load. What mhysql end?

share|improve this answer
    
Sleeping process do consume some memory. Lack of memory leads to swapping. Swapping results in high load. –  DukeLion May 17 '12 at 11:54
    
A sleeping process almost definitely contributes to load, it implies I/O wait , which can make load averages go through the roof. –  Ben Lessani - Sonassi Jun 20 '12 at 20:11
    
@sonassi: No. DukeLion is partly correct - sleeping processes consume memory meaning that less is available for other stuff - but with a remote MySQL DB, the processes are in iowait - but not waiting for local I/O operations. The swapping is causing the load - not the other way around. –  symcbean Jun 21 '12 at 8:55
    
@symcbean - hence why I said it implies I/O wait - not is responsible for (that would be the D state) –  Ben Lessani - Sonassi Jun 21 '12 at 11:13

Use this script: http://saiweb.co.uk/linux/apache/bash%20script/linux-the-sysadmin-script-part-3/

to determine how much memory your apache processes are using.

Then limit MaxClients in apache config, to amount of process you can run without swapping. In this case when the problem occurs you will not have high LA, and you can find the real source of problems. Here's my version of script (link above seems to be broken now)

#!/bin/bash
       if [ -z "$1" ]; then
                echo "Usage: sysadmin appmem app_name i.e. (sysadmin appmem apache)";
        else
                RRES=(`ps aux | grep "$1" | grep -v 'grep' | grep -v "$0" | awk '{print $6}'`);
                VRES=(`ps aux | grep "$1" | grep -v 'grep' | grep -v "$0" | awk '{print $5}'`);
                COUNT=0;
                VMEM=0;
                RMEM=0;
                for RSS in ${RRES[@]}
                do
                        RMEM=$(($RSS+$RMEM));
                done;
                for VIRT in ${VRES[@]}
                do
                        VMEM=$(($VIRT+$VMEM));
                        COUNT=$(($COUNT+1));
                done;
                VMEM=$(($VMEM/$COUNT));
                VMEM=$(($VMEM/1024));
                RMEM=$(($RMEM/1024));
                echo -e "$YELLOW ----- MEMORY USAGE REPORT FOR '$1' ----- $CLEAR";
                echo "PID Count: $COUNT";
                echo "Shared Mem usage: $VMEM MB";
                echo "Total Resident Set Size: $RMEM MB";
                echo "Mem/PID: $(($RMEM/$COUNT)) MB";
        fi
share|improve this answer
    
this will only tell you how much memory is being used by apache processes but won't tell you the actual cause. it can be a leak in pretty much any library compiled into apache...I'd advise something like Xdebug - bt be careful when using it! –  gyre May 17 '12 at 10:42
    
It is more possible that slow mysql request causing scripts to work for long time, pumping up concurrent apache processes. To many apache processes cause swapping and load average jumps. When this thing occurs -there is almost no chance to find root cause of slowdown. Limiting apache processes and avoiding swap is first thing to do. After that you need to find the actual bottleneck. –  DukeLion May 17 '12 at 10:51
    
one thing I am not understanding, if client side connection state is CLOSE_WAIT still same apache process at the mysql end executes sql queries. I have straced such processes and found this weird behaviour. –  Raj May 17 '12 at 11:22
1  
CLOSE_WAIT means that client has requested to terminate connection, but apache process is waiting for smth before it can close the socket. With high LA - it's quite common. –  DukeLion May 17 '12 at 11:29
    
The link in your answer appears to have rotted. Do you have one that works? –  Peter Feb 6 '13 at 4:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.