0

We have Magento application deployed on Apache with mod_php and mysql. I have observed that sometime apache server starts consuming high memory which causes memory swapping and results in high load on servers. whenever there is high load on apache server, the apache processes which are causing the high load were in sleep mode at mysql end and CLOSE_WAIT state at client side. Any help is appreciated to resolve this issue.

4
  • How high is your Load Average? It's a best practice to limit apache maxclients, so that server will not run out of memory and avoid swapping.
    – DukeLion
    May 17 '12 at 9:54
  • 1
    might also be worth to decrease MaxRequestsPerChild - but not too much. This will enable apache to kill child processes more often hence releasing memory it might be leaking. May 17 '12 at 9:58
  • sometime LA jumps to 30-40.
    – Raj
    May 17 '12 at 11:20
  • one thing I am not understanding, if client side connection state is CLOSE_WAIT still same apache process at the mysql end executes sql queries. I have straced such processes and found this weird behaviour.
    – Raj
    May 17 '12 at 11:23
2

@raj you'll probably find that the processes are in an S state as they are waiting on another process (in a D state) bound by I/O ie. For the machine to stop swapping.

So the key is to address two key issues, standard memory allocation and memory commitment.

When running Magento, it is fairly easy to massively over commit on RAM simply by setting PHP Max memory limits too high. Under normal circumstances, it won't have any ill effects, but the moment the app tries to use that memory, it will almost immediately go into swap (if you've run out of RAM that is).

Then, you'll likely find that Apache isn't the main concern for memory usage, but rather MySQL is more likely the culprit.

You haven't mentioned your server specification or any specific details for your store, so it would be impossible to accurately advise. But simply revising your Apache and MySQL configuration would be an easy place to start. Drop the number of Apache threads, MySQL connections, allocated memory commitments. Then start graphing with a tool like Munin to actually be able to correlate traffic, threads, CPU usage and memory usage. Without doing so, you are working blind.

But I would advise to stop poking around netstat as you are a long way off needing to diagnose TCP states; you need to start from the beginning.

2
  • @sonnassi Server details are as below. Two loadbalancers (HaProxy), 4 webservers with 8GB of Ram and one DB server with 8GB of Ram. all servers are hosted on Rackspace. Do u need any further information.
    – Raj
    Jul 27 '12 at 11:33
  • @Raj I have near the same set up, 2GB webserver and 556MB DB server on RS (bare CentOS using the private IP for web to DB com) Have you sorted out why Magento is leaking?
    – Quantum
    Jan 10 '13 at 4:47
1

Go read some good books on the subject of HTTP tuning and performance management (really - you're unlikely to get the answer here).

which causes memory swapping

Then your server is badly configured - a webserver should never start swapping. Apply the usual apache tuning ritual and reduce php max memory.

whenever there is high load on apache server, the apache processes which are causing the high load were in sleep mode at mysql end

Eh? A sleeping process by definition does not contribute to load. What mhysql end?

4
  • Sleeping process do consume some memory. Lack of memory leads to swapping. Swapping results in high load.
    – DukeLion
    May 17 '12 at 11:54
  • A sleeping process almost definitely contributes to load, it implies I/O wait , which can make load averages go through the roof. Jun 20 '12 at 20:11
  • @sonassi: No. DukeLion is partly correct - sleeping processes consume memory meaning that less is available for other stuff - but with a remote MySQL DB, the processes are in iowait - but not waiting for local I/O operations. The swapping is causing the load - not the other way around.
    – symcbean
    Jun 21 '12 at 8:55
  • @symcbean - hence why I said it implies I/O wait - not is responsible for (that would be the D state) Jun 21 '12 at 11:13
0

Use this script: http://saiweb.co.uk/linux/apache/bash%20script/linux-the-sysadmin-script-part-3/

to determine how much memory your apache processes are using.

Then limit MaxClients in apache config, to amount of process you can run without swapping. In this case when the problem occurs you will not have high LA, and you can find the real source of problems. Here's my version of script (link above seems to be broken now)

#!/bin/bash
       if [ -z "$1" ]; then
                echo "Usage: sysadmin appmem app_name i.e. (sysadmin appmem apache)";
        else
                RRES=(`ps aux | grep "$1" | grep -v 'grep' | grep -v "$0" | awk '{print $6}'`);
                VRES=(`ps aux | grep "$1" | grep -v 'grep' | grep -v "$0" | awk '{print $5}'`);
                COUNT=0;
                VMEM=0;
                RMEM=0;
                for RSS in ${RRES[@]}
                do
                        RMEM=$(($RSS+$RMEM));
                done;
                for VIRT in ${VRES[@]}
                do
                        VMEM=$(($VIRT+$VMEM));
                        COUNT=$(($COUNT+1));
                done;
                VMEM=$(($VMEM/$COUNT));
                VMEM=$(($VMEM/1024));
                RMEM=$(($RMEM/1024));
                echo -e "$YELLOW ----- MEMORY USAGE REPORT FOR '$1' ----- $CLEAR";
                echo "PID Count: $COUNT";
                echo "Shared Mem usage: $VMEM MB";
                echo "Total Resident Set Size: $RMEM MB";
                echo "Mem/PID: $(($RMEM/$COUNT)) MB";
        fi
6
  • this will only tell you how much memory is being used by apache processes but won't tell you the actual cause. it can be a leak in pretty much any library compiled into apache...I'd advise something like Xdebug - bt be careful when using it! May 17 '12 at 10:42
  • It is more possible that slow mysql request causing scripts to work for long time, pumping up concurrent apache processes. To many apache processes cause swapping and load average jumps. When this thing occurs -there is almost no chance to find root cause of slowdown. Limiting apache processes and avoiding swap is first thing to do. After that you need to find the actual bottleneck.
    – DukeLion
    May 17 '12 at 10:51
  • one thing I am not understanding, if client side connection state is CLOSE_WAIT still same apache process at the mysql end executes sql queries. I have straced such processes and found this weird behaviour.
    – Raj
    May 17 '12 at 11:22
  • 1
    CLOSE_WAIT means that client has requested to terminate connection, but apache process is waiting for smth before it can close the socket. With high LA - it's quite common.
    – DukeLion
    May 17 '12 at 11:29
  • The link in your answer appears to have rotted. Do you have one that works?
    – Peter
    Feb 6 '13 at 4:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.