2

I have a c1.medium tier server running Ubuntu 12.04 on AWS EC2. We use it as a development server, hosting between 5-8 Drupal 7 sites on it. Recently the server has been performing very slowly and I'm struggling to find why. I've searched logs, but can't detect anything obvious. I've notice whenever the server starts to crawl / crash - looking at the cloud watch graphs on the EC2 console, there is always an unusual spike. Any recommendations on how to investigate and identify these problems and where they're coming from.. I've attached the output of top and a snippet of my apache2.conf file.

Tasks: 117 total,   1 running, 113 sleeping,   3 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1733244k total,  1647596k used,    85648k free,     4612k buffers
Swap:   917500k total,   644984k used,   272516k free,    69512k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                
16417 mysql     20   0 1317m 199m 1784 S    0 11.8   3:19.84 mysqld                                                                                                                                                 
18087 www-data  20   0  483m 139m 3132 S    0  8.3   0:11.15 apache2                                                                                                                                                
18019 www-data  20   0  483m 136m 3208 S    0  8.1   0:13.52 apache2                                                                                                                                                
18088 www-data  20   0  477m 131m 2952 S    0  7.8   0:10.31 apache2                                                                                                                                                
17958 www-data  20   0  533m 130m 2948 S    0  7.7   0:16.77 apache2                                                                                                                                                
17959 www-data  20   0  450m 103m 3192 S    0  6.1   2:05.47 apache2                                                                                                                                                
17987 www-data  20   0  443m  97m 3188 S    0  5.8   0:10.76 apache2                                                                                                                                                
17961 www-data  20   0  438m  95m 3180 S    0  5.7   0:13.21 apache2                                                                                                                                                
18089 www-data  20   0  443m  83m 3172 S    0  4.9   0:06.69 apache2                                                                                                                                                
18068 www-data  20   0  412m  68m 3148 S    0  4.0   0:07.99 apache2                                                                                                                                                
18459 www-data  20   0  411m  68m 3204 S    0  4.0   0:01.27 apache2                                                                                                                                                
17960 www-data  20   0  410m  67m 3812 S    0  4.0   0:06.79 apache2                                                                                                                                                
18283 ubuntu    20   0 26720 6908    0 S    0  0.4   0:00.59 bash                                                                                                                                                   
18460 www-data  20   0  350m 2788  804 S    0  0.2   0:00.00 apache2                                                                                                                                                
13391 root      20   0  350m 1700   16 S    0  0.1   0:03.25 apache2                                                                                                                                                
17830 ubuntu    20   0 26768 1468  620 S    0  0.1   0:01.48 bash                                                                                                                                                   
18463 ubuntu    20   0 17332 1288  944 R    0  0.1   0:00.04 top                                                                                                                                                    
 8024 root      20   0 2632m  936    0 S    0  0.1  14:24.58 java                                                                                                                                                   
18134 root      20   0 73352  772    0 S    0  0.0   0:00.06 sshd                                                                                                                                                   
18282 ubuntu    20   0 73352  756    0 S    0  0.0   0:00.03 sshd                                                                                                                                                   
    1 root      20   0 24420  516    0 S    0  0.0   0:13.99 init                                                                                                                                                   
  909 postfix   20   0 27372  348  208 S    0  0.0   0:43.46 qmgr                                                                                                                                                   
  797 root      20   0 15972  320  232 S    0  0.0  25:04.39 irqbalance                                                                                                                                             
  900 root      20   0 25096  272  172 S    0  0.0   1:49.94 master                                                                                                                                                 
  664 syslog    20   0  248m  244    0 S    0  0.0  18:20.90 rsyslogd                                                                                                                                               
16201 ubuntu    20   0 73468  220    0 S    0  0.0   0:00.07 sshd                                                                                                                                                   
16904 ubuntu    20   0 73468  216    0 S    0  0.0   0:00.03 sshd   
Timeout 20
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 2
IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      12
    MaxClients           40
    MaxRequestsPerChild  10000
/IfModule>

Completely new to all of this.. any advice much appreciated. Cheers

2
  • Can you clarify what you mean by "crawl / crash" ? Is it HTTP requests that take a long time? MySQL queries? SSH traffic? Bash commands? That top output looks like a completely idle server.
    – Ladadadada
    Feb 28, 2013 at 16:16
  • A bit of a guess with the data provided, but: you have about 160M of free RAM, each new apache process is taking about 70M of RAM. If you launch a few extra apache processes (up to MaxClients), you could conceivably start swapping - and slow to a crawl. sar (sysstat) might get you a quick diagnosis (especially if you already have it setup), but atop will likely provide you with more detailed information. (Also: unusual spike in which graph(s) (IO, RAM, CPU, etc.)?
    – cyberx86
    Feb 28, 2013 at 19:13

2 Answers 2

1

Install atop and run it as a daemon logging to /var/log/atop.log. Then, next time you have a nasty load spike, you can run atop -r /var/log/atop.log and look at the state of the machine at the time it happened (use the t key to fast-forward through the log).

atop is superior to top in the usefulness and completeness of data it shows you.

1
  • atop is an excellent tool, but I think the listing we have from top is enough in this case.
    – mc0e
    Oct 16, 2014 at 19:09
0

Your memory use looks high. I can see that you're using quite a bit of swap space at the time you collected that info from top, but it's really the amount of data moving in and out of swap rather than the amount in use that's the issue. I'm guessing this is the case.

Is your swap used figure changing from reading to reading? If so, that's almost certainly your problem, and in any case you'll benefit from reducing memory usage and/or adding RAM to your VM.

also, if churning swap is your issue, then when your machine is doing anything, you'll see the number of processes in wait state rise. It's "0.0%wa" in your listing, but given that the system is showing 100% idle, I'm presuming it's in between the web requests that would cause problems.

Just doing a back of the envelope type calculation, if your apache processes are currently each consuming about 5% of your RAM (and some more) then allowing up to 40 of them (MaxClients) is trouble, and would use abouttwice as much RAM as you actually have.

For a smallish dev machine like yours, I'd fix the number of apache processes at a small number, without much dynamic movement, and recycle apache processes often. Also keep keepalive very short so these aren't tied up for long.

Timeout 20
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 1
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       3
    MaxSpareServers      5
    MaxClients           5
    MaxRequestsPerChild  10
</IfModule>

If you have more than one or two people using this at once, look at passing off static content delivery to something single threaded and light on memory use. You could deliver static content from a lighter weight httpd like nginx, but it'd be easier to just set up something like squid as a front end proxy cache, with perhaps 50 to 100 MB of cache.

Make sure you are using APC cache for caching compiled PHP. With APC you'll want to increase the ammount of space assigned to the cache, and that will actually reduce the RAM usage quite a bit, as the compiled PHP will be in shared mmapped memory rather than being compiled separately in each apache process on every request.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .