1

I noticed that my site was down today and when I ssh'd into my box and ran top, I noticed that my server load was super high. I quickly shut off apache and waited until the load dropped, but as soon as I started apache back up the load and processes start to climb very quickly. The site is a basic wordpress blog.

Here's the output of top

top - 15:24:50 up 11:19,  1 user,  load average: 29.76, 24.04, 19.27
Tasks: 129 total,  33 running,  96 sleeping,   0 stopped,   0 zombie
%Cpu(s): 30.5 us,  2.8 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 66.8 st
KiB Mem:   2048448 total,  1092188 used,   956260 free,   162896 buffers
KiB Swap:  1048572 total,        0 used,  1048572 free.   397860 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                   
 1061 mysql     20   0  897664 111416   7784 S  4.3  5.4  13:56.63 mysqld                                                                                                                                                                                                                                                    
 7205 ubuntu    20   0  406700  38676  23276 R  3.6  1.9   0:20.75 apache2                                                                                                                                                                                                                                                   
 7273 ubuntu    20   0  406840  41928  26636 R  3.6  2.0   0:13.05 apache2                                                                                                                                                                                                                                                   
 7137 ubuntu    20   0  407284  39664  23872 R  3.3  1.9   0:30.87 apache2                                                                                                                                                                                                                                                   
 7196 ubuntu    20   0  407208  40160  24464 R  3.3  2.0   0:21.61 apache2                                                                                                                                                                                                                                                   
 7202 ubuntu    20   0  406796  38720  23168 R  3.3  1.9   0:21.10 apache2                                                                                                                                                                                                                                                   
 7203 ubuntu    20   0  407576  42880  26596 R  3.3  2.1   0:20.74 apache2                                                                                                                                                                                                                                                   
 7218 ubuntu    20   0  406040  38772  24004 R  3.3  1.9   0:18.09 apache2                                                                                                                                                                                                                                                   
 7219 ubuntu    20   0  406952  39360  23744 R  3.3  1.9   0:18.09 apache2                                                                                                                                                                                                                                                   
 7222 ubuntu    20   0  406804  38420  23128 R  3.3  1.9   0:17.48 apache2                                                                                                                                                                                                                                                   
 7228 ubuntu    20   0  402540  36764  25320 R  3.3  1.8   0:17.75 apache2                                                                                                                                                                                                                                                   
 7244 ubuntu    20   0  407060  38652  23096 R  3.3  1.9   0:16.10 apache2                                                                                                                                                                                                                                                   
 7245 ubuntu    20   0  407052  38764  23220 R  3.3  1.9   0:15.67 apache2                                                                                                                                                                                                                                                   
 7252 ubuntu    20   0  406736  39336  23944 R  3.3  1.9   0:15.70 apache2                                                                                                                                                                                                                                                   
 7254 ubuntu    20   0  407316  39552  23712 R  3.3  1.9   0:14.81 apache2                                                                                                                                                                                                                                                   
 7256 ubuntu    20   0  407264  40444  24748 R  3.3  2.0   0:14.41 apache2                                                                                                                                                                                                                                                   
 7260 ubuntu    20   0  406544  38948  23908 R  3.3  1.9   0:15.01 apache2

Here's the number of connected IP's at the time:

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
     2 75.112.**
     1 54.251.**
     1 54.245.**
     1 54.232.**
     1 54.228.**

And here's my hardware info

  description: Computer
  product: HVM domU ()
  vendor: Xen
  version: 4.2.amazon
  serial: **
  width: 64 bits
  capabilities: smbios-2.4 dmi-2.4 ldt16 vsyscall32
  configuration: boot=normal uuid=**
*-core
     description: Motherboard
     physical id: 0
   *-firmware:0
        description: BIOS
        vendor: Xen
        physical id: 0
        version: 4.2.amazon
        date: 12/03/2014
        size: 96KiB
        capabilities: pci edd
   *-cpu:0
        description: CPU
        product: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
        vendor: Intel Corp.
        physical id: 1
        bus info: cpu@0
        slot: CPU 1
        size: 2500MHz
        capacity: 2500MHz
        width: 64 bits
        capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp x86-64 constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep erms
   *-memory:0
        description: System Memory
        physical id: 2
        capacity: 2GiB
      *-bank:0
           description: DIMM RAM
           physical id: 0
           slot: DIMM 0
           size: 2GiB
           width: 64 bits
      *-bank:1
           description: DIMM RAM
           physical id: 1
           slot: DIMM 0
           size: 2GiB
           width: 64 bits
   *-firmware:1
        description: BIOS
        vendor: Xen
        physical id: 3
        version: 4.2.amazon
        date: 12/03/2014
        size: 96KiB
        capabilities: pci edd
   *-cpu:1
        description: CPU
        vendor: Intel
        physical id: 4
        bus info: cpu@1
        slot: CPU 1
        size: 2500MHz
        capacity: 2500MHz
   *-memory:1
        description: System Memory
        physical id: 5
        capacity: 2GiB
   *-memory:2 UNCLAIMED
        physical id: 6
   *-memory:3 UNCLAIMED
        physical id: 7
   *-pci
        description: Host bridge
        product: 440FX - 82441FX PMC [Natoma]
        vendor: Intel Corporation
        physical id: 100
        bus info: pci@0000:00:00.0
        version: 02
        width: 32 bits
        clock: 33MHz
      *-isa
           description: ISA bridge
           product: 82371SB PIIX3 ISA [Natoma/Triton II]
           vendor: Intel Corporation
           physical id: 1
           bus info: pci@0000:00:01.0
           version: 00
           width: 32 bits
           clock: 33MHz
           capabilities: isa bus_master
           configuration: latency=0
      *-ide
           description: IDE interface
           product: 82371SB PIIX3 IDE [Natoma/Triton II]
           vendor: Intel Corporation
           physical id: 1.1
           bus info: pci@0000:00:01.1
           version: 00
           width: 32 bits
           clock: 33MHz
           capabilities: ide bus_master
           configuration: driver=ata_piix latency=64
           resources: irq:0 ioport:1f0(size=8) ioport:3f6 ioport:170(size=8) ioport:376 ioport:c100(size=16)
      *-bridge UNCLAIMED
           description: Bridge
           product: 82371AB/EB/MB PIIX4 ACPI
           vendor: Intel Corporation
           physical id: 1.3
           bus info: pci@0000:00:01.3
           version: 01
           width: 32 bits
           clock: 33MHz
           capabilities: bridge bus_master
           configuration: latency=0
      *-display UNCLAIMED
           description: VGA compatible controller
           product: GD 5446
           vendor: Cirrus Logic
           physical id: 2
           bus info: pci@0000:00:02.0
           version: 00
           width: 32 bits
           clock: 33MHz
           capabilities: vga_controller bus_master
           configuration: latency=0
           resources: memory:f0000000-f1ffffff memory:f3000000-f3000fff
      *-generic
           description: Unassigned class
           product: Xen Platform Device
           vendor: XenSource, Inc.
           physical id: 3
           bus info: pci@0000:00:03.0
           version: 01
           width: 32 bits
           clock: 33MHz
           capabilities: bus_master
           configuration: driver=xen-platform-pci latency=0
           resources: irq:28 ioport:c000(size=256) memory:f2000000-f2ffffff
*-network
     description: Ethernet interface
     physical id: 1
     logical name: eth0
     serial: **
     capabilities: ethernet physical
     configuration: broadcast=yes driver=vif ip=*** link=yes multicast=yes

Any suggestions as to how I can debug this further?

EDIT:

I thought I would add this as well. I dropped MaxClients from 30 to 10 as per @nochkin suggestion and the load drop to about 8.0.

<IfModule mpm_prefork_module>
    StartServers          3
    MinSpareServers       3
    MaxSpareServers       5
    MaxClients           10
    MaxRequestsPerChild   0
</IfModule>
1

You'll need to adjust the number of child processes you start to something lower. Depending on your configuration, that may be located in the httpd-mpm.conf or httpd.conf file.

2
  • I've set the MaxClients from 30 to 10 and the load has dropped to about 8.0, which is better, but not ideal.
    – Ken
    Dec 17 '14 at 16:12
  • You have a basic VM, and your website is not idle. You probably need further tuning which is out of scope of Apache configuration. For example, you can disable Apache modules you don't use to save on memory. Then you can read about wordpress-specific tuning for PHP to gain something there (APC, memcache, etc). MySQL tuning may gain additional score (it's easy to run into swap when doing it, so be careful). If you feel adventurous, you can even drop Apache and switch to something like nginx/php-fpm combination.
    – nochkin
    Dec 17 '14 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.