0

Environment

  • Ubuntu 12.04 / 1GB RAM / 40GB DISK / 1 CPU (RACKSPACE)
  • Wordpress 3.9
  • mysql Ver 14.14 Distrib 5.5.38, for debian-linux-gnu (x86_64) using readline 6.2
  • Apache2
  • mod_php5
  • mod_rewrite

When a fresh restart of apache2 and mysql accessing my website a couple of time is enough to use up all the 1GB memory, which stalls mysql hence my wordpress website.

After a restart of services

$ service apache2 restart
$ service mysql restart
$ atop

ATOP - staging-www                                               2014/07/30  23:52:11                                               ------                                                10s elapsed
PRC | sys    0.02s  | user   0.00s  |               | #proc     83  | #trun      1  | #tslpi    90  |               | #tslpu     0  | #zombie    0  | clones    59  |               | #exit     59  |
CPU | sys       1%  | user      0%  | irq       0%  |               | idle     99%  | wait      0%  |               |               | steal     0%  | guest     0%  | curf 2.59GHz  | curscal   ?%  |
CPL | avg1    0.00  |               | avg5    0.06  | avg15   0.21  |               |               | csw      786  |               | intr     252  |               |               | numcpu     1  |
MEM | tot   991.5M  | free  850.3M  | cache  51.5M  |               | dirty   0.1M  | buff    2.9M  | slab   15.2M  |               |               |               |               |               |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |               |               |               |               | vmcom 387.9M  | vmlim 495.8M  |
NET | transport     | tcpi       2  | tcpo       3  | udpi       0  | udpo       0  | tcpao      0  | tcppo      0  | tcprs      0  | tcpie      0  | tcpor      0  | udpnp      0  | udpip      0  |
NET | network       | ipi        2  | ipo        3  |               | ipfrw      0  | deliv      2  |               |               |               |               | icmpi      0  | icmpo      0  |
NET | eth0    ----  | pcki       2  | pcko       3  | si    0 Kbps  | so    0 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

  PID       RUID           EUID            THR        SYSCPU        USRCPU        VGROW        RGROW        RDDSK        WRDSK       ST       EXC       S       CPUNR        CPU       CMD        1/2
30194       root           root              1         0.01s         0.00s           0K           0K           0K           0K       --         -       R           0         0%       atop
30409       root           -          

Note the free 850.3M

I access my wordpress website once

ATOP - staging-www                                               2014/07/30  23:54:11                                               ------                                                10s elapsed
PRC | sys    0.09s  | user   0.19s  |               | #proc     90  | #trun      1  | #tslpi   112  |               | #tslpu     0  | #zombie    0  | clones    60  |               | #exit     59  |
CPU | sys       2%  | user      2%  | irq       0%  |               | idle     96%  | wait      0%  |               |               | steal     0%  | guest     0%  | curf 2.59GHz  | curscal   ?%  |
CPL | avg1    0.00  |               | avg5    0.04  | avg15   0.19  |               |               | csw     1503  |               | intr     406  |               |               | numcpu     1  |
MEM | tot   991.5M  | free  388.3M  | cache 125.8M  |               | dirty   0.2M  | buff    4.4M  | slab   17.0M  |               |               |               |               |               |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |               |               |               |               | vmcom   1.0G  | vmlim 495.8M  |
DSK |         xvda  | busy      0%  | read       0  |               | write     19  | KiB/r      0  | KiB/w     16  | MBr/s   0.00  |               | MBw/s   0.03  | avq     3.00  | avio 0.21 ms  |
NET | transport     | tcpi      19  | tcpo      22  | udpi       0  | udpo       0  | tcpao      0  | tcppo      0  | tcprs      0  | tcpie      0  | tcpor      1  | udpnp      0  | udpip      0  |
NET | network       | ipi       19  | ipo       22  |               | ipfrw      0  | deliv     19  |               |               |               |               | icmpi      0  | icmpo      0  |
NET | eth0    ----  | pcki      17  | pcko      20  | si    0 Kbps  | so    5 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |
NET | lo      ----  | pcki       2  | pcko       2  | si    0 Kbps  | so    0 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

  PID       RUID           EUID            THR        SYSCPU        USRCPU        VGROW        RGROW        RDDSK        WRDSK       ST       EXC       S       CPUNR        CPU       CMD        1/2
30587       www-data       www-data          1         0.05s         0.17s        4572K        5796K           0K           0K       --         -       S           0         2%       apache2
30599       mysql          mysql            16         0.02s         0.01s           0K         -84K           0K         288K       --         -       S           0         0%       mysqld
30194       root           root              1         0.02s         0.01s           0K           0K           0K           0K       --         -       R           0         0%       atop
30577       www-data       www-data          1         0.00s         0.00s       -9480K       -8960K           0K           0K       --         -       S           0         0%       apache2
30559       root           root              1         0.00s         0.00s           0K          20K           0K           0K       --         -       S           0         0%       apache2
30877       www-data       www-data          1         0.00s         0.00s       218.6M        5364K           0K           0K       N-         -       S           0         0%       apache2
  457       root           root              1         0.00s         0.00s           0K           0K           0K          36K       --         -       S           0         0%       xe-daemon
30936       root           root              1         0.00s         0.00s        4312K         348K           0K           0K       N-         -       S           0         0%       sleep
30513       root           -                 0         0.00s         0.00s           0K           0K            -            -       -E         0       E           -         0%       <sleep>

note the free 388.3M

Access a second time the same URL (home page)

ATOP - staging-www                                               2014/07/30  23:55:51                                               ------                                                10s elapsed
PRC | sys    0.60s  | user   2.77s  |               | #proc     90  | #trun      1  | #tslpi   112  |               | #tslpu     0  | #zombie    0  | clones     4  |               | #exit      0  |
CPU | sys       6%  | user     28%  | irq       0%  |               | idle     65%  | wait      0%  |               |               | steal     0%  | guest     0%  | curf 2.59GHz  | curscal   ?%  |
CPL | avg1    0.00  |               | avg5    0.03  | avg15   0.17  |               |               | csw     4154  |               | intr    1488  |               |               | numcpu     1  |
MEM | tot   991.5M  | free   81.4M  | cache 125.8M  |               | dirty   0.0M  | buff    4.5M  | slab   17.1M  |               |               |               |               |               |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |               |               |               |               | vmcom   1.3G  | vmlim 495.8M  |
DSK |         xvda  | busy      0%  | read       0  |               | write     31  | KiB/r      0  | KiB/w      9  | MBr/s   0.00  |               | MBw/s   0.03  | avq     0.00  | avio 0.00 ms  |
NET | transport     | tcpi     212  | tcpo     140  | udpi       2  | udpo       2  | tcpao      2  | tcppo      8  | tcprs      5  | tcpie      0  | tcpor      2  | udpnp      0  | udpip      0  |
NET | network       | ipi      214  | ipo      135  |               | ipfrw      0  | deliv    214  |               |               |               |               | icmpi      0  | icmpo      0  |
NET | eth0    ----  | pcki     196  | pcko     117  | si   93 Kbps  | so   27 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |
NET | lo      ----  | pcki      18  | pcko      18  | si    2 Kbps  | so    2 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

  PID       RUID           EUID            THR        SYSCPU        USRCPU        VGROW        RGROW        RDDSK        WRDSK       ST       EXC       S       CPUNR        CPU       CMD        1/1
30592       www-data       www-data          1         0.17s         0.74s       96188K       98620K           0K           4K       --         -       S           0         9%       apache2
30588       www-data       www-data          1         0.14s         0.77s       73932K       76228K           0K           4K       --         -       S           0         9%       apache2
30589       www-data       www-data          1         0.12s         0.51s       68408K       72084K           0K           4K       --         -       S           0         6%       apache2
30604       www-data       www-data          1         0.10s         0.49s       68408K       72084K           0K           0K       --         -       S           0         6%       apache2
30599       mysql          mysql            16         0.06s         0.24s           0K          12K           0K         256K       --         -       S           0         3%       mysqld
30194       root           root              1         0.01s         0.01s           0K           0K           0K           0K       --         -       R           0         0%       atop
30597       www-data       www-data          1         0.00s         0.01s         676K        3208K           0K           4K       --         -       S           0         0%       apache2
30590       www-data       www-data          1         0.00s         0.00s         676K        3212K           0K           0K       --         -       S           0         0%       apache2
30598       www-data       www-data          1         0.00s         0.00s         156K        1000K           0K           4K       --         -       S           0         0%       apache2
30605       www-data       www-data          1         0.00s         0.00s         156K        1000K           0K           8K       --         -       S           0         0%       apache2
  150       root           root              1         0.00s         0.00s           0K           0K           0K          52K       --         -       S           0         0%       kjournald

Note free 81.4M

After reloading the same page 4/5 times

I am getting Error establishing a database connection instead of my home page

ATOP - staging-www                                               2014/07/30  23:57:31                                               ------                                                10s elapsed
PRC | sys    0.52s  | user   0.19s  |               | #proc     89  | #trun      1  | #tslpi    96  |               | #tslpu     0  | #zombie    0  | clones    71  |               | #exit     63  |
CPU | sys       6%  | user      3%  | irq       0%  |               | idle     80%  | wait     11%  |               |               | steal     0%  | guest     0%  | curf 2.59GHz  | curscal   ?%  |
CPL | avg1    0.68  |               | avg5    0.19  | avg15   0.21  |               |               | csw    11267  |               | intr    8436  |               |               | numcpu     1  |
MEM | tot   991.5M  | free   61.9M  | cache  34.6M  |               | dirty   0.2M  | buff    1.0M  | slab   15.2M  |               |               |               |               |               |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |               |               |               |               | vmcom   1.2G  | vmlim 495.8M  |
PAG | scan   64113  |               |               | stall      0  |               |               |               |               | swin       0  |               |               | swout      0  |
DSK |         xvda  | busy     14%  | read    7704  |               | write     18  | KiB/r     26  | KiB/w     16  | MBr/s  19.79  |               | MBw/s   0.03  | avq     3.53  | avio 0.18 ms  |
NET | transport     | tcpi     110  | tcpo      69  | udpi       2  | udpo       2  | tcpao      0  | tcppo      6  | tcprs      8  | tcpie      0  | tcpor      0  | udpnp      0  | udpip      0  |
NET | network       | ipi      112  | ipo       74  |               | ipfrw      0  | deliv    112  |               |               |               |               | icmpi      0  | icmpo      0  |
NET | eth0    ----  | pcki     112  | pcko      74  | si   38 Kbps  | so   11 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

  PID       RUID           EUID            THR        SYSCPU        USRCPU        VGROW        RGROW        RDDSK        WRDSK       ST       EXC       S       CPUNR        CPU       CMD        1/3
30574       www-data       www-data          1         0.32s         0.11s       29956K       29096K       148.1M           0K       --         -       S           0         4%       apache2
30589       www-data       www-data          1         0.02s         0.02s        2104K        2648K        6308K           4K       --         -       S           0         0%       apache2
31118       mysql          -                 0         0.02s         0.01s           0K           0K            -            -       NE         1       E           -         0%       <mysqld>
31154       mysql          -                 0         0.02s         0.01s           0K           0K            -            -       NE         1       E           -         0%       <mysqld>
  814       root           root              2         0.02s         0.00s           0K         164K       17032K           0K       --         -       S           0         0%       nova-agent
30586       www-data       www-data          1         0.01s         0.01s         520K         172K          80K           0K       --         -       S           0         0%       apache2
    1       root           root              1         0.02s         0.00s           0K         352K       409.3M        9480K       --         -       S           0         0%       init
   21       root           root              1         0.02s         0.00s           0K           0K           0K           0K       --         -       S           0         0%       kworker/0:1
30870       www-data       www-data          1         0.00s         0.01s           0K       -2608K         432K           0K       --         -       S           0         0%       apache2
30568       www-data       www-data          1         0.00s         0.01s           0K        -548K          16K           4K       --         -       S           0         0%       apache2
30585       www-data       www-data          1         0.01s         0.00s           0K        -672K          44K           0K       --         -       S           0         0%       apache2
30194       root           root              1         0.01s         0.00s           0K           0K           8K           0K       --         -       R           0         0%       atop

Trying to restart mysql

root@staging-www:~# service mysql restart
stop: Unknown instance: 
start: Job failed to start
root@staging-www:~# 

It gets back on his feet after:

root@staging-www:~# service apache2 restart && service mysql restart

This obviously not due to a traffic, one single request already uses the memory to a third (?) and the subsequent requests just use up everything and stalls my machine and the memory never gets recovered (??)

On production I have upgraded to 2 GB, but I have the feelings that something is wrong with my setup, and I wonder if that could be something with my wordpress code. But I don't know where to start really.

The apache2 log /var/log/apache2/error.log show no errors.

Any other logs I should look at?

Does apache2 actually need 2 GB to be able to run? (that's seems unlikely right?)

UPDATE - My current apache2.conf

<IfModule mpm_prefork_module>
    StartServers          16
    MinSpareServers       16
    MaxSpareServers       32
    ServerLimit           400
    MaxClients            400
    MaxRequestsPerChild   10000
</IfModule>


<IfModule mpm_worker_module>
    StartServers          4
    ServerLimit           16
    MaxClients            1024
    MinSpareThreads       64
    MaxSpareThreads       192
    ThreadsPerChild       64
    MaxRequestsPerChild   0
</IfModule>

UPDATE - New tuning (still having memory issue)

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers       10
    ServerLimit           400
    MaxClients            150
    MaxRequestsPerChild   0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           16
    MaxClients            150
    MinSpareThreads       25
    MaxSpareThreads       75
    ThreadsPerChild       64
    MaxRequestsPerChild   0
</IfModule>

I have tuned down my MPM but I am still out of memory

ATOP - staging-www                                               2014/07/31  00:57:11                                               ------                                                10s elapsed
PRC | sys    0.89s  | user   2.90s  |               | #proc     79  | #trun      1  | #tslpi   101  |               | #tslpu     0  | #zombie    0  | clones    11  |               | #exit      0  |
CPU | sys       9%  | user     30%  | irq       0%  |               | idle     61%  | wait      1%  |               |               | steal     0%  | guest     0%  | curf 2.59GHz  | curscal   ?%  |
CPL | avg1    0.05  |               | avg5    0.39  | avg15   0.30  |               |               | csw     5914  |               | intr    2938  |               |               | numcpu     1  |
MEM | tot   991.5M  | free  410.6M  | cache 122.2M  |               | dirty   0.2M  | buff    3.7M  | slab   16.1M  |               |               |               |               |               |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |               |               |               |               | vmcom 877.0M  | vmlim 495.8M  |
DSK |         xvda  | busy      2%  | read    1425  |               | write     16  | KiB/r     34  | KiB/w     17  | MBr/s   4.82  |               | MBw/s   0.03  | avq     3.26  | avio 0.17 ms  |
NET | transport     | tcpi     179  | tcpo     116  | udpi       2  | udpo       2  | tcpao      2  | tcppo      8  | tcprs      1  | tcpie      0  | tcpor      2  | udpnp      0  | udpip      0  |
NET | network       | ipi      181  | ipo      111  |               | ipfrw      0  | deliv    181  |               |               |               |               | icmpi      0  | icmpo      0  |
NET | eth0    ----  | pcki     163  | pcko      93  | si   80 Kbps  | so   28 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |
NET | lo      ----  | pcki      18  | pcko      18  | si    2 Kbps  | so    2 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

  PID       RUID           EUID            THR        SYSCPU        USRCPU        VGROW        RGROW        RDDSK        WRDSK       ST       EXC       S       CPUNR        CPU       CMD        1/1
 6154       www-data       www-data          1         0.19s         0.77s       83896K       86248K          40K           0K       --         -       S           0        10%       apache2
 6152       www-data       www-data          1         0.18s         0.78s       73932K       76228K         660K           0K       --         -       S           0        10%       apache2
 6153       www-data       www-data          1         0.11s         0.53s       68408K       72084K           0K           4K       --         -       S           0         6%       apache2
 6449       www-data       www-data          1         0.16s         0.47s       285.4M       77444K           0K           0K       N-         -       S           0         6%       apache2
 6179       mysql          mysql            16         0.20s         0.35s       320.0M       49216K       48508K         256K       --         -       S           0         6%       mysqld
 6147       root           root              1         0.02s         0.00s           0K           0K           0K           0K       --         -       S           0         0%       apache2
30194       root           root              1         0.02s         0.00s           0K           0K           0K           0K       --         -       R           0         0%       atop
    4       root           root              1         0.01s         0.00s           0K           0K           0K           0K       --         -       S           0         0%       kworker/0:0
 6155       www-data       www-data          1         0.00s         0.00s         148K        1000K          36K           4K       --         -       S           0         0%       apache2
 6156       www-data       www-data          1         0.00s         0.00s         148K        1000K          36K           4K       --         -       S           0         0%       apache2
 6445       www-data       www-data          1         0.00s         0.00s       218.8M        6360K          36K           4K       N-         -       S           0         0%       apache2
 6448       www-data       www-data          1         0.00s         0.00s       218.7M        6360K          56K           8K       N-         -       S           0         0%       apache2
 6450       www-data       www-data          1         0.00s         0.00s       218.6M        5360K           0K           0K       N-         -       S           0         0%       apache2
 6451       www-data       www-data          1         0.00s         0.00s       218.6M        5360K           0K           0K       N-         -       S           0         0%       apache2
 6452       www-data       www-data          1         0.00s         0.00s       218.6M        5360K           0K           0K       N-         -       S           0         0%       apache2
 6455       www-data       www-data          1         0.00s         0.00s       218.6M        5360K           0K           0K       N-         -       S           0         0%       apache2
  962       ntp            ntp               1         0.00s         0.00s           0K         100K           0K           0K       --         -       S           0         0%       ntpd
  150       root           root              1         0.00s         0.00s           0K           0K           0K          48K       --         -       S           0         0%       kjournald

I have restarted apache2 to take in account the new config. but that doesn't seem enough.

Looks like it didn't take in account my settings, I have more that 5 apache2 services started.

When accessing my website through 5/6 url in different tabs of my browser I run into the same out of memory issue.

PRC | sys    0.60s  | user   3.05s  |               | #proc     80  | #trun      4  | #tslpi   105  |               | #tslpu     0  | #zombie    0  | clones     7  |               | #exit      0  |
CPU | sys       6%  | user     31%  | irq       0%  |               | idle     63%  | wait      0%  |               |               | steal     0%  | guest     0%  | curf 2.59GHz  | curscal   ?%  |
CPL | avg1    0.35  |               | avg5    0.10  | avg15   0.13  |               |               | csw     7047  |               | intr    3252  |               |               | numcpu     1  |
MEM | tot   991.5M  | free    9.2M  | cache   9.0M  |               | dirty   0.0M  | buff    0.1M  | slab   15.4M  |               |               |               |               |               |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |               |               |               |               | vmcom   1.4G  | vmlim 495.8M  |
PAG | scan   35266  |               |               | stall      0  |               |               |               |               | swin       0  |               |               | swout      0  |
DSK |         xvda  | busy      2%  | read     830  |               | write     74  | KiB/r     14  | KiB/w     13  | MBr/s   1.17  |               | MBw/s   0.10  | avq     1.83  | avio 0.26 ms  |
NET | transport     | tcpi     641  | tcpo     358  | udpi       5  | udpo       5  | tcpao      2  | tcppo      2  | tcprs      0  | tcpie      0  | tcpor      1  | udpnp      0  | udpip      0  |
NET | network       | ipi      646  | ipo      342  |               | ipfrw      0  | deliv    646  |               |               |               |               | icmpi      0  | icmpo      0  |
NET | eth0    ----  | pcki     630  | pcko     326  | si  340 Kbps  | so   95 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |
NET | lo      ----  | pcki      16  | pcko      16  | si    1 Kbps  | so    1 Kbps  | coll       0  |               | mlti       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

  PID       RUID           EUID            THR        SYSCPU        USRCPU        VGROW        RGROW        RDDSK        WRDSK       ST       EXC       S       CPUNR        CPU       CMD        1/1
 6156       www-data       www-data          1         0.22s         0.78s       103.0M       102.5M        3644K          12K       --         -       R           0        10%       apache2
 6152       www-data       www-data          1         0.04s         0.56s       -1280K       -2256K          36K          12K       --         -       S           0         6%       apache2
 6452       www-data       www-data          1         0.03s         0.49s        -396K       -1440K           0K           0K       --         -       S           0         5%       apache2
 7466       www-data       www-data          1         0.10s         0.26s       54108K       55692K        2644K           0K       --         -       R           0         4%       apache2

...

  • 1
    Why do you have so many apache2 processes? Try some basic tuning. – Michael Hampton Jul 31 '14 at 0:16
  • I don't know this came out of the box. I am using Chef to configure the server. May be I should mention that in my question. Thanks for the lead. I will update my question accordingly – zabumba Jul 31 '14 at 0:20
  • 1
    Yep, those are absurdly high numbers for a small VPS. – Michael Hampton Jul 31 '14 at 0:26
  • 1
    You still have too many Apache processes, considering their high memory usage. Are you running some memory hungry PHP app? – Michael Hampton Jul 31 '14 at 1:19
  • 1
    With a few dozen plugins, no doubt. So the answer is yes :) Keep on tuning. – Michael Hampton Jul 31 '14 at 1:22
0

Following Michael's recommendations and tuned down my apache2.conf a maximum on my staging instance. These are not production settings but they address my question.

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          1
    MinSpareServers       1
    MaxSpareServers       2
    ServerLimit           5
    MaxClients            3
    MaxRequestsPerChild   0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          1
    ServerLimit           3
    MaxClients            3
    MinSpareThreads       10
    MaxSpareThreads       20
    ThreadsPerChild       64
    MaxRequestsPerChild   0
</IfModule>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.