Environment
- Ubuntu 12.04 / 1GB RAM / 40GB DISK / 1 CPU (RACKSPACE)
- Wordpress 3.9
- mysql Ver 14.14 Distrib 5.5.38, for debian-linux-gnu (x86_64) using readline 6.2
- Apache2
- mod_php5
- mod_rewrite
When a fresh restart of apache2 and mysql accessing my website a couple of time is enough to use up all the 1GB memory, which stalls mysql hence my wordpress website.
After a restart of services
$ service apache2 restart
$ service mysql restart
$ atop
ATOP - staging-www 2014/07/30 23:52:11 ------ 10s elapsed
PRC | sys 0.02s | user 0.00s | | #proc 83 | #trun 1 | #tslpi 90 | | #tslpu 0 | #zombie 0 | clones 59 | | #exit 59 |
CPU | sys 1% | user 0% | irq 0% | | idle 99% | wait 0% | | | steal 0% | guest 0% | curf 2.59GHz | curscal ?% |
CPL | avg1 0.00 | | avg5 0.06 | avg15 0.21 | | | csw 786 | | intr 252 | | | numcpu 1 |
MEM | tot 991.5M | free 850.3M | cache 51.5M | | dirty 0.1M | buff 2.9M | slab 15.2M | | | | | |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 387.9M | vmlim 495.8M |
NET | transport | tcpi 2 | tcpo 3 | udpi 0 | udpo 0 | tcpao 0 | tcppo 0 | tcprs 0 | tcpie 0 | tcpor 0 | udpnp 0 | udpip 0 |
NET | network | ipi 2 | ipo 3 | | ipfrw 0 | deliv 2 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 2 | pcko 3 | si 0 Kbps | so 0 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/2
30194 root root 1 0.01s 0.00s 0K 0K 0K 0K -- - R 0 0% atop
30409 root -
Note the free 850.3M
I access my wordpress website once
ATOP - staging-www 2014/07/30 23:54:11 ------ 10s elapsed
PRC | sys 0.09s | user 0.19s | | #proc 90 | #trun 1 | #tslpi 112 | | #tslpu 0 | #zombie 0 | clones 60 | | #exit 59 |
CPU | sys 2% | user 2% | irq 0% | | idle 96% | wait 0% | | | steal 0% | guest 0% | curf 2.59GHz | curscal ?% |
CPL | avg1 0.00 | | avg5 0.04 | avg15 0.19 | | | csw 1503 | | intr 406 | | | numcpu 1 |
MEM | tot 991.5M | free 388.3M | cache 125.8M | | dirty 0.2M | buff 4.4M | slab 17.0M | | | | | |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 1.0G | vmlim 495.8M |
DSK | xvda | busy 0% | read 0 | | write 19 | KiB/r 0 | KiB/w 16 | MBr/s 0.00 | | MBw/s 0.03 | avq 3.00 | avio 0.21 ms |
NET | transport | tcpi 19 | tcpo 22 | udpi 0 | udpo 0 | tcpao 0 | tcppo 0 | tcprs 0 | tcpie 0 | tcpor 1 | udpnp 0 | udpip 0 |
NET | network | ipi 19 | ipo 22 | | ipfrw 0 | deliv 19 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 17 | pcko 20 | si 0 Kbps | so 5 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | lo ---- | pcki 2 | pcko 2 | si 0 Kbps | so 0 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/2
30587 www-data www-data 1 0.05s 0.17s 4572K 5796K 0K 0K -- - S 0 2% apache2
30599 mysql mysql 16 0.02s 0.01s 0K -84K 0K 288K -- - S 0 0% mysqld
30194 root root 1 0.02s 0.01s 0K 0K 0K 0K -- - R 0 0% atop
30577 www-data www-data 1 0.00s 0.00s -9480K -8960K 0K 0K -- - S 0 0% apache2
30559 root root 1 0.00s 0.00s 0K 20K 0K 0K -- - S 0 0% apache2
30877 www-data www-data 1 0.00s 0.00s 218.6M 5364K 0K 0K N- - S 0 0% apache2
457 root root 1 0.00s 0.00s 0K 0K 0K 36K -- - S 0 0% xe-daemon
30936 root root 1 0.00s 0.00s 4312K 348K 0K 0K N- - S 0 0% sleep
30513 root - 0 0.00s 0.00s 0K 0K - - -E 0 E - 0% <sleep>
note the free 388.3M
Access a second time the same URL (home page)
ATOP - staging-www 2014/07/30 23:55:51 ------ 10s elapsed
PRC | sys 0.60s | user 2.77s | | #proc 90 | #trun 1 | #tslpi 112 | | #tslpu 0 | #zombie 0 | clones 4 | | #exit 0 |
CPU | sys 6% | user 28% | irq 0% | | idle 65% | wait 0% | | | steal 0% | guest 0% | curf 2.59GHz | curscal ?% |
CPL | avg1 0.00 | | avg5 0.03 | avg15 0.17 | | | csw 4154 | | intr 1488 | | | numcpu 1 |
MEM | tot 991.5M | free 81.4M | cache 125.8M | | dirty 0.0M | buff 4.5M | slab 17.1M | | | | | |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 1.3G | vmlim 495.8M |
DSK | xvda | busy 0% | read 0 | | write 31 | KiB/r 0 | KiB/w 9 | MBr/s 0.00 | | MBw/s 0.03 | avq 0.00 | avio 0.00 ms |
NET | transport | tcpi 212 | tcpo 140 | udpi 2 | udpo 2 | tcpao 2 | tcppo 8 | tcprs 5 | tcpie 0 | tcpor 2 | udpnp 0 | udpip 0 |
NET | network | ipi 214 | ipo 135 | | ipfrw 0 | deliv 214 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 196 | pcko 117 | si 93 Kbps | so 27 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | lo ---- | pcki 18 | pcko 18 | si 2 Kbps | so 2 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/1
30592 www-data www-data 1 0.17s 0.74s 96188K 98620K 0K 4K -- - S 0 9% apache2
30588 www-data www-data 1 0.14s 0.77s 73932K 76228K 0K 4K -- - S 0 9% apache2
30589 www-data www-data 1 0.12s 0.51s 68408K 72084K 0K 4K -- - S 0 6% apache2
30604 www-data www-data 1 0.10s 0.49s 68408K 72084K 0K 0K -- - S 0 6% apache2
30599 mysql mysql 16 0.06s 0.24s 0K 12K 0K 256K -- - S 0 3% mysqld
30194 root root 1 0.01s 0.01s 0K 0K 0K 0K -- - R 0 0% atop
30597 www-data www-data 1 0.00s 0.01s 676K 3208K 0K 4K -- - S 0 0% apache2
30590 www-data www-data 1 0.00s 0.00s 676K 3212K 0K 0K -- - S 0 0% apache2
30598 www-data www-data 1 0.00s 0.00s 156K 1000K 0K 4K -- - S 0 0% apache2
30605 www-data www-data 1 0.00s 0.00s 156K 1000K 0K 8K -- - S 0 0% apache2
150 root root 1 0.00s 0.00s 0K 0K 0K 52K -- - S 0 0% kjournald
Note free 81.4M
After reloading the same page 4/5 times
I am getting Error establishing a database connection instead of my home page
ATOP - staging-www 2014/07/30 23:57:31 ------ 10s elapsed
PRC | sys 0.52s | user 0.19s | | #proc 89 | #trun 1 | #tslpi 96 | | #tslpu 0 | #zombie 0 | clones 71 | | #exit 63 |
CPU | sys 6% | user 3% | irq 0% | | idle 80% | wait 11% | | | steal 0% | guest 0% | curf 2.59GHz | curscal ?% |
CPL | avg1 0.68 | | avg5 0.19 | avg15 0.21 | | | csw 11267 | | intr 8436 | | | numcpu 1 |
MEM | tot 991.5M | free 61.9M | cache 34.6M | | dirty 0.2M | buff 1.0M | slab 15.2M | | | | | |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 1.2G | vmlim 495.8M |
PAG | scan 64113 | | | stall 0 | | | | | swin 0 | | | swout 0 |
DSK | xvda | busy 14% | read 7704 | | write 18 | KiB/r 26 | KiB/w 16 | MBr/s 19.79 | | MBw/s 0.03 | avq 3.53 | avio 0.18 ms |
NET | transport | tcpi 110 | tcpo 69 | udpi 2 | udpo 2 | tcpao 0 | tcppo 6 | tcprs 8 | tcpie 0 | tcpor 0 | udpnp 0 | udpip 0 |
NET | network | ipi 112 | ipo 74 | | ipfrw 0 | deliv 112 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 112 | pcko 74 | si 38 Kbps | so 11 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/3
30574 www-data www-data 1 0.32s 0.11s 29956K 29096K 148.1M 0K -- - S 0 4% apache2
30589 www-data www-data 1 0.02s 0.02s 2104K 2648K 6308K 4K -- - S 0 0% apache2
31118 mysql - 0 0.02s 0.01s 0K 0K - - NE 1 E - 0% <mysqld>
31154 mysql - 0 0.02s 0.01s 0K 0K - - NE 1 E - 0% <mysqld>
814 root root 2 0.02s 0.00s 0K 164K 17032K 0K -- - S 0 0% nova-agent
30586 www-data www-data 1 0.01s 0.01s 520K 172K 80K 0K -- - S 0 0% apache2
1 root root 1 0.02s 0.00s 0K 352K 409.3M 9480K -- - S 0 0% init
21 root root 1 0.02s 0.00s 0K 0K 0K 0K -- - S 0 0% kworker/0:1
30870 www-data www-data 1 0.00s 0.01s 0K -2608K 432K 0K -- - S 0 0% apache2
30568 www-data www-data 1 0.00s 0.01s 0K -548K 16K 4K -- - S 0 0% apache2
30585 www-data www-data 1 0.01s 0.00s 0K -672K 44K 0K -- - S 0 0% apache2
30194 root root 1 0.01s 0.00s 0K 0K 8K 0K -- - R 0 0% atop
Trying to restart mysql
root@staging-www:~# service mysql restart
stop: Unknown instance:
start: Job failed to start
root@staging-www:~#
It gets back on his feet after:
root@staging-www:~# service apache2 restart && service mysql restart
This obviously not due to a traffic, one single request already uses the memory to a third (?) and the subsequent requests just use up everything and stalls my machine and the memory never gets recovered (??)
On production I have upgraded to 2 GB, but I have the feelings that something is wrong with my setup, and I wonder if that could be something with my wordpress code. But I don't know where to start really.
The apache2 log /var/log/apache2/error.log show no errors.
Any other logs I should look at?
Does apache2 actually need 2 GB to be able to run? (that's seems unlikely right?)
UPDATE - My current apache2.conf
<IfModule mpm_prefork_module>
StartServers 16
MinSpareServers 16
MaxSpareServers 32
ServerLimit 400
MaxClients 400
MaxRequestsPerChild 10000
</IfModule>
<IfModule mpm_worker_module>
StartServers 4
ServerLimit 16
MaxClients 1024
MinSpareThreads 64
MaxSpareThreads 192
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
UPDATE - New tuning (still having memory issue)
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 400
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
StartServers 2
ServerLimit 16
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
I have tuned down my MPM but I am still out of memory
ATOP - staging-www 2014/07/31 00:57:11 ------ 10s elapsed
PRC | sys 0.89s | user 2.90s | | #proc 79 | #trun 1 | #tslpi 101 | | #tslpu 0 | #zombie 0 | clones 11 | | #exit 0 |
CPU | sys 9% | user 30% | irq 0% | | idle 61% | wait 1% | | | steal 0% | guest 0% | curf 2.59GHz | curscal ?% |
CPL | avg1 0.05 | | avg5 0.39 | avg15 0.30 | | | csw 5914 | | intr 2938 | | | numcpu 1 |
MEM | tot 991.5M | free 410.6M | cache 122.2M | | dirty 0.2M | buff 3.7M | slab 16.1M | | | | | |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 877.0M | vmlim 495.8M |
DSK | xvda | busy 2% | read 1425 | | write 16 | KiB/r 34 | KiB/w 17 | MBr/s 4.82 | | MBw/s 0.03 | avq 3.26 | avio 0.17 ms |
NET | transport | tcpi 179 | tcpo 116 | udpi 2 | udpo 2 | tcpao 2 | tcppo 8 | tcprs 1 | tcpie 0 | tcpor 2 | udpnp 0 | udpip 0 |
NET | network | ipi 181 | ipo 111 | | ipfrw 0 | deliv 181 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 163 | pcko 93 | si 80 Kbps | so 28 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | lo ---- | pcki 18 | pcko 18 | si 2 Kbps | so 2 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/1
6154 www-data www-data 1 0.19s 0.77s 83896K 86248K 40K 0K -- - S 0 10% apache2
6152 www-data www-data 1 0.18s 0.78s 73932K 76228K 660K 0K -- - S 0 10% apache2
6153 www-data www-data 1 0.11s 0.53s 68408K 72084K 0K 4K -- - S 0 6% apache2
6449 www-data www-data 1 0.16s 0.47s 285.4M 77444K 0K 0K N- - S 0 6% apache2
6179 mysql mysql 16 0.20s 0.35s 320.0M 49216K 48508K 256K -- - S 0 6% mysqld
6147 root root 1 0.02s 0.00s 0K 0K 0K 0K -- - S 0 0% apache2
30194 root root 1 0.02s 0.00s 0K 0K 0K 0K -- - R 0 0% atop
4 root root 1 0.01s 0.00s 0K 0K 0K 0K -- - S 0 0% kworker/0:0
6155 www-data www-data 1 0.00s 0.00s 148K 1000K 36K 4K -- - S 0 0% apache2
6156 www-data www-data 1 0.00s 0.00s 148K 1000K 36K 4K -- - S 0 0% apache2
6445 www-data www-data 1 0.00s 0.00s 218.8M 6360K 36K 4K N- - S 0 0% apache2
6448 www-data www-data 1 0.00s 0.00s 218.7M 6360K 56K 8K N- - S 0 0% apache2
6450 www-data www-data 1 0.00s 0.00s 218.6M 5360K 0K 0K N- - S 0 0% apache2
6451 www-data www-data 1 0.00s 0.00s 218.6M 5360K 0K 0K N- - S 0 0% apache2
6452 www-data www-data 1 0.00s 0.00s 218.6M 5360K 0K 0K N- - S 0 0% apache2
6455 www-data www-data 1 0.00s 0.00s 218.6M 5360K 0K 0K N- - S 0 0% apache2
962 ntp ntp 1 0.00s 0.00s 0K 100K 0K 0K -- - S 0 0% ntpd
150 root root 1 0.00s 0.00s 0K 0K 0K 48K -- - S 0 0% kjournald
I have restarted apache2 to take in account the new config. but that doesn't seem enough.
Looks like it didn't take in account my settings, I have more that 5 apache2 services started.
When accessing my website through 5/6 url in different tabs of my browser I run into the same out of memory issue.
PRC | sys 0.60s | user 3.05s | | #proc 80 | #trun 4 | #tslpi 105 | | #tslpu 0 | #zombie 0 | clones 7 | | #exit 0 |
CPU | sys 6% | user 31% | irq 0% | | idle 63% | wait 0% | | | steal 0% | guest 0% | curf 2.59GHz | curscal ?% |
CPL | avg1 0.35 | | avg5 0.10 | avg15 0.13 | | | csw 7047 | | intr 3252 | | | numcpu 1 |
MEM | tot 991.5M | free 9.2M | cache 9.0M | | dirty 0.0M | buff 0.1M | slab 15.4M | | | | | |
SWP | tot 0.0M | free 0.0M | | | | | | | | | vmcom 1.4G | vmlim 495.8M |
PAG | scan 35266 | | | stall 0 | | | | | swin 0 | | | swout 0 |
DSK | xvda | busy 2% | read 830 | | write 74 | KiB/r 14 | KiB/w 13 | MBr/s 1.17 | | MBw/s 0.10 | avq 1.83 | avio 0.26 ms |
NET | transport | tcpi 641 | tcpo 358 | udpi 5 | udpo 5 | tcpao 2 | tcppo 2 | tcprs 0 | tcpie 0 | tcpor 1 | udpnp 0 | udpip 0 |
NET | network | ipi 646 | ipo 342 | | ipfrw 0 | deliv 646 | | | | | icmpi 0 | icmpo 0 |
NET | eth0 ---- | pcki 630 | pcko 326 | si 340 Kbps | so 95 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
NET | lo ---- | pcki 16 | pcko 16 | si 1 Kbps | so 1 Kbps | coll 0 | | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/1
6156 www-data www-data 1 0.22s 0.78s 103.0M 102.5M 3644K 12K -- - R 0 10% apache2
6152 www-data www-data 1 0.04s 0.56s -1280K -2256K 36K 12K -- - S 0 6% apache2
6452 www-data www-data 1 0.03s 0.49s -396K -1440K 0K 0K -- - S 0 5% apache2
7466 www-data www-data 1 0.10s 0.26s 54108K 55692K 2644K 0K -- - R 0 4% apache2
...
