0

For a few months now I've been experiencing random crashes on my 2GB RAM VPS Linode server. Someone convinced me to switch to self-managed hosting and I for some reason thought I could handle it despite having no experience in the field. I'm going to provide as much information as possible to help:

Site in question is a Wordpress/Woocommerce site, it does have quite a lot of plugins so I'm aware that's not ideal but they are mostly required. Concurrent users is usually around 10, doesn't really go over 20. I've noticed that I do tend to get a lot of random bots coming to site that sometimes tends to cause spike traffic, but even that should be manageable I feel given I have a CDN and w3cache set up. The site runs smoothly with no problem 99% of the time. I've even run a load test using loadimpact.com and the server handled it fine.

Here are some logs from latest crash, maybe someone can get some clues:

Error Log

[Fri Jun 19 03:06:07 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting

MYSQL LOG

150619 03:06:09 mysqld_safe Number of processes running now: 0
150619 03:06:12 mysqld_safe mysqld restarted
150619  3:06:15 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150619  3:06:15 [Note] Plugin 'FEDERATED' is disabled.
150619  3:06:15 InnoDB: The InnoDB memory heap is disabled
150619  3:06:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150619  3:06:15 InnoDB: Compressed tables use zlib 1.2.3
150619  3:06:15 InnoDB: Using Linux native AIO
150619  3:06:16 InnoDB: Initializing buffer pool, size = 320.0M
150619  3:06:16 InnoDB: Completed initialization of buffer pool
150619  3:06:17 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150619  3:06:17  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.

SYSLOG MESSAGES

Jun 19 03:06:07 panel kernel: php invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 19 03:06:07 panel kernel: php cpuset=/ mems_allowed=0
Jun 19 03:06:07 panel kernel: CPU: 1 PID: 24844 Comm: php Not tainted 4.0.5-x86_64-linode58 #1
Jun 19 03:06:07 panel kernel: 0000000000000000 ffff880055fac740 ffffffff8193b626 ffff880055fac100
Jun 19 03:06:07 panel kernel: ffffffff819360a4 0100000000000400 ffff88007fc17000 0000000000000020
Jun 19 03:06:07 panel kernel: ffff88007fc17000 0000000000000000 0000000000000000 0000000000000000
Jun 19 03:06:07 panel kernel: Call Trace:
Jun 19 03:06:07 panel kernel: [<ffffffff8193b626>] ? dump_stack+0x40/0x50
Jun 19 03:06:07 panel kernel: [<ffffffff819360a4>] ? dump_header.isra.10+0x78/0x1e3
Jun 19 03:06:07 panel kernel: [<ffffffff81941bb6>] ? _raw_spin_unlock_irqrestore+0x2e/0x3f
Jun 19 03:06:07 panel kernel: [<ffffffff81172f4a>] ? oom_kill_process+0xbe/0x380
Jun 19 03:06:07 panel kernel: [<ffffffff810dc57d>] ? has_ns_capability_noaudit+0x13/0x1b
Jun 19 03:06:07 panel kernel: [<ffffffff8117369a>] ? __out_of_memory+0x43d/0x47d
Jun 19 03:06:07 panel kernel: [<ffffffff8117381b>] ? out_of_memory+0x52/0x67
Jun 19 03:06:07 panel kernel: [<ffffffff811777e5>] ? __alloc_pages_nodemask+0x708/0x846
Jun 19 03:06:07 panel kernel: [<ffffffff811a4a7d>] ? alloc_pages_current+0xb2/0xcf
Jun 19 03:06:07 panel kernel: [<ffffffff8117205d>] ? filemap_fault+0x26f/0x394
Jun 19 03:06:07 panel kernel: [<ffffffff8118f78d>] ? __do_fault+0x3f/0x79
Jun 19 03:06:07 panel kernel: [<ffffffff81193009>] ? handle_mm_fault+0x370/0xd80
Jun 19 03:06:07 panel kernel: [<ffffffff811426a1>] ? __audit_syscall_exit+0x208/0x224
Jun 19 03:06:07 panel kernel: [<ffffffff8104219d>] ? __do_page_fault+0x321/0x37b
Jun 19 03:06:07 panel kernel: [<ffffffff81943f08>] ? page_fault+0x28/0x30
Jun 19 03:06:07 panel kernel: Mem-Info:
Jun 19 03:06:07 panel kernel: Node 0 DMA per-cpu:
Jun 19 03:06:07 panel kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 19 03:06:07 panel kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jun 19 03:06:07 panel kernel: Node 0 DMA32 per-cpu:
Jun 19 03:06:07 panel kernel: CPU    0: hi:  186, btch:  31 usd:   3
Jun 19 03:06:07 panel kernel: CPU    1: hi:  186, btch:  31 usd:   0
Jun 19 03:06:07 panel kernel: active_anon:348396 inactive_anon:116687 isolated_anon:32
Jun 19 03:06:07 panel kernel: active_file:351 inactive_file:528 isolated_file:0
Jun 19 03:06:07 panel kernel: unevictable:0 dirty:0 writeback:6 unstable:0
Jun 19 03:06:07 panel kernel: free:3411 slab_reclaimable:8339 slab_unreclaimable:5537
Jun 19 03:06:07 panel kernel: mapped:169339 shmem:168953 pagetables:8887 bounce:0
Jun 19 03:06:07 panel kernel: free_cma:0
Jun 19 03:06:07 panel kernel: Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:2880kB inactive_anon:3140kB active_file:0kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:2676kB shmem:2664kB slab_reclaimable:44kB slab_unreclaimable:164kB kernel_stack:0kB pagetables:132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
Jun 19 03:06:07 panel kernel: lowmem_reserve[]: 0 1972 1972 1972
Jun 19 03:06:07 panel kernel: Node 0 DMA32 free:5736kB min:5656kB low:7068kB high:8484kB active_anon:1390704kB inactive_anon:463608kB active_file:1488kB inactive_file:1892kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:2080768kB managed:2023584kB mlocked:0kB dirty:0kB writeback:24kB mapped:675052kB shmem:673148kB slab_reclaimable:33312kB slab_unreclaimable:21984kB kernel_stack:4304kB pagetables:35416kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 19 03:06:07 panel kernel: lowmem_reserve[]: 0 0 0 0
Jun 19 03:06:07 panel kernel: Node 0 DMA: 4*4kB (UM) 5*8kB (U) 8*16kB (UM) 4*32kB (UM) 3*64kB (UE) 10*128kB (EM) 6*256kB (UEM) 1*512kB (U) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
Jun 19 03:06:07 panel kernel: Node 0 DMA32: 97*4kB (UEMR) 10*8kB (MR) 0*16kB 0*32kB 1*64kB (R) 1*128kB (R) 0*256kB 0*512kB 1*1024kB (R) 0*2048kB 1*4096kB (R) = 5780kB
Jun 19 03:06:07 panel kernel: 174507 total pagecache pages
Jun 19 03:06:07 panel kernel: 4598 pages in swap cache
Jun 19 03:06:07 panel kernel: Swap cache stats: add 1673544, delete 1668946, find 6783627/7234079
Jun 19 03:06:07 panel kernel: Free swap  = 0kB
Jun 19 03:06:07 panel kernel: Total swap = 262140kB
Jun 19 03:06:07 panel kernel: 524191 pages RAM
Jun 19 03:06:07 panel kernel: 0 pages HighMem/MovableOnly
Jun 19 03:06:07 panel kernel: 14317 pages reserved
Jun 19 03:06:07 panel kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jun 19 03:06:07 panel kernel: [ 1541]     0  1541     2710        0       9       2      127         -1000 udevd
Jun 19 03:06:07 panel kernel: [ 2129]     0  2129     2295       34       8       2       90             0 dhclient
Jun 19 03:06:07 panel kernel: [ 2175]     0  2175     6402        0      12       2       75         -1000 auditd
Jun 19 03:06:07 panel kernel: [ 2238]     0  2238    60749        0      21       3      214             0 rsyslogd
Jun 19 03:06:07 panel kernel: [ 2264]    25  2264    57836        0      44       2     4143             0 named
Jun 19 03:06:07 panel kernel: [ 2295]    81  2295     5373        0      14       2       63             0 dbus-daemon
Jun 19 03:06:07 panel kernel: [ 2350]   495  2350    80168        0      26       5       86             0 memcached
Jun 19 03:06:07 panel kernel: [ 2368]     0  2368    16569       11      34       2      166         -1000 sshd
Jun 19 03:06:07 panel kernel: [ 2662]     0  2662     4943       23      19       5       54             0 dovecot
Jun 19 03:06:07 panel kernel: [ 2665]   498  2665    10600        0      41       6      144             0 pop3-login
Jun 19 03:06:07 panel kernel: [ 2666]   498  2666    10639        0      41       6      153             0 imap-login
Jun 19 03:06:07 panel kernel: [ 2667]    97  2667     3369        0      18       5       49             0 anvil
Jun 19 03:06:07 panel kernel: [ 2668]     0  2668     3402        0      18       5       78             0 log
Jun 19 03:06:07 panel kernel: [ 2671]   498  2671    10636       18      41       6      120             0 pop3-login
Jun 19 03:06:07 panel kernel: [ 2672]     0  2672     4145        0      19       5      282             0 config
Jun 19 03:06:07 panel kernel: [ 2673]   498  2673    10635        0      42       6      145             0 imap-login
Jun 19 03:06:07 panel kernel: [ 2683]    47  2683    18053       15      49       7      136             0 exim
Jun 19 03:06:07 panel kernel: [ 2807]     0  2807    43014      217     108       6    12952             0 /usr/local/cpan
Jun 19 03:06:07 panel kernel: [ 2895]     0  2895    43249     4306      97       6     9070             0 spamd child
Jun 19 03:06:07 panel kernel: [ 2952]   501  2952     4042        0      14       4       33             0 newrelic-daemon
Jun 19 03:06:07 panel kernel: [ 2953]     0  2953    28675        0      16       2       52             0 abrtd
Jun 19 03:06:07 panel kernel: [ 2954]   501  2954    60989      123      29       4      544             0 newrelic-daemon
Jun 19 03:06:07 panel kernel: [ 2970]     0  2970    28181        0      23       4       65             0 abrt-dump-oops
Jun 19 03:06:07 panel kernel: [ 2981]     0  2981    34674        9      23       2      160             0 pure-ftpd
Jun 19 03:06:07 panel kernel: [ 2983]     0  2983    34047        7      27       2      114             0 pure-authd
Jun 19 03:06:07 panel kernel: [ 2995]     0  2995    29217        8      16       2      148             0 crond
Jun 19 03:06:07 panel kernel: [ 3009]     0  3009     5291       11      14       2       34             0 atd
Jun 19 03:06:07 panel kernel: [ 3521]     0  3521    35818      389      90       6     6493             0 cpsrvd (SSL) - 
Jun 19 03:06:07 panel kernel: [ 3760]     0  3760    10172      158      34       4     1275             0 queueprocd - wa
Jun 19 03:06:07 panel kernel: [ 3832]     0  3832    28548        0      66       5     4769             0 dnsadmin - serv
Jun 19 03:06:07 panel kernel: [ 3858]     0  3858    21764      452      54       5     3245             0 tailwatchd
Jun 19 03:06:07 panel kernel: [ 3934]     0  3934    29015        2      65       5     5416             0 cpdavd - accept
Jun 19 03:06:07 panel kernel: [ 3964]     0  3964     7951       11      27       4     1315             0 cpanellogd - sl
Jun 19 03:06:07 panel kernel: [ 4002]     0  4002     1035        1       9       4       19             0 agetty
Jun 19 03:06:07 panel kernel: [ 4004]     0  4004     1031        0       9       4       17             0 mingetty
Jun 19 03:06:07 panel kernel: [ 4006]     0  4006     1031        0       9       4       18             0 mingetty
Jun 19 03:06:07 panel kernel: [ 4008]     0  4008     1031        0       9       4       18             0 mingetty
Jun 19 03:06:07 panel kernel: [ 4011]     0  4011     2709        0       9       2      128         -1000 udevd
Jun 19 03:06:07 panel kernel: [ 4012]     0  4012     2709        0       9       2      128         -1000 udevd
Jun 19 03:06:07 panel kernel: [ 4013]     0  4013     1031        0       9       4       18             0 mingetty
Jun 19 03:06:07 panel kernel: [ 4015]     0  4015     1031        0       9       4       18             0 mingetty
Jun 19 03:06:07 panel kernel: [ 4017]     0  4017     1031        0       8       4       17             0 mingetty
Jun 19 03:06:07 panel kernel: [16053]     0 16053    18080      444      59       5      138             0 httpd
Jun 19 03:06:07 panel kernel: [19702]     0 19702     2883        1      15       4       71             0 mysqld_safe
Jun 19 03:06:07 panel kernel: [19964]   497 19964   393396    49834     210       7    17743             0 mysqld
Jun 19 03:06:07 panel kernel: [22735]     0 22735     3368       26      17       5        0             0 ssl-params
Jun 19 03:06:07 panel kernel: [24373]     0 24373    20166     2064      63       5        9             0 leechprotect
Jun 19 03:06:07 panel kernel: [24374]    99 24374    18247      625      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24375]    99 24375    18259      641      48       5      127             0 httpd
Jun 19 03:06:07 panel kernel: [24378]    99 24378    18283      650      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24379]    99 24379    18207      582      48       5      131             0 httpd
Jun 19 03:06:07 panel kernel: [24380]    99 24380    18294      651      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24383]    99 24383    18251      632      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24388]    99 24388    18275      651      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24395]    99 24395    18232      610      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24438]     0 24438     4252       51      12       2        0             0 anacron
Jun 19 03:06:07 panel kernel: [24457]    99 24457    18249      629      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24460]    99 24460    18243      622      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24559]    99 24559    18234      611      48       5      129             0 httpd
Jun 19 03:06:07 panel kernel: [24688]    99 24688    18283      646      48       5      131             0 httpd
Jun 19 03:06:07 panel kernel: [24757]    97 24757     7930      113      30       5        0             0 auth
Jun 19 03:06:07 panel kernel: [24838]   507 24838    84744    21633     134       6        0             0 php
Jun 19 03:06:07 panel kernel: [24840]   507 24840    83776    20878     133       6        0             0 php
Jun 19 03:06:07 panel kernel: [24841]   507 24841    82483    18820     128       7        0             0 php
Jun 19 03:06:07 panel kernel: [24844]   507 24844    82937    20101     130       7        0             0 php
Jun 19 03:06:07 panel kernel: [24845]   507 24845    82227    18788     129       7        0             0 php
Jun 19 03:06:07 panel kernel: [24846]   507 24846    82291    19040     129       6        0             0 php
Jun 19 03:06:07 panel kernel: [24847]   507 24847    82887    19792     131       7        0             0 php
Jun 19 03:06:07 panel kernel: [24848]   507 24848    82934    20094     130       7        0             0 php
Jun 19 03:06:07 panel kernel: [24850]   507 24850    82894    19887     130       6        0             0 php
Jun 19 03:06:07 panel kernel: [24852]   507 24852    82548    19709     129       6        0             0 php
Jun 19 03:06:07 panel kernel: [24859]   507 24859    82803    19724     131       6        0             0 php
Jun 19 03:06:07 panel kernel: [24862]    99 24862    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24866]   507 24866    81465    17804     126       6        0             0 php
Jun 19 03:06:07 panel kernel: [24875]    99 24875    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24876]    99 24876    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24877]   507 24877    79415    14127     121       6        0             0 php
Jun 19 03:06:07 panel kernel: [24878]   507 24878    79221    13720     118       7        0             0 php
Jun 19 03:06:07 panel kernel: [24885]    99 24885    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24886]    99 24886    18182      523      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24887]    99 24887    18182      523      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24888]    99 24888    18182      523      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24892]   507 24892    77328     9824     111       6        0             0 php
Jun 19 03:06:07 panel kernel: [24898]    99 24898    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24899]    99 24899    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24900]    99 24900    18182      513      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24901]    99 24901    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24902]    99 24902    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24903]    99 24903    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24904]    99 24904    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24905]    99 24905    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24906]   507 24906    75260     6007     103       6        0             0 php
Jun 19 03:06:07 panel kernel: [24907]   507 24907    61544    14697     114       6        0             0 php
Jun 19 03:06:07 panel kernel: [24908]   507 24908    75724     6894     105       6        0             0 php
Jun 19 03:06:07 panel kernel: [24909]   507 24909    74348     5097     101       6        0             0 php
Jun 19 03:06:07 panel kernel: [24910]   507 24910    75746     6708     104       7        0             0 php
Jun 19 03:06:07 panel kernel: [24911]   507 24911    75746     6630     104       6        0             0 php
Jun 19 03:06:07 panel kernel: [24912]   507 24912    75746     6705     104       6        0             0 php
Jun 19 03:06:07 panel kernel: [24913]   507 24913    76743     8656     108       6        0             0 php
Jun 19 03:06:07 panel kernel: [24922]   507 24922    73138     3847      99       7        0             0 php
Jun 19 03:06:07 panel kernel: [24923]   507 24923    72900     3513      96       6        0             0 php
Jun 19 03:06:07 panel kernel: [24925]   507 24925    72704     3279      98       6        0             0 php
Jun 19 03:06:07 panel kernel: [24926]   507 24926    72856     3395      98       6        0             0 php
Jun 19 03:06:07 panel kernel: [24927]    99 24927    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24928]    99 24928    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24929]    99 24929    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24930]    99 24930    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24931]    99 24931    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24932]    99 24932    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24933]    99 24933    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24934]    99 24934    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24935]    99 24935    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24936]   507 24936    72900     3454      98       7        0             0 php
Jun 19 03:06:07 panel kernel: [24937]   507 24937    72900     3512     100       6        0             0 php
Jun 19 03:06:07 panel kernel: [24938]   507 24938    72898     3510      99       6        0             0 php
Jun 19 03:06:07 panel kernel: [24940]    99 24940    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24941]    99 24941    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24942]    99 24942    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24944]   507 24944    72709     3228      98       6        0             0 php
Jun 19 03:06:07 panel kernel: [24945]   507 24945    72707     3210      96       6        0             0 php
Jun 19 03:06:07 panel kernel: [24946]   507 24946    72505     2756      97       6        0             0 php
Jun 19 03:06:07 panel kernel: [24947]   507 24947    72770     3394      99       6        0             0 php
Jun 19 03:06:07 panel kernel: [24948]   507 24948    72898     3511      98       7        0             0 php
Jun 19 03:06:07 panel kernel: [24950]   507 24950    72920     3491      99       6        0             0 php
Jun 19 03:06:07 panel kernel: [24952]    99 24952    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24953]    99 24953    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24954]    99 24954    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24956]   507 24956    72273     2332      95       6        0             0 php
Jun 19 03:06:07 panel kernel: [24959]   507 24959    71635     1659      96       6        0             0 php
Jun 19 03:06:07 panel kernel: [24960]   507 24960    72358     2601      97       7        0             0 php
Jun 19 03:06:07 panel kernel: [24961]   507 24961    72431     2797      96       6        0             0 php
Jun 19 03:06:07 panel kernel: [24962]   507 24962    72565     2478      97       6        0             0 php
Jun 19 03:06:07 panel kernel: [24963]    99 24963    18149      480      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24964]   507 24964    72505     2727      98       6        0             0 php
Jun 19 03:06:07 panel kernel: [24965]   507 24965    71871     1883      95       6        0             0 php
Jun 19 03:06:07 panel kernel: [24987]    99 24987    18149      467      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24988]    99 24988    18149      472      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24989]    99 24989    18149      467      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24990]    99 24990    18149      479      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24991]    99 24991    18149      477      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24992]   507 24992    24451       62      72       6        0             0 php
Jun 19 03:06:07 panel kernel: [24993]    99 24993    18149      478      48       5      132             0 httpd
Jun 19 03:06:07 panel kernel: [24994]    99 24994     2998       13      14       4        0             0 suphp
Jun 19 03:06:07 panel kernel: [24995]   507 24995     5810        9      12       4        0             0 php
Jun 19 03:06:07 panel kernel: [24996]   507 24996     3880        9      10       4        0             0 php
Jun 19 03:06:07 panel kernel: [24998]   507 24998     6373       16      15       5        0             0 php
Jun 19 03:06:07 panel kernel: [24999]   507 24999     2767        1       3       3        0             0 php
Jun 19 03:06:07 panel kernel: [25000]     0 25000    18080      432      47       5      135             0 httpd
Jun 19 03:06:07 panel kernel: Out of memory: Kill process 19964 (mysqld) score 117 or sacrifice child
Jun 19 03:06:07 panel kernel: Killed process 19964 (mysqld) total-vm:1573584kB, anon-rss:199336kB, file-rss:0kB
Jun 19 03:06:07 panel kernel: php invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Jun 19 03:06:07 panel kernel: php cpuset=/ mems_allowed=0

ANd here's my apache settings:

Starts Servers 2 
Minimum Spare Servers 6 
Maximum Spare Servers 12 
Server Limit 256 
Max Clients 50 
Max Requests Per Child 3000 
Keep-Alive Off

Mysql my.cnf settings:

[mysqld]
innodb_file_per_table=1
open_files_limit=1024000
max_connections=75
key_buffer=32M
max_allowed_packet=268435456
thread_stack=128K
table_cache=32
innodb_buffer_pool_size=320M
default-storage-engine=MyISAM
query_cache_size=8M
tmp_table_size=16M
max_heap_table_size=16M
thread_cache_size=4
table_open_cache=32

Please help, I've tried my best to track this thing down but I can't really pinpoint what is causing it.

BTW some additional random tidbits:

-Some of the pages on my site have LOTS of pictures (like over 100), not sure if these count as extra clients/requests and affecting performance somehow.

-My Wordpress memory limit is set around 400MB.

-Mysql database for this site is around 175MB. 91MB of it coming from WP_Options.

-Mysqltuner.pl keeps telling me to increase tmp_table_size and max_heap_table_size even though I've set them as high as 200M!

-Here's a screenshot of my htop right now. Seems mysql is constantly using 40% of memory and Swap memory is almost full for some reason. Is this normal? :

enter image description here

UPDATE: I've lowered Max Clients to 35 but still getting some weird mysql crash. Newrelic error message is Unknown storage engine 'InnoDB' (my mysqltuner.pl is saying InnoDB is disabled but that it's default storage engine). My site is telling me to reinstall Wordpress like a fresh install, and I have to restart mysql or the server for that to go away.

  • 1
    Too many concurrent http/php processes for the given 2GB amount of ram. Get more ram or lower apache server max clients or optimize your php to consume less memory. Unfortunately when oom, mysql will get killed because it consumes more memory than any single php process. – Dan Jun 19 '15 at 16:07
  • The crash report said too few clients so I was going to increase it, but I'll try reducing the max clients based on your recommendation and see. Thank you. Btw, what would happen in the above case if I had lower max client? Does everything get served slower or would it still end up crashing? – Champy Jun 19 '15 at 16:37
  • 1
    Thing is... the moment you are out of memory, it goes downhill really fast. Reduce maxclients (or get more memory) untill you fit into memory and have no more crashes. If you cannot process client requests fast enough, they will be delayed. – Dan Jun 20 '15 at 19:48
  • Reduced Max Clients to 35 but getting some weird mysql crash telling me to reinstall Wordpress (is it a corrupt databse or something) and I have to restart mysql or server to fix it temporarily. NEwrelic error message is "Unknown storage engine 'InnoDB'". Btw my mysqltuner.pl is saying that InnobDB is disabled but that InnoDB Storage engine is the default storage engine. – Champy Jun 21 '15 at 10:37
  • Uh oh. You should mysqlcheck all databases, see which ones are corrupt and fix or restore them from backups. A badly damaged mysql database can crash the mysql server. – Dan Jun 21 '15 at 12:03
0

For some reason you are running into the maxclients parameter.

Do you have any more information as of amount of requests per second when the server goes down? It looks like your swap is being filled completely, causing the server to crash. Do you have any info on which components are using how much memory?

  • Should I increase the maxclients parameter? I've heard this can cause more issues but perhaps for my setup I can afford to increase it? I'm not sure about the amount of requests per second during crash, is it possible to retroactively check somehow? – Champy Jun 19 '15 at 15:48
  • Don't increase them. Think of it this way. For every Apache child process, it's going to use RAM. Let's say that each Apache process is taking up 30 MB of RAM. Right now you're allowing a maximum of 50 child processes. If you have 50 child processes at once, that's roughly 1.5GB RAM. Add on let's say 200 MB for MySQL and then another 300 MB for the rest of the system... you're out of memory. So bumping that MaxClients number up is just going to allow the system to spawn more children when you hardly have the resources to handle your current config. So no, do not increase the MaxClients. – Safado Jun 19 '15 at 18:49
  • Can you get me an output of your my.cnf and do you have mysql slow query log enabled? Might want to take a look at it, as it seems to be creating temp tables filling up all your memory. Looking at the processlist of mysql might also give more info. – Exsisto Jun 20 '15 at 19:07
  • I shared my.cnf settings in the OP. The mysqltuner.pl recommendations is telling me to increase some of those temp table settings it seems though: query_cache_size (> 8M) tmp_table_size (> 16M) max_heap_table_size (> 16M) table_open_cache (> 32) – Champy Jun 21 '15 at 10:33
  • Mysqltuner.pl keeps asking for bigger numbers in these fields no matter how much I increase it. Is this normal behavior? – Champy Jun 22 '15 at 13:22
0

An easy way to check the memory usage on your server is to install a tool called htop. htop will give you the vitals of your server on a per-process basis. In your case where you want to check memory, you could load htop and hit M to sort by Memory (or > to pick a category).

Once you find out which of your applications is using the most memory(I'm guessing Apache, considering you reach Max clients and triggered OOM killer all at the same time), then you can fine tune it. I don't doubt you have a capacity problem here, though. As shown in your logs, you've reached your Max Clients for Apache, but you can't safely increase that right now because you're unsure of your RAM. This is affecting your customers. Get a good set of monitoring tools and keep on top of it.

Also, it might not hurt to check your Apache access logs at the time of the crash. You say you typically don't get more than 20 concurrent users, yet you reached your max limit of 50 Apache processes. If this was legitimate traffic, then you'll need better tune your server and possibly upgrade resources. If it wasn't, then maybe try and find a way to filter the garbage (easier said than done).

  • I have htop setup but I don't really know what to look out for. Here's a current screenshot, although I guess I have to somehow catch it while it's in the process of crashing for it to be useful?: i.imgur.com/Nw9kvVg.jpg I checked the raw access logs and there were about 15-20 different pages being requested over the span of ~10 seconds. One time I got this attack listed here (blogseye.com/vulnerable-plugins) and one person trying to access all those files, even though they weren't on my server, crashed it. – Champy Jun 19 '15 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.