Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Recently, seemingly out of nowhere the Apache2 daemon on our VPS has random issues with what looks to be forkbombing.

First log entries before things to go hell

[Tue Jun 25 23:07:18 2013] [error] [client 173.245.51.242] PHP Warning: Invalid argument supplied for foreach() in /var/www/libraries/joomla/access/access.php on line 409 
[Tue Jun 25 23:07:19 2013] [error] [client 108.162.224.23] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 

Logs prior to any issues show more PHP errors than I'd expect, developers who maintain the site didn't find any issues here

[Tue Jun 25 23:18:01 2013] [error] [client 103.22.200.24] PHP Notice: Trying to get property of non-object in /var/www/libraries/joomla/plugin/helper.php on line 123 
[Tue Jun 25 23:18:05 2013] [error] [client 173.245.51.242] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 
[Tue Jun 25 23:18:19 2013] [error] [client 103.22.200.63] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 
[Tue Jun 25 23:18:24 2013] [error] [client 103.22.200.24] PHP Notice: Trying to get property of non-object in /var/www/libraries/joomla/plugin/helper.php on line 123 
[Tue Jun 25 23:18:33 2013] [error] [client 173.245.53.153] PHP Notice: Trying to get property of non-object in /var/www/libraries/joomla/plugin/helper.php on line 123 
[Tue Jun 25 23:18:35 2013] [error] [client 108.162.225.139] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 
[Tue Jun 25 23:18:52 2013] [error] [client 108.162.231.144] PHP Notice: Trying to get property of non-object in /var/www/administrator/includes/application.php on line 276 
[Tue Jun 25 23:18:59 2013] [error] [client 108.162.231.144] PHP Notice: Undefined property: stdClass::$params in /var/www/administrator/includes/application.php on line 277 
[Tue Jun 25 23:19:05 2013] [error] [client 108.162.231.243] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 
[Tue Jun 25 23:19:22 2013] [error] [client 108.162.219.100] PHP Notice: Trying to get property of non-object in /var/www/libraries/joomla/plugin/helper.php on line 123 
[Tue Jun 25 23:19:39 2013] [error] [client 103.22.200.163] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 
[Tue Jun 25 23:19:38 2013] [error] [client 173.245.49.138] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/libraries/joomla/language/helper.php on line 55 
[Tue Jun 25 23:19:50 2013] [error] [client 103.22.200.24] PHP Notice: Trying to get property of non-object in /var/www/administrator/includes/application.php on line 276 
[Tue Jun 25 23:19:50 2013] [error] [client 103.22.200.24] PHP Notice: Undefined property: stdClass::$params in /var/www/administrator/includes/application.php on line 277 

I have seen this happen in htop, virtual memory gets exhausted and start seeing this

[Tue Jun 25 23:54:45 2013] [warn] child process 30976 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 30978 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 30979 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 30829 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 30830 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 32009 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 31929 still did not exit, sending a SIGTERM 
[Tue Jun 25 23:54:45 2013] [warn] child process 31037 still did not exit, sending a SIGTERM 

At this point everything slows to a crawl and system is basically unuseable

[Tue Jun 25 23:54:50 2013] [error] could not make child process 30976 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 30830 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 31929 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 30834 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 31050 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 30438 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 31052 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 32080 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 30838 exit, attempting to continue anyway 
[Tue Jun 25 23:54:50 2013] [error] could not make child process 32179 exit, attempting to continue anyway 

Restarting httpd fixed things although sometimes system is too slow and VM requires restart.

Some basics:

Linux version 2.6.32-358.11.1.el6.x86_64 (mockbuild@c6b7.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Jun 12 03:34:52 UTC 2013

[user@server ~]$ httpd -V
Server version: Apache/2.2.15 (Unix)
Server built:   May 16 2012 22:32:26
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

[user@server ~]$ php -v
PHP 5.3.3 (cli) (built: Jul 12 2013 20:35:47)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

[user@server ~]$ mysql -v
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 264224
Server version: 5.1.69 Source distribution

top - 00:52:46 up 19 days,  2:47,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 152 total,   1 running, 151 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1016516k total,   883812k used,   132704k free,    61112k buffers
Swap:  2064376k total,   157088k used,  1907288k free,   227368k cached
  • Server is set to defaults for max_childs and other mpm_worker values
  • Site is behind Cloudflare hence the few IP addresses in above logs
  • VM CPU is 1Ghz, 1GB RAM, no disk IO limits and none of my virtual disks are full
  • Attempted switch to mpm_prefork but had issues with MySQL support
  • PHP modules is nothing out of the ordinary
  • Site runs webmin as a front-end, it's role is front-end webserver, MTA (postfix),MDB (dovecot) and file server (proftpd)

I'm not sure what direction I should be going in to proceed, I can enable PHP debugging if it's something in Joomla that's causing the issue but my experience in that is limited so I'd prefer to know where to go first instance.

At present this has only happened 3 times in the last few months and although there is no real pattern it doesn't seem to be load related as it has happened late at night.

Any suggestions would be greatly appreciated.

share|improve this question
2  
What makes you think this is an Apache problem instead of a PHP problem, which seems far more likely? –  daff Jul 29 '13 at 15:45
    
Is it possible that your server could have been compromised? –  Tom O'Connor Jul 29 '13 at 16:53

1 Answer 1

You definitely have a PHP loop that is getting stuck. Check these files:

  • /var/www/libraries/joomla/access/access.php
  • /var/www/libraries/joomla/language/helper.php
  • /var/www/administrator/includes/application.php
share|improve this answer
    
I can excute these scripts directly so I'm thinking it's the content in which it's being executed, do you have any suggestions for how to find the loop? –  Grant Moritz Jul 31 '13 at 13:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.