1

This just started happening three weeks or so ago. The content of my website hasn't changed, it's just a phpBB forum using MySQL as a backend.

Nothing has changed in well over a year as far as content, pages served, etc. but recently, every two days or so, the server just shuts down and cannot be accessed at all (FTP, HTTP, MySQL), I have to notify my service provider to physically restart the machine.

I had thought it was tied to these SIGTERM errors I find in the logs but I found elsewhere that the SIGTERM is likely my provider restarting the server for me.

Problem is I have no idea how to fix these kinds of things or find the root cause as my skills in this area are lacking. My service provider has basically told me that they don't offer the kind of support I need for the package I have (VPS) and I that I pretty much have the keys to the whole thing and am on my own.

Anyone have any ideas what could be going on?

Apache/2.2.3 (CentOS)

20051115

Linux 2.6.18-028stab057.4 #1 SMP Fri Aug 1 10:47:59 MSD 2008 x86_6

Error log info (last activity seems to be at 5:01 with the service provider restarting (SIGTERM) at 9:17):

[Sat Oct 31 00:08:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 00:21:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 00:23:04 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 00:25:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 00:42:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 00:42:52 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 03:10:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 03:37:01 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 03:37:32 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 04:17:42 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 06:04:54 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 09:45:55 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 10:15:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 10:15:49 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 10:29:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 10:30:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 10:31:03 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 10:55:04 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 11:14:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 11:20:55 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 11:47:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 12:00:35 2009] [error] [client 208.53.131.15] File does not exist: /var/www/vhosts/default/htdocs/phpMyAdmin
[Sat Oct 31 12:07:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 13:01:07 2009] [error] [client 208.53.131.15] File does not exist: /var/www/vhosts/default/htdocs/phpmyadmin
[Sat Oct 31 13:10:31 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 13:59:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 14:31:37 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 14:31:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 14:31:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 14:35:20 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 14:38:54 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 14:54:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 15:24:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 15:32:17 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 15:40:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 15:41:48 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 15:42:00 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 16:17:52 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 16:32:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 16:39:13 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 16:39:16 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 16:39:21 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 16:55:28 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 17:04:00 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 17:04:04 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 17:06:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 17:20:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 18:58:26 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 18:58:27 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 18:58:28 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 18:58:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 18:58:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 19:23:00 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 20:23:04 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 20:23:14 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 20:33:13 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 22:06:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 22:22:13 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sat Oct 31 23:57:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 00:15:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 00:27:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 00:27:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 00:28:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 00:29:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 00:59:58 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 01:06:05 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 01:42:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 02:31:26 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 02:31:40 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 02:31:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 05:07:17 2009] [notice] SIGHUP received.  Attempting to restart
WARNING: MaxClients of 256 exceeds ServerLimit value of 10 servers,
 lowering MaxClients to 10.  To increase, please see the ServerLimit
 directive.
[Sun Nov 01 05:07:17 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 05:07:17 2009] [notice] Digest: done
[Sun Nov 01 05:07:19 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 05:07:19 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 05:07:19 2009] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Sun Nov 01 05:07:19 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 15:16:11 2009] [error] [client 127.0.0.1] Invalid method in request get
[Sun Nov 01 15:18:53 2009] [notice] caught SIGTERM, shutting down
[Sun Nov 01 15:19:27 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 15:19:27 2009] [notice] Digest: done
[Sun Nov 01 15:19:27 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 16:42:31 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:59:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:37:42 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:54 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:55 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:14:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:40:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:10:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:35:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:46 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:25:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:26:03 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:56:57 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:21:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:16 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sun Nov 01 22:25:17 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:18 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:20 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:21 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:46:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:52:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:58 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:00:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:09:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:23:49 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:27:14 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:36:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:54:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:13:23 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:39:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:52:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:53:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:40 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:23:01 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:43:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:45:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:46:27 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:23:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:37 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:48 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:35:33 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 04:46:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 05:01:07 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 05:01:09 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 05:01:09 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:11 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 09:17:58 2009] [notice] caught SIGTERM, shutting down
[Mon Nov 02 09:18:58 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Digest: generating secret for digest authentication ...
[Mon Nov 02 09:18:58 2009] [notice] Digest: done
[Mon Nov 02 09:18:58 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Mon Nov 02 09:32:34 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/

4 Answers 4

4

I'm not saying this is what's happening but based on my own experience as a CentOS admin, it's most likely runaway apache/php processes taking down the server. I've seen this numerous times on CentOS 5. It's frustrating because there's usually not a trace of what happened in the log files. The machine just grinds to a halt due to physical memory and swap being sucked up by apache/php processes. You would think linux memory management or some daemon would jump in and say "hey stop" but it doesn't. It'll let apache grind your system to a halt.

Having said that, to see what's happening you'll need something that can monitor and log resource usage. I like to use a program called atop. Atop is a lot like the top program but it also takes a snapshot of resource usage at defined intervals. It's pretty simple to install.

wget http://www.atcomputing.nl/Tools/atop/packages/atop-1.23.tar.gz 
tar -zxvf atop-1.23.tar.gz
cd atop-1.23 && make install

Open /etc/atop/atop.daily with a text editor and change INTERVAL=600 to INTERVAL=60

Run the command /etc/atop/atop.daily from a command prompt to start it. Wait a few minutes and run atop -r /var/log/atop/atop_20091118 with the correct date of course.

Hit the t key to go forward in time and T to go back. Next time your server crashes do this and check the MEM free and SWP free lines. If you have memory problems these will be in red. Also look for numerous httpd lines under CMD. If apache/php is your problem there'll be a bunch of them.

If this is the case, I recommend looking at you're MaxClients setting in httpd.conf. If set too high, apache will gladly eat all of your memory causing your machine to crash. Apache/php can easily eat 40-50MB/process. If you multiply 40mb x MaxClients you'll get a rough idea of how much memory apache can potentially use. MaxClients usually defaults to 150 on CentOS so apache can potentially use 6GB of memory by default. This doesn't include memory your system needs for itself and other processes to run. Try setting it to a more realistic value based on the amount of memory you have like 40 if you have 2G of memory and see if that helps. Also if you have KeepAlive On, set KeepAliveTimeout to a low number like 2 or 3.

In my opinion CentOS's apache/php compilation is a real pos that should never have seen the light of day. It's buggy and crash prone. If you run a serious site, I highly recommend compiling your own version of apache/php or even using one of the newer high performance webservers like lighttpd or nginx with fgci php.

1

This can be tricky to track down. The first thing I would check is 'dmesg' for signs of disk write failures, possible memory corruption or kernel errors related to failing hardware. Periodically check 'top' for processes that might be running amok, as well as 'free' to check for signs of memory starvation of excessive swap usage. If the VM has any kind of attach virtual serial console you may be able to get output or otherwise get to the virtual machine when it is inaccessible which would help to diagnose issues. Also don't check disk space 'df' just to make sure you aren't running out of space/swap since that could conceivably cause some strange behavior.

Also just as an aside if really nothing has changed and you aren't running different code / process load, you should keep an eye out for unrecognized processes (ps -auxf), or the existence of files in your directory structure that may indicate that your virtual host was compromised and you are just suffering the ill effects of being hacked, or even kernel modules (lsmod) that are out of the ordinary. Unless you rigorously maintain your site its always possible that it has succumbed to either a known vulnerability in a publicly accessible service or a brute force attack on remote management.

1

install chkrootkit on your machine and run that

have a look at what dmesg and /var/log/messages say - neither of these should be empty, if they are there is a problem

0

If all your services are vanishing (FTP, HTTP, etc) then your problem is on the host, not with Apache.

Being a VPS are you sure this isn't an intensive process running and effectively locking up the machine? Check cron for scheduled jobs:

crontab -l

Also start looking in your system logs:

/var/log/message etc

1
  • There aren't any cron jobs running as far as I know and all of the logs are empty except for the error and access logs in /var/logs/httpd. Also, am I correct in assuming that looking at the error log above that at [Sun Nov 01 05:07:19 2009] is where the crash happened and at [Sun Nov 01 15:18:53 2009] is where my host restarted the VPS for me? Do I really need to press my host on these issues? They swear it's not their problem no matter how many times I tell them I have changed nothing on the site in years.
    – Tom
    Nov 2, 2009 at 18:04

You must log in to answer this question.