Apache Server overload / Fake Processes?

Question

My webspace provider is telling me that there are fake processes coming from my space and thus he has shut it down. So all my domains are also unreachable at the moment.

He doesn't really know what to do and the only info he gave me is this:

--<PROZESS-AUSZUG>--

web277 7111 0.0 0.0 36192 2816 ? S 18:55 0:00 /usr/sbin/apache2 -k start
web277 7319 0.0 0.0 36192 2912 ? S 18:56 0:00 /usr/sbin/apache2 -k start
web277 7625 0.5 0.1 58800 13204 ? S 18:58 0:10 /usr/sbin/apache2 -k start
web277 7647 0.0 0.0 36192 2916 ? S 18:58 0:00 /usr/sbin/apache2 -k start
web277 9801 4.7 41.6 3572212 3403664 ? S 19:01 1:29 /usr/sbin/apache2 -k start
web277 9809 0.2 0.1 85196 11940 ? S 19:01 0:04 /usr/sbin/apache2 -k start
web277 9831 0.0 0.0 36192 2908 ? S 19:01 0:00 /usr/sbin/apache2 -k start
web277 10617 0.2 0.0 54120 7992 ? S 19:05 0:03 /usr/sbin/apache2 -k start
web277 10639 0.0 0.0 36192 2900 ? S 19:05 0:00 /usr/sbin/apache2 -k start
web277 10645 0.0 0.0 36484 1992 ? S 19:05 0:00 /usr/sbin/apache2 -k start
web277 10892 0.2 0.0 53300 7880 ? S 19:06 0:03 /usr/sbin/apache2 -k start
web277 10914 0.0 0.0 36192 2908 ? S 19:06 0:00 /usr/sbin/apache2 -k start
web277 11606 0.1 0.1 85216 12160 ? S 19:10 0:02 /usr/sbin/apache2 -k start
web277 11634 0.0 0.0 36192 2928 ? S 19:10 0:00 /usr/sbin/apache2 -k start
web277 15084 0.0 0.0 36192 3308 ? S 19:18 0:00 /usr/sbin/apache2 -k start

--<PROZESS-AUSZUG>--

Any idea guys/gals?

Answer 1

Most of that looks pretty normal.

What he has given you is the output of ps aux so I've added the header to help explain the fields:

USER   PID  %CPU  %MEM     VSZ     RSS TTY  STAT START   TIME COMMAND
web277 9801  4.7  41.6 3572212 3403664 ?    S    19:01   1:29 /usr/sbin/apache2 -k start

The only process that stands out for me is the one above. It is using 41% of the memory available to the machine and 4.7% of the available CPU (which is a lot for a single Apache process). It has used 1 minute and 29 seconds of CPU time, which is a lot for a single Apache process. I strongly suspect this is what your hosting provider is complaining about.

It looks to me like that particular Apache process is stuck in an infinite loop. You could kill it to temporarily solve your problem and get your sites back online (sudo kill 9801) but it might be better to find out what it is doing first so you can stop the problem happening again.

If your hosting provider has ExtendedStatus on set in his Apache config, ask him to give you the line from /server-status that corresponds to PID 9801. This will include the request URI which should enable you to reproduce this problem whenever you want.

If you know what the application is doing when it gets that URI, find the infinite loop (assuming that my guess about that being the problem is correct) and fix it.

If that isn't enough to solve it, you can also use sudo strace -p 9801 to find out exactly what that process is doing right now. This command usually produces a lot of output and interpreting that output is the subject of an entire book, not an answer here on ServerFault.

Answer 2

You may want to look into this vulnerability

http://www.claudiokuenzler.com/blog/206/another-timthumb-wordpress-hack-external-upload-httpd-process

especially if you can't find a file on your server at /usr/sbin/apache2, as the scripts can fake the name that shows up.