3

When I run ps aux | grep /usr/local/apache/bin/httpd I get the following output.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data  9837  0.0  0.0  23112  1360 ?        S    Oct15   0:00 /usr/local/apache/bin/httpd -DSSL
www-data  9841  0.0  0.0  23112  1568 ?        S    Oct15   0:16 /usr/local/apache/bin/httpd -DSSL
www-data 29178  0.0  0.0  23112  1064 ?        S    Oct04   1:51 /usr/local/apache/bin/httpd -DSSL

What I find interesting though is that /usr/local/apache/bin/httpd doesn't exist. There is no /usr/local/apache/bin/ directory at all.

sudo cat /proc/9837/cmdline returns /usr/local/apache/bin/httpd -DSSL

When I run /proc/9837$ sudo /usr/local/apache/bin/httpd -DSSL I get:

/usr/local/apache/bin/httpd: command not found

When I run sudo ls -l /proc/9837/exe I get:

lrwxrwxrwx 1 www-data www-data 0 2012-10-17 02:06 /proc/9837/exe -> /usr/bin/perl

My question is, why is this happening? Shouldn't /proc/<pid>/cmdline and /proc/<pid>/exe be related? If this is not normal, what steps can I take to find out what caused it?

These processes remain even after running sudo /etc/init.d/apache2 stop

For reference, I'm running Ubuntu Ubuntu 10.04.4 LTS running Apache2 from the default apt repository.


sudo ls -l /proc/9837/fd outputs

lr-x------ 1 www-data www-data 64 2012-10-17 02:47 0 -> /dev/null
l-wx------ 1 www-data www-data 64 2012-10-17 02:47 1 -> pipe:[37796710]
l-wx------ 1 www-data www-data 64 2012-10-17 02:47 2 -> /var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 2012-10-17 02:47 3 -> socket:[37796725]
lr-x------ 1 www-data www-data 64 2012-10-17 02:47 4 -> pipe:[40055427]

stat /proc/9837/root returns

  File: `/proc/9837/root' -> `/'
  Size: 0           Blocks: 0          IO Block: 1024   symbolic link
Device: 3h/3d   Inode: 49853155    Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2012-10-17 02:07:00.240782014 -0400
Modify: 2012-10-17 02:06:43.860777313 -0400
Change: 2012-10-17 02:06:43.860777313 -0400
2
  • Does /var/log/apache2/error.log actually exists? Is it possible that you have a perl program running as a wrapper to a chrooted Apache environment? You could check for a possible chroot() environment by inspecting /proc/[pid]/root. (by the way, this is a good reference) Oct 17, 2012 at 6:57
  • /proc/$PID/root looks like a symlink to /. /var/log/apache2/error.log is valid as well.
    – Citricguy
    Oct 17, 2012 at 7:10

2 Answers 2

2

Smells fishy. Programs can override their $0. The program pretends to be Apache, but it's perl in fact.

Go to /proc/$PID/fd and have a look what files it's got opened, maybe this gives you a clue what's going on.

2
  • I added the output to the end of the question. Is it possible to retrieve the original command elsewhere if /proc/$PID/cmdline has been overwritten?
    – Citricguy
    Oct 17, 2012 at 6:58
  • @Citricguy In fact, I'm already looking for that. I thought I've seen that before, but somehow I can't find it right know. Will keep looking. Oct 17, 2012 at 7:00
0

you said that you had installed apache2 from default repository. can you run a consistency check based on that package to check what is missing ?

It seems that someone / something / somehow deleted some of the apache files. But apache had been started before that, so it still remains in memory

3
  • The original/system apache2 is matches whats in Ubuntu's repositories and can restart without issue. This apache2 process loads from /usr/sbin/apache2 as it should.
    – Citricguy
    Oct 17, 2012 at 8:50
  • up to which depth you have the directories ? /usr/local/apache ? /usr/local/apache/bin/ ? Oct 17, 2012 at 9:04
  • Only as high as /usr/local/, but I don't think this is where Apache is located in Ubuntu 10.04 (Lucid). I believe the binaries live in /usr/sbin but I'll verify as this is a good point.
    – Citricguy
    Oct 17, 2012 at 9:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .