Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When I run ps aux | grep /usr/local/apache/bin/httpd I get the following output.

www-data  9837  0.0  0.0  23112  1360 ?        S    Oct15   0:00 /usr/local/apache/bin/httpd -DSSL
www-data  9841  0.0  0.0  23112  1568 ?        S    Oct15   0:16 /usr/local/apache/bin/httpd -DSSL
www-data 29178  0.0  0.0  23112  1064 ?        S    Oct04   1:51 /usr/local/apache/bin/httpd -DSSL

What I find interesting though is that /usr/local/apache/bin/httpd doesn't exist. There is no /usr/local/apache/bin/ directory at all.

sudo cat /proc/9837/cmdline returns /usr/local/apache/bin/httpd -DSSL

When I run /proc/9837$ sudo /usr/local/apache/bin/httpd -DSSL I get:

/usr/local/apache/bin/httpd: command not found

When I run sudo ls -l /proc/9837/exe I get:

lrwxrwxrwx 1 www-data www-data 0 2012-10-17 02:06 /proc/9837/exe -> /usr/bin/perl

My question is, why is this happening? Shouldn't /proc/<pid>/cmdline and /proc/<pid>/exe be related? If this is not normal, what steps can I take to find out what caused it?

These processes remain even after running sudo /etc/init.d/apache2 stop

For reference, I'm running Ubuntu Ubuntu 10.04.4 LTS running Apache2 from the default apt repository.

sudo ls -l /proc/9837/fd outputs

lr-x------ 1 www-data www-data 64 2012-10-17 02:47 0 -> /dev/null
l-wx------ 1 www-data www-data 64 2012-10-17 02:47 1 -> pipe:[37796710]
l-wx------ 1 www-data www-data 64 2012-10-17 02:47 2 -> /var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 2012-10-17 02:47 3 -> socket:[37796725]
lr-x------ 1 www-data www-data 64 2012-10-17 02:47 4 -> pipe:[40055427]

stat /proc/9837/root returns

  File: `/proc/9837/root' -> `/'
  Size: 0           Blocks: 0          IO Block: 1024   symbolic link
Device: 3h/3d   Inode: 49853155    Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2012-10-17 02:07:00.240782014 -0400
Modify: 2012-10-17 02:06:43.860777313 -0400
Change: 2012-10-17 02:06:43.860777313 -0400
share|improve this question
Does /var/log/apache2/error.log actually exists? Is it possible that you have a perl program running as a wrapper to a chrooted Apache environment? You could check for a possible chroot() environment by inspecting /proc/[pid]/root. (by the way, this is a good reference) – Alexander Janssen Oct 17 '12 at 6:57
/proc/$PID/root looks like a symlink to /. /var/log/apache2/error.log is valid as well. – Citricguy Oct 17 '12 at 7:10

Smells fishy. Programs can override their $0. The program pretends to be Apache, but it's perl in fact.

Go to /proc/$PID/fd and have a look what files it's got opened, maybe this gives you a clue what's going on.

share|improve this answer
I added the output to the end of the question. Is it possible to retrieve the original command elsewhere if /proc/$PID/cmdline has been overwritten? – Citricguy Oct 17 '12 at 6:58
@Citricguy In fact, I'm already looking for that. I thought I've seen that before, but somehow I can't find it right know. Will keep looking. – Alexander Janssen Oct 17 '12 at 7:00

you said that you had installed apache2 from default repository. can you run a consistency check based on that package to check what is missing ?

It seems that someone / something / somehow deleted some of the apache files. But apache had been started before that, so it still remains in memory

share|improve this answer
The original/system apache2 is matches whats in Ubuntu's repositories and can restart without issue. This apache2 process loads from /usr/sbin/apache2 as it should. – Citricguy Oct 17 '12 at 8:50
up to which depth you have the directories ? /usr/local/apache ? /usr/local/apache/bin/ ? – Nikolaidis Fotis Oct 17 '12 at 9:04
Only as high as /usr/local/, but I don't think this is where Apache is located in Ubuntu 10.04 (Lucid). I believe the binaries live in /usr/sbin but I'll verify as this is a good point. – Citricguy Oct 17 '12 at 9:51

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.