Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running Apache 2.2.17 with the peruser MPM on FreeBSD 8.2-RC1 on Amazon's EC2 (so it's XEN). It was installed from ports.

My problem is that, although Apache is running, listening for, and accepting connections, it doesn't actually respond to any or show them in the log at all.

If I telnet to the port it's listening on and type out an HTTP request:

GET / HTTP/1.1
Host: asdfasdf

And hit enter a couple of times, it just sits there... Nothing. No response requesting with a browser either. There doesn't appear to be anything helpful in the error log:

[Sun Jan 09 16:56:24 2011] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Sun Jan 09 16:56:25 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 09 16:56:25 2011] [notice] Digest: done
[Sun Jan 09 16:56:25 2011] [notice] Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 

The access log stays empty:

root:/var/log# wc httpd-access.log 
       0       0       0 httpd-access.log
root:/var/log#

I've tried with accf_http and accf_data both enabled and disabled, and with both the stock configuration and my customized config. I also tried uninstalling apache22-peruser-mpm and just installing straight apache22... Still no luck. I tried removing all of the LoadModule lines from httpd.conf and just re-enabled the ones that were necessary to parse the config. Ended up with only the following loaded:

root:/usr/local/etc/apache22# /usr/local/sbin/apachectl -M
Loaded Modules:
 core_module (static)
 mpm_peruser_module (static)
 http_module (static)
 so_module (static)
 authz_host_module (shared)
 log_config_module (shared)
 alias_module (shared)
Syntax OK
root:/usr/local/etc/apache22#

Same results.

Apache is definitely what's listening on port 80:

root:/usr/local/etc/apache22# sockstat -4 | grep httpd
root     httpd      43789 3  tcp4 6 *:80                  *:*
root     httpd      43789 4  tcp4   *:*                   *:*
root:/usr/local/etc/apache22#

And I know it's not a firewall issue as there is nothing running locally, and connecting from the local box to 127.0.0.1:80 results in the same issue.

Does anyone have any idea what's going on? Why it would be doing this? I've exhausted all of my debugging expertise. :/

Thanks for any suggestions!

EDIT: As per Phil Hollenback's suggestion, I ran a trace on the system calls. This is what I found through both ktrace and truss.

48443 httpd    CALL  select(0,0,0,0,0xbf7feab4)
48443 httpd    RET   select 0
48443 httpd    CALL  gettimeofday(0xbf7feae4,0)
48443 httpd    RET   gettimeofday 0
48443 httpd    CALL  wait4(0xffffffff,0xbf7fea98,WNOHANG|WUNTRACED,0)
48443 httpd    RET   wait4 -1 errno 10 No child processes
48443 httpd    CALL  select(0,0,0,0,0xbf7feab4)
48443 httpd    RET   select 0
48443 httpd    CALL  gettimeofday(0xbf7feae4,0)
48443 httpd    RET   gettimeofday 0
48443 httpd    CALL  wait4(0xffffffff,0xbf7fea98,WNOHANG|WUNTRACED,0)
48443 httpd    RET   wait4 -1 errno 10 No child processes
48443 httpd    CALL  select(0,0,0,0,0xbf7feab4)

Same thing repeated over and over. (Those are the last few lines.) What appears to be happening is it's running through to the select, hangs a moment, returns 0, then tries again. I don't know if that's normal behaviour, but it seems okay to me as it sounds like it's just polling the socket?

What I am kind of concerned about is the wait returning an error that the process doesn't exist. The output of truss (which I started the process with, getting the initial setup) shows a call to fork() that seems to return fine (returns a PID, no error).

In case it helps, the output of truss: http://nucleardog.com/apache-truss.txt (Updated for httpd -X.)

I did actually connect during that run - select never seems to return a new connection? As well, I turned LogLevel up to Debug in the Apache config and tried running again - it never logs any connection.

I think this is perhaps more of a FreeBSD issue than Apache, then, as it seems Apache never receives the request. :/

Any (more) ideas?

EDIT 2: This appears to be an issue with something in FreeBSD, given that Apache never seems to receive the connection request. I made a post at the FreeBSD forums (forums.freebsd.org/showthread.php?p=118612) to see if I can find any help there. Thanks folks for your time!

share|improve this question
1  
The wait4() error is a red herring, it appears that rather than keep track of the pids of its children, apache just repeatedly calls wait4(-1,...) which will work on any child that has exited. The no child process error is that no children have exited. –  DerfK Jan 10 '11 at 1:14
    
are you tracing syscalls in the parent process? That won't be enlightening. –  covener Jan 10 '11 at 1:21
    
That's why I recommend running apache with the -X option so it doesn't fork any children. –  Phil Hollenback Jan 10 '11 at 4:36
    
After the recommendation of -X from Phil, I re-ran truss. The original link has been updated, although it doesn't seem terribly different. Could just be my untrained eyes. (I was under the impression -f to truss would trace the calls of any child processes as well?) –  NuclearDog Jan 10 '11 at 19:49

1 Answer 1

up vote 1 down vote accepted

Try the following:

  1. pstree | less to find the master apache process pid
  2. strace -p <pid> -f to watch the system calls from the process form step 1.
  3. connect to apache with telnet or a web browser.

You may need to install pstree and strace from freebsd ports. You should see scrolling output from strace as it logs the system calls (file opens, etc.) generated by the running apache process. This should give you some clue as to whether apache is actually running and just not producing output for some reason.

This page about debugging apache contains the above info as well as other troubleshooting tips you can use.

Also you mention your access log but not your error log, have you looked at that? There could be some additional diagnostic info there.

EDIT: Duh, you did say you looked at error log, sorry. Also yes, truss is the default freebsd call tracing tool, not strace. Using truss is a better answer since you don't have to compile it from ports like strace.

share|improve this answer
    
The error log's the second quoted section with the warning about SSL. I'm compiling and installing pstree and strace right now. Thanks! –  NuclearDog Jan 10 '11 at 0:00
    
I didn't get strace working, but didn't manage to use ktrace and truss to ferret out what Apache is doing. Additionally details added to the question if you're interested. :) –  NuclearDog Jan 10 '11 at 0:10
    
Try running httpd with the -X debug option, which only runs one worker process and doesn't detach from the console httpd.apache.org/docs/2.2/programs/httpd.html - that should be simpler to troubleshoot. –  Phil Hollenback Jan 10 '11 at 0:48
    
Tried it with -X. Seems to give me about the same output. Also tried running Apache with LogLevel set to Debug - it's showing no output on the connection (meaning it's probably never actually receiving the connection request from the kernel?!). I tried with accf loaded/unloaded. Updated (although similar) truss output at the previously posted URL if you get the chance / have any more ideas. Seems you're the only one who can help! :) –  NuclearDog Jan 10 '11 at 19:09
    
Try shutting down your apache server completely and running something else on the port. For example, you could use netcat: while true; do nc -l -p 80 -q 1 < somepage.html; done and see if the problem is on the machine or in apache. –  Phil Hollenback Jan 10 '11 at 20:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.