Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a CentOS 5.5 server running Apache on port 80 as well as some other applications. All works fine until I for some reason need to restart the httpd process. Doing so returns:

sudo /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

First I thought perhaps httpd had frozen and was still running, but that was not the case. So I ran netstat to find out what was using port 80:

sudo netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 *:7203                      *:*                         LISTEN      24012/java          
tcp        0      0 localhost.localdomain:smux  *:*                         LISTEN      3547/snmpd          
tcp        0      0 *:mysql                     *:*                         LISTEN      21966/mysqld        
tcp        0      0 *:ssh                       *:*                         LISTEN      3562/sshd           
tcp        0      0 *:http                      *:*                         LISTEN      3780/python26

Turns out that my python process had taken over listening to http in the instant that httpd was restarting. So, I killed python and tried starting httpd again - but ran into the same error. Netstat again:

sudo netstat -tlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 *:7203                      *:*                         LISTEN      24012/java          
tcp        0      0 localhost.localdomain:smux  *:*                         LISTEN      3547/snmpd          
tcp        0      0 *:mysql                     *:*                         LISTEN      21966/mysqld        
tcp        0      0 *:ssh                       *:*                         LISTEN      3562/sshd           
tcp        0      0 *:http                      *:*                         LISTEN      24012/java

Now my java process had taken over listening to http. I killed that too and could then successfully restart httpd.

But this is a terrible workaround. Why will these python and java processes start listening to port 80 as soon as httpd is restarted? How to solve?

Two other comments. 1) Both java and python processes are started by apache from a php script. But when apache is restarted, they should not be affected. And 2) I have the same setup on two other machines running Ubuntu and there's no problem there.

Any ideas?

Edit:

The Java process listens to port 7203 and the python process supposedly doesn't listen to any port. For some reason, they start listening to port 80 when apache is restarted. This hasn't happened before. On Ubuntu it runs fine. For some reason, on my current CentOS 5.5 machine, this problem arises.

share|improve this question

migrated from stackoverflow.com Feb 25 '11 at 5:04

This question came from our site for professional and enthusiast programmers.

1  
You might want to try unix.stackexchange.com –  Nimrod Feb 25 '11 at 4:08
    
Why java and python are started by php? Are they part of the page? Do you use mod_fastcgi? –  Mircea Vutcovici Oct 25 '11 at 1:02
1  
This just happened to me in CentOS 5.8, only the process that stole the ports was ntpd. I'd like a real explanation and solution too. –  JCCyC Oct 26 '12 at 18:05

4 Answers 4

The problem might be the way how Apache starts the sub-processes. They might get spawned by forking and letting the cloned Apache processes become other processes. Every clone inherits all open file-handles of the parent process, including the open handle of TCP port 80.

Netstat only shows one application associated with the open file handle, while three processes keep the handle open.

Possible solutions to your problem:

  1. If it is important to keep the sub-processes running during a Apache restart, the simplest solution would be to start the 3 Processes as separate system-services.

  2. If they depend on a running Apache, a 'apache stop' command should also terminate them. This could be achieved by editing the /etc/init.d/apache script.

  3. If you are forced to start them from Apache, you need to start them as real "daemon processes"!

I wrote a daemon creator for a PyQt-app inspired by http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/. You could start this script from Apache, call exec_as_daemon for each of your sub-processes and close it.

# Copyright: (c) 2011 phobie <pydaemon.phobie@safersignup.com>
# License: CC-by-sa 3.0
import os
def exec_as_daemon(s_path_cmd, l_args=[]):
    i_pid = os.fork()
    if i_pid != 0:
        # Always remember to gobble your zombie children
        os.wait()
        # Back to parent
        return
    # Detach from parent
    os.setsid()
    # Do not block any mounts
    os.chdir('/')
    # Reset file creation rights
    os.umask(0)
    i_pid = os.fork()
    if i_pid != 0:
        # Close the direct child process
        os._exit(0)
    # Get the maximum count of open file handles
    try:
        import resource
        i_fd_max = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
        if i_fd_max == resource.RLIM_INFINITY:
            i_fd_max = 1024
    except ImportError:
        i_fd_max = 1024
    # Try to close all possible file handles
    for i_cur_fd in range(0, i_fd_max):
        try:
            os.close(i_cur_fd)
        except OSError:
            pass
    # Assosiate STDIN with /dev/null
    os.open(os.devnull if hasattr(os, "devnull") else '/dev/null', os.O_RDWR)
    # STDOUT to /dev/null
    os.dup2(0, 1)
    # STDERR to /dev/null
    os.dup2(0, 2)
    l_args.insert(0, s_path_cmd)
    # Let the clone become a other process
    os.execv(s_path_cmd, l_args)

if __name__ == '__main__':
    exec_as_daemon('/usr/bin/java', ['-jar', '/usr/local/bin/apache_helper_daemon.jar'])
    exec_as_daemon('/usr/local/bin/apache_helper_daemon.py')
share|improve this answer

In theory, only one process should be listening on a given IP/port. If you need more than one application to listen on the same port, you need a reverse proxy kind of setup that will determine based on the content received which process to get the message.

It would be helpful to know what your Python and Java apps are doing. If they are also servers that listen on port 80, they are getting stuck at the point of opening port 80 for listening while Apache is running and as soon as you kill Apache, the next one in the process queue gets through and opens the port. You will need to change your Python and Java codes to listen on different ports.

share|improve this answer

You are failing to clean shutdown your apache master process and it's children are left hanging on to sockets. Instead of running restart, run a service httpd stop. If this fails to exit cleanly, you need to investigate further the java and python apps you've enabled through apache.

If a service httpd stop doesn't work correctly, uninstall your java and python apps until you can cleanly restart. Then, investigate why those processes are not terminating correctly.

share|improve this answer

It is strange what is happening. As a workaround use reload instead restart:

sudo /etc/init.d/httpd reload

Try to find out about the new java and python process. You can try to find out when they have started, and the command line parameters:

ps -edf|egrep 'python|java|PID'

Check also the /etc/init.d/httpd for anything related to java or python.

Use an audit tool to find out when a program is started, by whom, etc.:

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.