Network port open, but no process attached?

Question

I have a weird situation going on with an open network port. My main question is, why would there not be a program associated with an open TCP port:

netstat -ln --program
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:5666                0.0.0.0:*                   LISTEN      -  

For my specific case, there is supposed to be an nrpe daemon (opsview install) listening on port 5666, but there is no nrpe daemon running. If I try to start it, it exits immediately.

lsof -i :5666 doesn't show any output either. There is no (x)inetd running on my system.

UPDATE

Yes, I was running those commands as root. Telnet would could, but there was never any response.

After further investigation, I found a kernel error in dmesg: this was an EC2 instance (actually several of them) running an older kernel (2.6.16 is apparently unstable). The fix to stop the crash was to upgrade kernels.

It looks like the way the kernel crashed caused the process to go away and leave the port open.

Answer 1

Ports open by the kernel won't show up with program name. Some NFS and OCFS stuff come to mind. Maybe it's something like that?

Or it could be a kernel bug. Check kernel logs for OOPS and BUG.

Answer 2

Have you run netstat and lsof as root or with sudo? Notice the last column:

netstat -ln --program
tcp        0      0 192.168.21.1:53         0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -

sudo netstat -ln --program
tcp        0      0 192.168.21.1:53         0.0.0.0:*               LISTEN      2566/named      
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      2566/named      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3125/sshd

From the netstat manpage:

You will also need superuser privileges to see this information on sockets you don’t own.

How do you know there isn't one running? If the port is in use it makes sense that it would exit immediately with a 'socket in use' error. what happens when you telnet to the port?

telnet localhost 5666

Answer 3

execute 'netstat --tcp --udp --listening --program' as root user. other wise you it won't give PID/Program Name

then use kill -9 PID command

Answer 4

I actually wrote a small shell script to help id these occassional questions:

#! /bin/bash
([ "$1" = "" ] || [ "$2" = "" ]) && echo "Usage: tracer <space> <port>" && exit 0
for i in `fuser -n $1 $2`
 do
  ps aux | grep $i | grep -v 'grep'
 done

save as /usr/local/bin/tracer; output:

root@mo-log:/usr/flows# tracer tcp 80
80/tcp:             
root     27904  0.0  0.0 111668  3292 ?        Ss   Aug04   0:03 /usr/sbin/apache2 -k start
www-data 32324  0.0  0.0 335332  3560 ?        Sl   Aug05   0:00 /usr/sbin/apache2 -k start
www-data 32327  0.0  0.0 335324  3560 ?        Sl   Aug05   0:00 /usr/sbin/apache2 -k start

You will need root privileges to use it

Answer 5

I was able to track down the process by getting its inode via netstat and then using that inode with lsof. See my more detailed answer in https://serverfault.com/a/847910/94376.

Answer 6

Sometimes, nfs related programs cannot be seen on the program list.

Also, LDAP pam modules and libnss_ldap open connections to ldap servers, but there is not an actual process holding the connection open, so netstat -tnp shows an active connection without a process.