netstat shows a listening port with no pid but lsof does not

Question

This question is similar to Network port open, but no process attached?

I've tried everything from there, reviewed the logs, etc... and can't find anything.

My netstat shows a TCP listening port and a UDP port without a pid. When I search lsof for those ports nothing comes up.

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:44231           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:55234           0.0.0.0:*                           - 

The following commands display nothing:

lsof | grep 44231
lsof | greo 55234
fuser -n tcp 44231
fuser -n udp 55234

After rebooting, those "same" two connections are there except with new port numbers:

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:45082           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:37398           0.0.0.0:*                           - 

And once again, the lsof and fuser commands show nothing.

Any ideas what they are? Should I be concerned about them?

Answer 1

From data you provided I'd say it's related to some NFS mounts or something using RPC.

you can check with rpcinfo -p for ports that might be used by some of RPC related services.

Here is how it looks on my system

# netstat -nlp | awk '{if ($NF == "-")print $0}'
tcp        0      0 0.0.0.0:55349           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:18049           0.0.0.0:*                           - 

# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  10249  status
    100024    1   tcp  10249  status
    100021    1   udp  18049  nlockmgr
    100021    3   udp  18049  nlockmgr
    100021    4   udp  18049  nlockmgr
    100021    1   tcp  55349  nlockmgr
    100021    3   tcp  55349  nlockmgr
    100021    4   tcp  55349  nlockmgr

Answer 2

Some processes/pids are only available to root. Try

sudo netstat -antlp

it should return the pid of every open port that's not in a TIME_WAIT state

or, if you want to know process ID related to specific port (let us say 8765 for example) use the code

netstat -tulpn | grep :8765

Answer 3

Based on hint from @user202173 and others I have been able to use the following to track down the process that owns a port even when it is listed as - in netstat.

Here was my starting situation. sudo netstat shows port with PID/Program of -. lsof -i shows nothing.

$ sudo netstat -ltpna | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  -
$ sudo lsof -i :8785
$

Now let's go fishing. First let's get the inode by adding -e to our netstat call.

$ sudo netstat -ltpnae | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      199179     212698803   -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  0          0           -

Next use lsof to get the process attached to that inode.

$ sudo lsof | awk 'NR==1 || /212698803/'
COMMAND      PID    TID                USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
envelope_ 145661 145766               drees   15u     IPv6          212698803        0t0        TCP *:8785 (LISTEN)

Now we know the process id so we can look at the process. And unfortunately it's a defunct process. And its PPID is 1 so we can't kill its parent either (see How can I kill a process whose parent is init?). In theory init might eventually clean it up, but I got tired of waiting and rebooted.

$ ps -lf -p 145661
F S UID         PID   PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 Z drees    145661      1  2  80   0 -     0 exit   May01 ?        00:40:10 [envelope] <defunct>

Answer 4

I don't know what these are specifically, but kernel modules (NFS for example) do not have a PID to associate with these sockets. Look for something suspect in lsmod.

Answer 5

I dont know if this can be useful. I had the same problem and what I did is the following: First, I called netstat with options -a(all) and -e(extended). With the latter option I can see the Inode associated to the used port. Then, I called lsof |grep with the inode number obtained and I got the PID of process associated to that inode. That worked in my case.

Answer 6

I've turned @studgeek's excellent answer into a one-line / shell function. Broken into several lines for clarity(ish):

lookup_port_pid() {  
   netstat -lptnae |
   awk -v port="$1" '/^(udp|tcp)/ && (lasti=split($4,hostport,":")) && hostport[lasti] == port { print $8; exit; }' |  { 
     read inode; [[ "$inode" ]] && 
     lsof | 
     awk -v inode=" $inode " 'NR==1 { print; z=index($0,"DEVICE")-10; }  index(substr($0,z),inode) {  print; } '; 
  }; 
}

Usage: lookup_port_pid 443

Maybe it doesn't look like a one-liner anymore, but that's how I wrote it :)

The complex 2nd awk is there to prevent false positives -- ie, inode numbers appearing elsewhere in the lsof output, and is somewhat resistant against changes in the lsof output format, and variations in the line spacing. Normally, something like $6 would do fine, but lsof can sometimes push two columns together without whitespace or a running command could contain whitespace, etc.

Answer 7

Is there any traffic coming or going from this port, check that with tcpdump -vv -x s 1500 port 37398 -w trace.out Saves your capture in the file trace.out you can then open it with wireshark, or tcpdump -vv port 37398 and see whats going on directly.

Try to telnet to that port use netcat for the udp socket maybe you get some kind of banner that helps.

Get rkhunter and check your system for a backdoor.

Compare the md5 hash of lsof/netstat with the one from your install media, assuming the files where not updatet.