How to debug linux networking: Address already in use

Question

I have a Slackware linux box where I cannot start any service that listens on one particular port on localhost. By using strace I found out that the error happens on the bind() call, and the error is EADDRINUSE (Address already in use):

bind(3, {sa_family=AF_INET, sin_port=htons(874), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)

This happens with any process I try to start listening on that port, so it is not related to the process itself. The above strace output comes from the command strace -ff nc -l -p 874 -s 127.0.0.1.

So, this suggests there is a process already listening on localhost port 874. However, I can't seem to find it. The following commands all return nothing:

netstat -aplunt | grep :874
netstat -na | grep :874
lsof -i :874
lsof -i tcp | grep 874
fuser 874/tcp
socklist | grep 874
iptables -t filter -S | grep 874
iptables -t nat -S | grep 874
iptables -t mangle -S | grep 874
conntrack -L | grep 874

If I try to listen on 0.0.0.0:874 it fails with the same error. Listening on one of the IP addresses configured on a nic works OK, and listening to 127.0.0.2:874 also works OK. Listening on a different port works fine, also on 127.0.0.1 or 0.0.0.0.

So, now I am curious. How can I find out why the network stack returns EADDRINUSE here? What other things could I look at, or what other commands can I run to get more information?

Additional info:

• Kernel 4.1.31.
• Selinux is not used here.
• Trying to connect to 127.0.0.1 with telnet returns "Connection refused"
• I'm running the commands as root

Answer 1

If your host is an NFS client, it may be using source port 874 for an NFS mount. I suspect that because the connection does not originate from userspace it may not be visible to the tools you've used so far.

Consider one of the following:

• Adjust the sysctls sunrpc.min_resvport and sunrpc.max_resvport (default 665 and 1023) to change the range of source ports that the NFS client uses
• Use a listening port outside of this range
• Use the noresvport option on the NFS mount to use the non-privileged range (may have security implications)