5

AFAIK, port exhaustion happens when a host is trying to establish a lot of connections to another server and the sessions used up all the available ephemeral ports. e.g.

192.168.1.1:port -> 10.20.30.40:443

This does not seem common but it is an issue for load-balancer. My question is, is there a way to get an early warning that it is getting close?

I know I can use netstat to get all the connections but if there are tens of thousands of connection. It would take quite a bit of resource to run the command, sort the target and count the number of connections sharing the same client IP, server IP and server port. And do it on a regular basis.

Is there an easier way? As the TCP stack is allocating port, could it some how know that it is having trouble hashing an unused port and generate an error or log?

1
  • Port exhaustion is not based on the tuple of (dest ip, dest port) - you can exhaust all your ephemeral ports to a single destination host and not have any to initiate connections to other machines. You only need to track the ports in use, as others have answered below.
    – mfinni
    Sep 7 at 16:07

2 Answers 2

3

you could write a bpf software using bpftrace to watch all connections opening and closing and maintain counters for each client,server,server port combination. That seems to be overly complex for the usecase. Slicing and dicing /proc/net/tcp{,6} (the same data that netstat shows) would seem to be quite fast, and would certainly be done on machines with tens of thousands of connections in sub second time frames.

0

if you simply type netstat, then yes, a system or device with thousands of connections will take some resources. It will try to resolve all those IPs! When you're watching connections, use netstat -n (only IPs, don't try to resolve them) and then you can simply count the number of connections, or specific ports... whatever. You can implement this a number of ways, depending on how much effort you want to put into it. BPF, DTrace, a simple CLI thing, SNMP counter. Be really fancy and do it with perl/python/winnebago or what not. Most load balancers have a limit when they start to crap out. Totally depends on what you're using and how it's being used though. With that said however, there are only a set number of ports available to the TCP stack. after 65,535, you're done! Some systems will let you assign arbitrary high ports and some networks will blindly forward anything they can't parse or understand (i.e. a TCP connection to 1.1.1.1:239321232). It's great fun!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.