Linux networking port exhaustion

Question

I've done as much research as I could on this without digging right off in the kernel source. There seems to be a large amount of disinformation/incorrect info on the subject, so I hope this answers the question for me and others once and for all.

Strictly IPv4 speaking, is port exhaustion actually possible? Let me explain:

• Seemingly there are 65535 ports available for use. 0 isn't available.
• I have read that port exhaustion requires the (src ip, src port, dst ip, dst port) tuple to be unique.
• To be clear and assuming I can use 100% of ephemeral ports via sysctl net.ipv4.ip_local_port_range setting

And this is the question: Is this how it works?

• I could have 65k connections from 127.0.0.1:(x) to 127.0.0.1:80
• I could have 65k connections from 127.0.0.1:(x) to 127.0.0.1:555
• Basically once again, the question is (srcip,srcport,dstip,dstport) must be unique, correct?
• I could not open any more than 65k connections from ip "A" to IP "B", Port "N"
• Likewise, a single IP could not open more than 65k connections to my webserver at x.x.x.x:80, however I could support much more than 65k overall as long as they are from different source IPs?

Finally, I am a little confused about (outgoing) ephemeral ports and incoming ports which are listening. I realize once the connection is established each side of the connection is a peer and equal, but before that happens:

For example, if indeed the (srcip,srcport,dstip,dstport) tuple must be unique, why is it if I enable, for example

net.ipv4.ip_local_port_range = 1024 65535

Which allows the use of ephemeral ports from 1024-65535, that if I have services that bind on port 3306 (mySQL, for example), they will sometimes fail to start because the port is in use.

Is this relating the fact that: (And this is a statement I am asking to be validated):

• (srcip,srcport,dstip,dstport) are needed to be unique for each connection with the port range of 1-65535 (not paying attention to the OS's usage of ephemeral ports)
• However, for a socket to bind, it could be seen as (srcip,srcport, *, *). Or another way to put this, is the IP must not be using that port for any reason to bind to?

I can verify the above behavior, i.e. I use the exact sysctl line above, and because of it I moved mySQL to a port lower than 1024 because it would occasionally and very randomly fail to restart because assuming the OS was using that port (3306) for an ephemeral port.

Answer 1

You have two main questions here:

1.

Strictly IPv4 speaking, is port exhaustion actually possible?

Yes. Take, for example, a load balancing router sending all connections to a NAT IP address. This is likely to happen when you have many SRC IPs connecting to the bottleneck of a single DST IP.

This means that your webserver could have a bunch of connections like:

root@buglab:~# netstat -pnt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 173.200.1.18:80      10.100.1.100:49923        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.200.1.200:10155        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.10.1.10:14400        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.10.1.10:50652        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.20.1.20:57554        ESTABLISHED 13939/nginx: worker 

and that's perfectly fine. However, if all the 'Foreign Addresses' were the same, this can cause an issue (e.g. 'big router that performs NAT <---> server with one IP address').

If I had to postulate as to why ephemeral port exhaustion isn't a common issue, I'd suggest it's because every port requires a listening service and enough resources to respond -- another resource (memory, cpu) is normally a bottleneck first.

However, I've personally come across a few port exhaustion issues when working at a load balancing company.

2. Why can a used port present an issue for a listening service?

"Which allows the use of ephemeral ports from 1024-65535, that if I have services that bind on port 3306 (mySQL, for example), they will sometimes fail to start because the port is in use."

The mySQL server cannot bind to that port if it's in use - say by localhost:3306 or on all interfaces. For example, see the 0.0.0.0:80 line in the following netstat output?

root@buglab:~# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      PID/Program name
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      964/php-fpm.conf)
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1660/mysqld     
tcp        0      0 0.0.0.0:842             0.0.0.0:*               LISTEN      1317/inetd      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      13938/nginx     

That means that port 80 is listening across all interfaces local to the server. If another process holds port 80 before my nginx server starts, nginx will not be able to take control of that port and will likely fail its startup procedure.

Normally, port 3306 is fine because listening services have pre-defined ports (or ranges) that are requested from the host machine - e.g. port 80 and 443 for webservers.

Answer 2

I could have 65k connections from 127.0.0.1:(x) to 127.0.0.1:80 I could have 65k connections from 127.0.0.1:(x) to 127.0.0.1:555

a:yes, if you local port is 1-65535, connections count is 65k-1(one port is listen)

Basically once again, the question is (srcip,srcport,dstip,dstport) must be unique, correct? a: yes

I could not open any more than 65k connections from ip "A" to IP "B", Port "N" Likewise, a single IP could not open more than 65k connections to my webserver at x.x.x.x:80, however I could support much more than 65k overall as long as they are from different source IPs?

a: A single IP could not open more than 65k connections to my webserver at x.x.x.x:80, because srcip,srcport,dstip,dstport is unique. yes you could support much more than 65k,if soruce ip is diffrent