0

I have a Linux server with an Erlang application which shows an odd limit of exactly 65536 max tcp incoming connections.

The Erlang application is written with the Cowboy framework.

I have tuned the kernel parameters in this way:

/etc/sysctl.conf:

# Increase system file descriptor limit
fs.file-max = 300000

# Discourage Linux from swapping idle processes to disk (default = 60)
vm.swappiness = 10

# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

# Disable source routing and redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0

# Log packets with impossible addresses for security
net.ipv4.conf.all.log_martians = 1

/etc/security/limits.conf:

*         soft    nofile          300000
*         hard    nofile          300000

I also checked the maximum number of Erlang processes, but it doesn't seem the limiting factor:

1> erlang:system_info(process_limit).
262144

The test connections are made by 4 client machines from 4 different IP addresses (so it isn't a problem with client limits) and after an initial authentication, the connections are left idle, so server CPU utilization is below 50% and memory utilization is also below 35%.

EDIT: I'm not sure if this is important: the Erlang server application is listening on port 8000 and I get it to listen on port 80 too with this iptables rule:

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8000

What else should I look into to remove the 65536 connections limit?

0

Check how is coded your application. There is a limit of 65535 port, and those in that range: 0-1024 are reserved.

If your application use ephemeral's port; from RFC-6056

  1. Ephemeral Ports

2.1. Traditional Ephemeral Port Range

The Internet Assigned Numbers Authority (IANA) assigns the unique
parameters and values used in protocols developed by the Internet
Engineering Task Force (IETF), including well-known ports [IANA].
IANA has reserved the following use of the 16-bit port range of TCP
and UDP:

o The Well-Known Ports, 0 through 1023.

o The Registered Ports, 1024 through 49151

o The Dynamic and/or Private Ports, 49152 through 65535

The dynamic port range defined by IANA consists of the 49152-65535 range, and is meant for the selection of ephemeral ports.

In last step ask the vendor if it's a hardcoded value. (an int_16)

  • It is a server application which binds to port 8000, so it should not be a limit with ephemeral ports – Enrico Detoma Sep 22 '15 at 16:40
  • @enricodetoma I would still ask the vendor, as like i told it can be a simple error, could be only output error – yagmoth555 Sep 22 '15 at 17:11
  • I have the Erlang source code, so I will try to debug, if there are no other kernel parameters to change. – Enrico Detoma Sep 22 '15 at 21:38
0

If your 4 different clients connect to server from the same host, you cannot have more than 65536 active incoming connections at the same time, as you can generate only 65536 ports for same incoming IP address.

  • The clients are 4 different machines, so 4 different ip addresses – Enrico Detoma Sep 22 '15 at 16:42
0

I found the problem after looking at dmesg output:

nf_conntrack: table full, dropping packet

So the problem was in the TCP redirection 80 => 8000 using iptables.

I removed that iptables rule and I made Erlang listen on both ports 80 and 8000 directly, and the 65k limit was removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.