Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We've previously had issues in our environment where it seemed we were reaching FreeBSDs max connection limit. We took action demonstrated by http://nginx.org/en/docs/freebsd_tuning.html and bumped up our connections limit to 500: kern.ipc.somaxconn: 500

We're still having issues where we expect to see additional connections from clients to this particular server and we are not, they're missing.

We don't know whether this is an application issue or a networking issue. But I was wondering if there is some other limit we're hitting now, maybe with max number of established connections?

Here's some graphs showing our connection behavior which looks strangely like it's limited at 300 connections:

Connections to the aforementioned box Missing Commands

Approximate number of missing commands on our server ^

EDIT:
Additional details about the application: The application is a tclsh (tcl shell) listening on a specific tcp port using the socket application to receive connections. Maybe this is some tcl-based limitation or a socket application limit?

Troubleshooting Details: When I run nmap to basically "ping" the desired port over and over again

for i in {1..600}; do nmap -p 2069 serverIP; done

I seem to get the following with netstat:

netstat -Lan | grep 2069
tcp4  193/0/128      *.2069
tcp4  193/0/128      *.2069
tcp4  193/0/128      *.2069
tcp4  193/0/128      *.2069

This seems to mean that I am actually maxing out kern.ipc.somaxconn defaults. But we have already set that value to much higher than the default.

Even when I monitor established connections using:

netstat -an | grep 2069 | wc -l

I only get a grand total of 192 connections on 2069. Meaning it's not accepting more on that particular port.

share|improve this question
    
I would use a benchmarking tool like apachebench (ab) to first determine if there is actually a limit you are hitting. I would also take tcpdumps of the traffic during this time to see what's actually happening. You'll have to determine the cause before you can find a proper solution. –  yoonix Apr 23 at 19:25
    
I'm just wondering if there is any other inherit limits with max number of free connections or some other kind of network/tcp limits I might be hitting. Debugging this particular application is not very easy like it would be with apache as it is an in-house custom piece of software that infrastructure team does not have ownership of. –  rainereality Apr 24 at 2:33
    
It's most likely application scalability issue not connected to kernel. –  SaveTheRbtz Apr 24 at 8:23
    
I suspect this, but I'm not familiar enough with kernel limits to definitively say so. Any good methods of proving this? How can I easily open additional established connections and prove it to be an application issue? –  rainereality Apr 24 at 11:25
    
Additional details about the application: The application is a tclsh (tcl shell) listening on a specific tcp port using the socket application to receive connections. Maybe this is some tcl-based limitation or a socket application limit? –  rainereality Apr 24 at 11:55

2 Answers 2

File handles come to my mind. First check with ulimit -n (or, depending on the shell, limit -n) if it returns 1024. If it does, bump up the file handle limit with ulimit -n 16384, limit -n 16384 or so. See if that helps.

share|improve this answer
    
Doesn't look like a file handle limit. Looks like a connections per socket limit. –  rainereality Apr 25 at 13:10
up vote 0 down vote accepted

Seems like it was actually an application limit. The process listening on the 2069 socket was built with a max number of 192 listen connections.

I'm assuming a lot of applications are built like with these limits in place and its likely that somaxconn just increases total allowable listening sockets vs what the application is actually built for.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.