Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have an application with pool of connections to MySQL. At some point, in production, every 1-3 seconds some socket is closed. As a result my app restarts this connection.

This is pretty big pool of 30 connections / application server talking to single instance of MySQL. In total probably there is 1000 connections on database side.

When I restart the application process, the problem goes away. I look for some reason connected to sockets, file descriptors, kernel.. Applications opens and closes ~10-30 connection every second, in order to have about 4-5k of incoming connections.

How to avoid it?

lsof -nPi tcp 

reports around 5k connections (and correct number of outgoing to mysql).

/proc/sys/net/ipv4/ip_local_port_range

shows 10k to 60k ports available

ulimit -n is set to high number (100k)

Vmstat shows no swap. I am at 50% of CPU and 25% of memory.

What can I check next? How to fix it?

share|improve this question
    
Is there a firewall between database and client? Are all the connections from all the ports used? Are the connections closed at some point or are they just opened when the client starts up and stay open during the lifetime of the client? –  Andreas Florath Jan 12 '13 at 19:36
    
1. No firewall app <-> db. No, I do not thinks so -- I have only 5k connections and 50k+ ports available (for outgoing). No orphan sockents & TIME_WAIT issues. These are lifetime connections. Too keep them alive I have keepalive in mysql layer (select 1) every minute. –  user1971693 Jan 14 '13 at 3:21
    
(cont) Ad. the lifetime connection thing: this is weird -- when connections go sideways, the new ones are also broken soon. So this is rather state of my app (parent linux process) or state of the host. –  user1971693 Jan 14 '13 at 3:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.