I have an application with pool of connections to MySQL. At some point, in production, every 1-3 seconds some socket is closed. As a result my app restarts this connection.
This is pretty big pool of 30 connections / application server talking to single instance of MySQL. In total probably there is 1000 connections on database side.
When I restart the application process, the problem goes away. I look for some reason connected to sockets, file descriptors, kernel.. Applications opens and closes ~10-30 connection every second, in order to have about 4-5k of incoming connections.
How to avoid it?
lsof -nPi tcp
reports around 5k connections (and correct number of outgoing to mysql).
shows 10k to 60k ports available
ulimit -n is set to high number (100k)
Vmstat shows no swap. I am at 50% of CPU and 25% of memory.
What can I check next? How to fix it?