0

I'm running a series of server programs on Windows Server 2016 that are using a MySQL server (MariaDB 10.0.27). At first everything is fine, the servers connect to the database, clients can connect to the server programs, everything is within expectations. But after the programs have been running for a few hours I suddenly can't create any new connections to the MySQL server anymore. The server programs are still running, I can connect to them, and they appear to still be using the database server, but when I try to connect to the MySQL server myself I get the following error message:

ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (10055 "Unknown error")

At first the error message said the following, but a search led me to increase the number of available TCP ports, as a lack of those was supposedly the cause for that error.

(HY000/2002): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

My MaxUserPort registry setting is now currently 51024. However, the problem didn't disappear, only the error message changed.

Now when I close the server programs I can create new connections again just fine, without changing anything about the continiously running MySQL server, and after I start them again I can also still create new connections for a while. I was thinking it might be a MySQL connection number restriction somehow, but that setting is currently at 1500 in my.ini and I don't see any errors in the MySQL console.

max_connections = 1500

Also, I noticed that once I can't create new connections anymore, I also can't open any websites in my browser, getting the error ERR_CONNECTION_FAILED, which makes me think about that TCP port limit again. But when I run netstat I see less than 300 connections, and it's my understanding that even with default settings this should be fine. I can also start a web server but I can't open anything on it via HTTP from the same machine. I can access the web server's content from the outside though.

I'm at a total loss of what's going on here. Does anyone have an idea?

2
  • Sure sounds like you are hitting some resource limit: memory, handles, processes, open files, DB connections, sockets. Take a look at your server programs to make sure they are shutting down connections and releasing resources properly.
    – stark
    Oct 15 '19 at 17:59
  • @stark Unfortunately I can't change much about the server programs, though I know that they were working as is at some point in the past for someone else, which makes me believe that it's an issue with the system's configuration. What I can say is that the hard drive and the RAM have air to breath, so it's not them. The number of processes in the task manager is perfectly normal as well, not to mention that I can still start applications, and open sockets/connections would appear in netstat, would they not? =/
    – Mars
    Oct 15 '19 at 18:18
1

The problem turned out to be socket/handle exhaustion. I first used ProcessExplorer to monitor the server processes, and they kept creating new handles, namely sockets, presumably trying to connect somewhere. I guess they didn't properly dispose those connections or something, because over time those several processes accumulated >60k handles each before I couldn't create new connections anymore.

I then used TCPView to monitor the server's connections and they kept trying to connect to something on a local port, where the servers expected a NoSQL server that I didn't know about before. As soon as I ran that database server the programs successfully connected to it and stopped creating new connections in an endless loop.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.