I have an application running on a CentOS machine that connects to a remote DB server on another CentOS machine. Every once in a while, without any discernable warning signs, the application seems to just hang.
I spent a lot of time debugging and found that the problem is at the TCP/socket layer - there is a blocking read operation where the application is waiting to get a full response back from the database, but never does. As such, it just hangs.
I did a tcp dump and see that when this hanging occurs, the connection on the app server has a Recv-Q size of zero (0). At that same time, the corresponding connection on the DB server has its Send-Q buffer "frozen" at a very high number of bytes (this number doesn't move).
So, not being a "network guy", it seems to me like the DB is trying to send a response (a whole lot of bytes), and for some reason, the app server isn't able to receive any of it.
Is this indicative of a networking issue? If so, what could be going on here, and how do I troubleshoot? Is this Linux on the DB server stepping in and preventing the DB server from truly sending anything from its Send-Q buffer? Again, how to troubleshoot? Thanks in advance!