Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an application running on a CentOS machine that connects to a remote DB server on another CentOS machine. Every once in a while, without any discernable warning signs, the application seems to just hang.

I spent a lot of time debugging and found that the problem is at the TCP/socket layer - there is a blocking read operation where the application is waiting to get a full response back from the database, but never does. As such, it just hangs.

I did a tcp dump and see that when this hanging occurs, the connection on the app server has a Recv-Q size of zero (0). At that same time, the corresponding connection on the DB server has its Send-Q buffer "frozen" at a very high number of bytes (this number doesn't move).

So, not being a "network guy", it seems to me like the DB is trying to send a response (a whole lot of bytes), and for some reason, the app server isn't able to receive any of it.

Is this indicative of a networking issue? If so, what could be going on here, and how do I troubleshoot? Is this Linux on the DB server stepping in and preventing the DB server from truly sending anything from its Send-Q buffer? Again, how to troubleshoot? Thanks in advance!

share|improve this question
1  
Hi, so I looked closer at the TCP implementation under Linux and I think an empty Recv-Q does not mean that the data was read by the application. It just means that it was passed from the IP layer to the TCP layer. The RWIN is calculated after the data is stored in the receive buffer so the Recv-Q would be empty. Here is (kind of) short summary: ece.virginia.edu/cheetah/documents/papers/TCPlinux.pdf So you would still need to debug your application to see why the data is not removed from the receive buffer... –  Sebastian Apr 19 '13 at 13:47
    
Thanks again @Sebastian (+1) - so to summarize, you think that: (1) the DB is sending back data, then (2) the app stops reading off the Recv-Q, then (3) all the data on the Recv-Q transfers from the IP layer to TCP, and (?) gets lost somehow, then (4) Linux kicks in and sets RWIN to 0, then (5) the DB server stops sending data, so that (6) Recv-Q stop receiving data and shows that it has 0 bytes. Is that the gist of it? Thanks again! –  Mara Apr 19 '13 at 14:07
    
No, not lost. The Data is saved to the receive buffer and not read by the application. Then the receive window gets set to 0 as the data fills the receive buffer. That's what I suspect. –  Sebastian Apr 19 '13 at 14:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.