0

Where I work, we have had some issues with connections to the database. We get a lot of failures which appear as "recv failed" on the client side. I have used Wireshark to try to debug the problem and gotten a bit further, but now I'm pretty much stuck.

First, a bit about the infrastructure:

  • The internal IT-systems are run as virtual machines (servers) on 1-2 physical machines. They consist of several applications on Glassfish, as well as a SQL-server.
  • In addition, there are some servers at a external provider, including a SQL-server here as well.
  • Between the internal systems and the external ones there is a firewall, that routes the traffic.

The problem is the communication between the applications in the internal zone, and the database in the external zone. The problem only occurs on some (at the time actually just one) of the virtual servers, and is not application-specific (it happens both through Glassfish and JDBC connection pools, as well as SQL-clients like SQuirreL.)

And, it gets stranger, since small SQL-statements are run, but once they get to a certain length, nothing happens, until the connection is closed on the client-side (with recv failed).

Here is what I found with Wireshark:

  • On the servers that work, a small SQL query appear in Wireshark as a single TDS-packet, with a direct TDS packet response (does not show TCP acks and such..) Large queries are typically first sent as TDS, then resent part by part as TCP, acked correctly and then result is returned. (In one case I saw that it first tried a 2400 byte TDS, then a 1514 byte TCP, and then 590 byte TCP and got acks for the last ones..)
  • On the server that doesn't work, small queries are sent as a TDS-packet (and this works.) Large queries on the other hand, it first tries to send the entire query as a TCP-packet, then gets an ACK with seq and ack = 1 (indicating no data received?), then it tries to send a 1514 byte TCP packet, getting no ACK back. It tries this a few times more, before the connection is broken.
  • If the maximum result is ticket in SQuirreL it first sends a small TDS-packet stating the restrictions on the result, then tries to send the query, but only recieves a resend of the ack of the first one. Resends of the query receive no acks.

I'm left pretty much blinded as to where to look next, does anyone have any pointers?

Update: Doing a restart on the virtual server solved the problem - for now at least..

Update #2: .. and now the problem is back..

0

We had a network expert look at this, and it turned out to be erroneous MTU-settings that caused this. From what I understand most of the network infrastructure supported jumbo frames, so the MTU was set to 9000. But there was one component (VPN-tunnel) who didn't support this, so frames over a certain size were truncated. Changing all MTU-settings 1500/1460 fixed the problem.

|improve this answer|||||
  • It would be a serious flaw in the VPN software, if it really truncates packets exceeding the MTU. It is supposed to send an error back to the source indicating what the MTU is and drop the original packet. And the sender is supposed to use this information to retransmit the data in packets of a more suitable size. – kasperd Feb 4 '15 at 21:07
  • I think this actually might have happened, only it didn't split the packets into small enough units. What I saw was that it first tried a packet of ~2400 bytes, then retried with ~1518 bytes, neither of which got through. So maybe the fault was with this.. I'm no network specialist so not sure. Luckily someone who is managed to fix it based on the information provided :) That being said, the network is supposedly something that has been added to and tweaked over the years, they are working on renewing the whole thing these days (to fix some other, minor problems). – Tobb Feb 5 '15 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.