4

My web-services server sometimes does not receive correct HTTP requests and returns "500 - Internal Server Error". Using tcpdump and Wireshark on the server, I found out that HTTP requests are splitted into 2 TCP packets, and that sometimes, the server tries to process the request before the second packet could arrive.

This wireshark capture has been taken on the server side.

So what I see is that :

  • The first fragment of the HTTP request is received at 54.659
  • It is received again at 71.168
  • The second (and last) fragment of the request is received at 99.869 (that is 45 seconds after the first one)
  • Four millisecond before, at 99.865, the server kind of timed-out and tried to process an incomplete request (which gives a 500 Error)

I don't know where to look now. I would say that it is a network issue but I have several TCP streams where the server tries to process the request several milliseconds before it is completely received. On the other hand, TCP paquets that takes 45+ seconds to arrive means that the network is really bad.

Do you have any pointers on how to investigate more ?

Improve this question
4
  • Look at the load on the server. Is it out of virtual-memory or swapping heavily to disk? What about database connections, are they working well? Look at IO load and CPU load on the server. Is there a cache somewhere that is misbehaving?
    – Mike Pennington
    Jun 20, 2012 at 11:54
  • Can you show the IP and TCP headers of the 2 packets? Is the request really split into two or is the tcp packet split into 2 fragments?
    – ott--
    Jun 20, 2012 at 12:24
  • @MikePennington The server does not swap. The CPU load is not very high as lots of DB query are done (the DB is on the same subnet). I don't think there is any cache involved but I will have a look.
    – Pierre Laporte
    Jun 20, 2012 at 14:02
  • @ott-- Wireshark shows that the first packet is a "TCP segment of a reassembled PDU". My guess is that there are two distinct TCP pakets (#48175 and #60486 in the screenshot)
    – Pierre Laporte
    Jun 20, 2012 at 14:05

3 Answers 3

Reset to default
2

I don't know where to look now

Nowhere. Seriously.

Happens.

hat is 45 seconds after the first one

That is HUGH. Seriously. Internet latency Europe to USA is around 150ms. You are 30 times that much - it is a drop without resend. Sadly, unless you control BOTH SIDES (!) you can not control the behavior of the client. Stuff like that happens.

If that is your LAN - it is seriously crappy. It that is internet, that is just how it is. The main question is how bad it is - if it is "some connections of thousands" it could be a serious network problem on the other end. If that happens at times for nearly everyone, it is closer to your side (connection, data center etc.). Yesterday, for example, we had a situation like that here - some stupid *** DDOS's one of my links. SERIOUS congestion could lead to packets getting nearly impossible to get through. But if that is not you - there is nothing you can do.

This is like someone coming too late to a meeting and telling you he had a serious traffic jam. unless the Jam is in your street you can not know. The Internet can be at times in places quite crappy.

Improve this answer
4
  • I definitely agree with you. I think you meant 300x the Europe<->USA latency instead of 30, which means you are even more right. One thing is sure is that I reproduced this on my machine using a VPN network to reach the server. I have been told that this error has been seen in production but I haven't any information yet.
    – Pierre Laporte
    Jun 20, 2012 at 14:07
  • Could be a totally overloaded link - but then, internet is not guaranteed, though likely good on the backhaul. Investigate network links on both sides - not technology only, also contractual agreements and load factors.
    – TomTom
    Jun 20, 2012 at 15:03
  • I think you pointed me on the right path. The servers are rent in an external datacenter and it is highly probable that the bandwidth is contractually limited. I'm going to investigate on that. Many thanks for your help !
    – Pierre Laporte
    Jun 20, 2012 at 16:02
  • No problem. Sometimes this arrogant asshole (me) has a good idea ;) My own company is distributed, too - so we know a lot about link overloads ;) QOS is a MUST for stuff like that, as is a decent router that faily distributes bandwidth. Regular queues will NOT work properly on a totally overlaoded LAN link.
    – TomTom
    Jun 20, 2012 at 16:34
2

The retransmission might be the clue. The client didn't receive the ACK that the server sent for that packet. The servers also got two SYNs from the same source port right at the start of the dump.

The IP addresses there are both local so I'm guessing this isn't going over the internet. Do you have access to the client?

Try doing the same packet trace on the client and server simultaneously. I expect you will see large discrepancies between what is sent and what is received.

Then trace the cables. You have latency and packet loss so large I would be looking for an air-gap that the electrons are jumping across when the voltage climbs high enough or a stripped Cat-5 cable with rusty connections and water dripping on it.

Replace networking parts (devices, ports and cables) until the problem goes away.

Improve this answer
2
  • To be even more precise, the client is my machine and the server is reached through a VPN (so over the internet). I will tcpdump both my machine and the server and keep you informed.
    – Pierre Laporte
    Jun 20, 2012 at 14:15
  • What VPN? Something TCP based (SSL) is very bad when a packet is dropped. You need something either UDP based or standards based (IpSec, PPTP) which both do not rely on a TCP stream (PPTP uses GRE packets - each independent.
    – TomTom
    Jun 20, 2012 at 15:05
1

Have you tried specifying the cacheability of your responses to disallow caching by intermediate devices?

Cache-Control: no-cache

or

Cache-Control: private

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .