1

I have a Tomee 1.5.1 machine running on AWS m1.medium instance with Windows Server 2008.

I put on the server a static .jpg file of 2.5 MB, and it takes me around 25 seconds to download it on my machine with Chrome.

When observing the traffic in Wireshark, I see that receiver the Window Size is constant on 64 KBbytes, when the receiver window is 256 and the scaling factor is 256 (shift left of 8).

enter image description here

The typical exchange is:

capture

Factors eliminated:

  1. The server is overloaded - it is not.

  2. The server has low bandwidth - it is not, at least when the connection is initialized from the server. When running ookla speedtest on the server machine to some benchmark providers around my location is shows up/downlink of around 60 Mbps.

3.The receiver is slow/misconfigured - it is not - the same file gets downloaded from some file sharing server in 2 sec. The window size jumps as traffic/congestion increases as it should and is much larger.

enter image description here

Edit 1

Disabling heuristics on the cliend didn't help

Edit 2

Additional question is why the sender seems to send the data is series of 1460*6+240=9000 bytes, and to have not more then one "outstanding" series, that means he sends the new bunch once he got ack to the one-before-previous bunch

enter image description here

Edit 3:

It turns out, the http connector has a setting socketBuffer - "The size (in bytes) of the buffer to be provided for socket output buffering. -1 can be specified to disable the use of a buffer. By default, a buffers of 9000 bytes will be used." - These are exactly 9000 bytes we see of the outgoing batches.

Likely, the tcp driver stores the outgoing packets in the buffer to retransmit them in the case of need, until it gets the ack, and for some reason the client acks are getting back in batches, hence the batching of the outgoing traffic.

If we change it to 4MB, the batching increases to 64KB...

If the traffic is not SSL encrypted. If the SSL is on, the batching is 16kb.

Need to check out the following:

  1. What is the optimal out buffer size on connections with low ping? I think about 64k can be ok.
  2. How is SSL output buffer configured
  3. Try newer version of tomee - there are updates to the http connector
  4. Check out the NIO connector
  5. Why the ACKs are arriving in batches?

Edit 4:

  1. The buffer size of 64k makes a little difference. I settled with 128k to be on the safe side.

  2. If we use Tomee 7 on Windows Server 2016, the SSL works as fast as regular http traffic. I don't know for now when between tomcat5-win2008 and tomcat7-win2016 the change occurred.

  • Hmm, the arrows in that picture are a bit confusing, but yes the batching is interesting. BTW: haveyou consiered updating 2008 to a more recent OS with new network stacks? – eckes May 11 '17 at 18:28
  • 1
    I think the rather small bursts of 7 data ACK packages with a terminal PSH are created by the server application (in your case a Servlet) which regularly flushes the data (or writes in application buffers of size 9k). This does not use the whole window. So maybe it helps to write with larger byte arrays into the servletoutputstream or crank up the buffers used (why it would flush is beyond me, but something to look for!). On the client side it looks like it is waiting very long and only send ACKs when the PSH is received. Here it would probably help to reduce the receive buffers. – eckes May 11 '17 at 18:47
  • 1
    Yeah, it seems that the sender buffer is indeed small - looks like it's 18k. Maybe i'll try to look into tweaking the send buffer. Similar SO question: stackoverflow.com/questions/10289604/tcp-receive-window. I am considering to leave AWS for some other hosting provider, and maybe also change the OS, instead of getting into TCP optimization. – alex440 May 12 '17 at 21:24
1

With Windows there seem to be two things discussed, first of all check your current configuration:

netsh int tcp show global

It is often suggested to turn off auto tuning, however this will also turn off scaling. So using restricted seems to actually be better for scaling. Turning the auto tuning off will also disable scaling, so don't do it.

Also some people report that disabling the window scaling heuristic on the client makes the feature be requested more consistently. The 256*255 seems to be a typical combination when Windows underestimates the link quality.

Also check out this article on PSH optimizations on client and server side (especially for Windows). It does look to me from your sequence diagrams that the client waits too long to ack packets (waiting for the PSH from the server). http://smallvoid.com/article/winnt-tcp-push-flag.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.