Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The core of this problem is that our application uses websockets for real-time interfaces. We are testing our app in a new environment but strangely we're noticing an increasing delay in TCP websocket packets associated with an increase in websocket activity.

For example, if one websocket event occurs without any other activity in a 1-minute period, the response from the server is instantaneous. However, if we slowly increase client activity the latency in server response increases with a linear relationship (each packet will take more time to reach the client with more activity).

For those wondering this is NOT app-related since our logs show that our server is running and responding to requests in under 100ms as desired. The delay starts once the server processes the request and creates the TCP packet and sends it to the client (and not the other way around).

Architecture This new environment runs with a Virtual IP address and uses keepalived on a load balancer to balance the traffic between instances. Two boxes sit behind the balancer and all traffic runs through it. Our host provider manages the balancer and we do not have control over that part of the architecture.

Theory Could this somehow be related to something buffering the packets in the new environment?

Thanks for your help.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

Buffering sounds like a reasonable theory. I would take a packet capture from your app servers to make sure that you don't see anything like retransmits or other possibly abnormal behavior in the tcp stream (i.e. tcp window zeroing out?). Wireshark with a capture filter for the client IP would work for this.

If you verify that the packet capture looks clean, asking your provider to run a packet capture on their load balancer so you can analyize is a reasonable request.

Lastly, have you tested from multiple locations and different machines? Perhaps buffering is somewhere between the client and the provider, or something strange is happening with the client (running a packet capture on the client as well as your servers at the same time can be enlightening as well).

share|improve this answer
    
I have tested with multiple clients/connections but not multiple locations. However, the problem is non-present in the standard IP (old) environment. I am using Wireshark but didn't think about retransmits or window zeroing. I'm going to do a little more research and get back with a response after some testing :) thx! –  crockpotveggies Jun 3 '12 at 1:30
    
Well of the two answers yours is the most accurate. Turns out the firewall in our provider was designed to do a lot of bloated functions that just interfere with the websocket. thanks for the insight. –  crockpotveggies Jun 3 '12 at 23:02

It's actually an expected behavior. As the amount of data increases the size of the transmit window increases which, to a point, will increase the size of the packets sent. Larger packets mean more efficiency (less overhead and time taken for acknowledgements) but they also mean more latency. This is the tradeoff for maintaining reliable delivery while also maintaining any kind of throughput in networks of widely varying latencies and bandwidth.

Check out the PSH flag to cause TCP to flush buffers sooner. It might make a marginal difference under some circumstances. If you're looking for consistently low latency for lots of small messages, however, you might be better served looking at UDP - which means you'll have to account for assuring delivery on your own but you might realize a bit more consistency.

share|improve this answer
    
PSH might be a reasonable option since I think we can program that into the websocket server itself. My only thought towards UDP is I don't think it is supported by websockets (see stackoverflow.com/questions/4657033/…). Will respond soon once I can test some of this out! –  crockpotveggies Jun 3 '12 at 1:27
    
Well unfortunately I found out that the PSH flag was already being set by my websocket server. I'll have to look deeper. The UDP was a great idea but unfortunately it isn't part of the websocket spec. –  crockpotveggies Jun 3 '12 at 5:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.