7

I would like some help finding the reason for the reduced data transfer rate in my application.

I have 12 embedded systems and a Linux server. The embedded systems send data to the server over TCP on an Ethernet link through a switch. The following is a TCP StreamGraph made from a Wireshark capture of the traffic from one board.

TCP StreamGraph

As you can see, the data transfer happens at around 5.8MBit/s up to around 0.25 seconds. This is as fast as I can expect the embedded system to go. After this, delays are inserted in the transfer. The following shows a closeup of the graph:

Zoomed StreamGraph

The staircase shaped curve at the bottom labeled ACK shows how much data has been ACKed by the server at any given time. The corresponding curve labeled RWIN shows how much there would be space for in the buffers on the datapc. The smaller vertical segments labeled SENT DATA are the actual packets sent.

At point A, the server acks the data as fast as it is sent, but then for a duration of 23ms no acks are sent by the server. The embedded system is allowed to send up to RWIN without waiting for an ACK, but it does not do so because it needs to keep the sent data around until they are acked (in case they need to be retransmitted) and the send buffer space is limited.

Then, at point B, all received data is ACKed at once and normal acking and sending resumes for 2.5ms before another pause happens.

The Wireshark capture was made from a different PC which was connected to a port on the switch that was set up to mirror all data sent and received on the port to which the embedded system was connected.

The Linux server runs a Java application which processes the data and stores them on disk. It shows no signs of having maxed out the CPU. The operating system is Ubuntu Server 12.04 with default network settings.

I can see that I could probably benefit from allocating more send buffer space in the embedded system to match the amount of receive window space in the Linux server, but this does not seem to be the limiting factor here.

My questions are:

  1. What could be the reason for the Linux server pausing the ACKs even though it is obviously able to receive everything just fine?
  2. How can I go about debugging this?
6
  • It could be a sign of congestion. Did you observe dup ACK ? If I understand well RWIN is the receiver's window (rwnd) : then can you also graph cwnd (the congestion window i.e. sender's window) ? Mar 17, 2015 at 14:12
  • No dup acks observed. Do you mean the sender's send window?
    – martinhans
    Mar 17, 2015 at 14:12
  • OK, I understand what you mean now. No, it looks like the cwnd is constant.
    – martinhans
    Mar 17, 2015 at 14:20
  • That's not the sending peer's window value in the TCP stream (which is rwnd) but an internal variable part of the TCP stack, different on each host, which means the amount of segments that can be sent without waiting for an ACK and that changes when congestion signs appear. You should find it using ss -int for instance. Mar 17, 2015 at 14:28
  • 2
    Given that I have the Wireshark capture, is there another way to answer your question about whether there is congestion? The sender must deduce this from the network traffic somehow.
    – martinhans
    Mar 17, 2015 at 14:49

2 Answers 2

1

Try to turn off Ethernet PAUSE frames with ethtool -A devname autoneg off rx off tx off

If that don't help, it can be a TCP windows scaling problem and/or an IRQ storming issue on the sending or receiver host. You can investigate both problems trying different settings with ethtool and sysctl entries regulating TCP traffic.

Without other informations, it is quite difficult to tell what's happening here...

0

The other obvious question is why do the clients stop sending? Normally the client wouldn't stop and wait for the ACK before sending the next TCP packet. Are they possibly sending single byte messages that are being delayed by Nagle's Algorithm?

https://en.wikipedia.org/wiki/Nagle%27s_algorithm

If they are and your Linux server is using TCP delayed acknowledgment then you could expect ACK delays of up to 500ms.

https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment

If this is the situation then it's easily fixed by using larger messages or disabling Nagle's Algorithm on the embedded systems (TCP_NODELAY).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .