3

I'm observing the below behavor in a TCP connection between a server application and a client which run on two different Red Hat machines.

Example 1

  1. Client acks everything until now (ACK with WND = 1424)
  2. Server sends 296 bytes to TCP layer and it is sent to the client(PUSH/ACK)
  3. Server sends 296 bytes to TCP layer and it is sent to the client(PUSH/ACK)
  4. Server sends 296 bytes to TCP layer and it is sent to the client(PUSH/ACK)
  5. Server sends/tries to send 239 bytes to TCP layer but: either one of the following two happens
    1. the OS waits ~30ms until ACK from client is received before sending these bytes (this x bytes send will be referred as delayed message below)
    2. the tcp send call returns EWOULDBLOUCK or EAGAIN (I think this is not possible because 239+239+239+239 < 1424. Please correct me if I'm wrong

Example 2

  1. Client acks everything until now (ACK with WND=1424)
  2. Server sends 239 bytes to TCP layer and it is sent to the client
  3. Server sends/tries to send 239 bytes to TCP layer and one of the two things in the above happens

The problem I want to understand is why the server application or the OS waits until the ACK when window size for the client is big enough? Is this something I can tune in the OS level. Can I force server to send all bytes until the window of the client is filled?

As per my understanding there are two possible reasons

  1. Server application does not pass the 'delayed message' to TCP level becuase the send system call returned EWOULDBLOUCK or EAGAIN.
  2. Application does pass the message to TCP layer, but the operating system does not send it until an ACK from the client is received.

In all the delayed message scenarios, there is an ACK from the client before the SEND of the delayed message. Also the working logic of the application suggests that it does pass the message to the TCP layer.

Can you please suggest where I should focus on the troubleshooting of this.

  1. Is it some OS level setting I can set in either the client or server side
  2. Is this somehow a fault in the application server?
  3. Am I wrong in assuming OS doesn't return EWOULDBLOCK until messages fill up the window

edit: Reason for duplicate: I was not sure where the issue is; network/os. I have added aditional details. Also, I need a POV from TCP implementation of linux so I think this question belongs here.

Improve this question
2
3

This is called “Delayed ACK” and it’s the way TCP protocol works. See https://en.m.wikipedia.org/wiki/TCP_delayed_acknowledgment for more details. You can disable this feature if it affects your applications’s performance. Say iSCSI has delayed ACK turned off by default.

Improve this answer
4
  • But I have set TCP_NO_DELAY in the server application box – PasanW Oct 6 '19 at 2:32
  • I have set tcp_low_latency=1 – PasanW Oct 6 '19 at 8:39
  • 1
    TCP_NODELAY is for Nagle algorithm, it has nothing to do with delayed ACK, these are two different things. See extrahop.com/company/blog/2016/… for more details. – RiGiD5 Oct 6 '19 at 9:21
  • 1
    This is distro-dependent. If you however control application’s source code you can play with these settings independently of your environment. See stackoverflow.com/questions/1615447/disable-tcp-delayed-acks for similar case. – RiGiD5 Oct 6 '19 at 9:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.