37

I've been looking at Linux tuning params and see some configs where SACK is turned off. Can anyone explain this?

This would be tuning for a busy web server.

3 Answers 3

41

A basic TCP ACK says "I received all bytes up to X." Selective ACK allows you to say "I received bytes X-Y, and V-Z."

So, for instance, if a host sent you 10,000 bytes and bytes 3000-5000 were lost in transit, ACK would say "I got everything up to 3000." The other end would have to send bytes 3001-10000 again. SACK could say "I got 1000-2999, and 5001-10000" and the host would just send the 3000-5000.

This is great over a high bandwidth, lossy (or high delay) link. The problem is that it can cause severe performance issues in specific circumstances. Normal TCP ACKs will make the server treat a high-bandwidth, lossy connection with kid gloves (send 500 bytes, wait, send 500 bytes, wait, etc). SACK lets it adapt to the high delay because it knows exactly how many packets were actually lost.

Here is where bad things can happen. An attacker can force your server to keep a massive retransmission queue for a long time, then process that whole damn thing over and over and over again. This can peg the CPU, eat up RAM, and consume more bandwidth than it should. In a nutshell, a lightweight system can initiate a DoS against a beefier server.

If your server is robust and doesn't serve large files, you're pretty well insulated against this.

If you're mostly serving an intranet or other low-latency group of users, SACK buys you nothing and can be turned off for security reasons with no performance loss.

If you're on a low-bandwidth link (say 1Mbps or less as a completely arbitrary rule of thumb), SACK can cause problems in normal operations by saturating your connection and should be turned off.

Ultimately, it's up to you. Consider what you're serving, to whom, from what, and weigh the degree of your risk against the performance effects of SACK.

There is a great overview of SACK and its vulnerability here.

3
  • FTR: since Linux 4.18 there is SACK compression enabled. It could e.g. improve performance on wireless networks. Also, somewhat relevant: original developer's comment.
    – Hi-Angel
    Jun 30, 2018 at 17:36
  • 1
    It seems to me that most answers here are outdated. This is enabled by default now in linux, so pretty much every linux server out there is using SACK, if that caused issues it would be found already
    – Freedo
    May 18, 2021 at 3:57
  • @Freedo Do you have something to support your claim? I was just noticing that Cloudflare and Sucuri don't seem to have SACK enabled.
    – Paul
    Dec 30, 2021 at 22:56
12

Another reason that TCP SACK is often disabled is that there is an amazing amount of network gear out there that fails to handle this option correctly. We see this all the time with a high-speed file transfer product that we provide that uses TCP. The most common issue is that of gateway devices that do things like randomize sequence numbers for TCP packets transiting through the device from internal networks to external, but that don't "un-randomize" the TCP SACK options that might be sent from the remote end. If the actual SACK values are not translated back to the proper values by these devices, then then TCP session will never complete in the face of packet loss when the remote end tries to use SACK to get the selective ACK benefits.

Probably this would be less of an issue if people were to more aggressively apply preventive software maintenance to this gear, but they tend not to.

1
  • 2
    See this RedHat KB Article: Why do TCP connections from a client system behind an ADSL router hang intermittently under Red Hat Enterprise Linux? kbase.redhat.com/faq/docs/DOC-26683
    – gm3dmo
    Mar 22, 2010 at 7:53
7

I can confirm from bitter experience that tcp_sack = 1 causes stalled data transfer over sftp/rsync/scp etc with files in excess of around 12mb when using certain Cisco ASA firewall appliances.

EVERY Time it would be stalled.

We were transferring over a dedicated 100mbps link between host A and host B in two different data centres, both using cisco firewall and switch hardware with centos.

This can be mitigated somewhat by modifying buffer sizes - e.g. I could not transfer 1GB file via sftp from host A to host B unless I set the sftp buffer to 2048, but I could regardless if host B was pulling the file from A.

Experiments with the same file using rsync and send/receive buffer tuning allowed me to get up yo around 70mb of a 1GB file pushed from A to B.

However, the ultimate answer was to disable tcp_sack on host A. Initially by setting tcp_sack = 0 in the kernel on-the-fly - but ultimately - I added it to my /etc/sysctl.conf

2
  • 2
    fwiw, Cisco ASA Firewall here also. The unidirectional nature of the problem was disconcerting, we've been chasing this for months. scp worked more or less 'at speed' one way, but regularly stalled and timed out the other direction. Disable tcp_sack was a cure.
    – user371274
    Aug 18, 2016 at 21:43
  • @jean-loup I hope you are not suggesting to change equipment. I had this problem in the last job, and fixed it with configuration changes. unix.stackexchange.com/questions/391125/… Jun 19, 2019 at 15:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .