Low latency TCP settings on Ubuntu

Question

There is a server for measurements running on Ubuntu in my lab. And there is C program, which receives data through TCP connection and should as soon as possible send a reply.

Configuration

• CPUs: 2 processors x 4 cores - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
• RAM: 12 GB
• NIC: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller / 82546EB Gigabit Ethernet Controller
• Network switch: Cisco Catalyst 2960
• Data info: Data blocks come approx. each 10 milliseconds. Data block size is approx. 1000 byte.

Network latency when receiving packets is very critical (tens of microseconds are important). I optimized the program to the maximum, but I have no experience tweaking Ubuntu.

What can be configured in Ubuntu to reduce the local delay of processing/sending packets?

Answer 1

Honestly, I wouldn't be using Ubuntu for this... but there are options that can be applied to any Linux variant.

You'll want to increate your network stack buffers:

net.core.rmem_default = 10000000
net.core.wmem_default = 10000000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

If the application is writing to disk, maybe a scheduler/elevator change would be necessary (e.g. the deadline elevator).

At the server level, you can modify the CPU governor and power and CPU frequency management (P-States, C-States).

At the OS level, you can change the realtime priority of your application (chrt), optimizing to reduce interrupts, pinning it to a CPU or group of CPUs (taskset), and stopping any unnecessary services or daemons.

You can also see some suggestions at: How to troubleshoot latency between 2 linux hosts

It's difficult to get more specific without knowing the hardware or networking equipment involved.

Answer 2

If you're going the route of high performance typically you'll want to run as few other (scheduled) processes as possible as they'll interfere with your application.

Linux, like the classical UNIX operating systems, is designed to run multiple applications concurrently in a fair way and tries to prevent resource starvation and you'll be aiming for the opposite, starve everything else except your application. Simple steps at the OS level are changing the nice level and real time priority of your application, changing the scheduler or going for a real-time kernel.

TCP/IP is typically tuned to prevent connection drops and make efficient use of the bandwidth available. To get the lowest latency possible out of a very fast link, rather than getting the highest bandwidth possible out of a a connection where some intermediate links are more constrained, you're going to adjust the tuning of the network stack.

 sysctl -a 

will show you a host of kernels settings you can tune. Settings depend on whether or not you're using IPv4 or IPv6 and what exactly you already do in your application but of interest may be:

• net.ipv4.tcp_window_scaling=1 RFC 1323 - support for IPV4 TCP window sizes larger than 64K - generally needed on high bandwidth networks
• net.ipv4.tcp_reordering=3 The maximum times an IPV4 packet can be reordered in a TCP packet stream without TCP assuming packet loss and going into slow start.
• net.ipv4.tcp_low_latency=1 intended to give preference to low latency over higher throughput; setting =1 disables IPV4 tcp prequeue processing
• net.ipv4.tcp_sack=0 setting to 1 enables selective acknowledgment for IPV4, which requires enabling tcp_timestamps and adds some packet overhead, which you don't need if you don't experience packetloss
• net.ipv4.tcp_timestamps=0 Only advised in cases where sack is needed.
• net.ipv4.tcp_fastopen=1 Enable to send data in the opening SYN packet.

Most if not all are documented better in the kernel source.

You can of course code raw TCP sockets and largely by-pass the kernel TCP/IP stack altogether.

Often highly tuned systems run in a trusted network and will have their local (iptables) firewalls disabled.