Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

There is a server for measurements running on Ubuntu in my lab . And there is C program, which receives data through TCP connection and should as soon as possible send a reply. Network latency when receiving packets is very critical (tens of microseconds are important). I optimized the program to the maximum, but I have no experience tweaking Ubuntu.

What can be configured in Ubuntu to reduce the local delay of processing/sending packets?

UPDATE:

Network equipment is usual NIC: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller / Intel Corporation 82546EB Gigabit Ethernet Controller

Data info: Data blocks come approx. each 10 milliseconds. Data block size is approx. 1000byte.

UPDATE 2:

CPUs: 2 processors x 4 cores - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz

RAM: 12 GB

Network switch: Cisco Catalyst 2960

share|improve this question
    
Yes, I'd like to know the server's make/model. –  ewwhite Aug 25 at 14:19
    
you should dig way deeper. read some stuff about kernel optimisation for high frequency trading. F.e. a sales Cisco Paper: cisco.com/c/dam/en/us/products/collateral/switches/… so get a decent PCI-E Card on both sides will save some aswell. Most likely (depending on how much time you want to spend on this) you will rebuild at least the kernel with different settings, removing a lot of stuff ubuntu does need but you don't. So as ewwhite did write in the comments, ubuntu might not be perfect for the lowest settings. –  Dennis Nolte Aug 25 at 15:29
    
With the hardware listed, it's 2008-era equipment (Intel 5300-series CPUs). Back then, there weren't too many special low-latency hardware changes possible. I'd set the system BIOS to run in high-performance mode and disable CPU C-States. –  ewwhite Aug 25 at 20:36
    
@ewwhite Yes, you are right about 2008-era equipment. I will try your suggestions. Thank you! –  Alex V Aug 25 at 21:48
    
Any ability to tweak this software for TCP_NODELAY? –  Matt Aug 25 at 22:38

2 Answers 2

up vote 4 down vote accepted

Honestly, I wouldn't be using Ubuntu for this... but there are options that can be applied to any Linux variant.

You'll want to increate your network stack buffers:

net.core.rmem_default = 10000000
net.core.wmem_default = 10000000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

If the application is writing to disk, maybe a scheduler/elevator change would be necessary (e.g. the deadline elevator).

At the server level, you can modify the CPU governor and power and CPU frequency management (P-States, C-States).

At the OS level, you can change the realtime priority of your application (chrt), optimizing to reduce interrupts, pinning it to a CPU or group of CPUs (taskset), and stopping any unnecessary services or daemons.

You can also see some suggestions at: How to troubleshoot latency between 2 linux hosts

It's difficult to get more specific without knowing the hardware or networking equipment involved.

share|improve this answer
1  
This isn't really the appropriate venue for religious debates. Take it elsewhere, such as chat. –  Michael Hampton Aug 25 at 22:56
1  
@MichaelHampton There were interesting links in discussion related to the question: Red Hat Realtime Tuning Guide. –  Alex V Aug 26 at 6:22

If you're going the route of high performance typically you'll want to run as few other (scheduled) processes as possible as they'll interfere with your application.

Linux, like the classical UNIX operating systems, is designed to run multiple applications concurrently in a fair way and tries to prevent resource starvation and you'll be aiming for the opposite, starve everything else except your application. Simple steps at the OS level are changing the nice level and real time priority of your application, changing the scheduler or going for a real-time kernel.

TCP/IP is typically tuned to prevent connection drops and make efficient use of the bandwidth available. To get the lowest latency possible out of a very fast link, rather than getting the highest bandwidth possible out of a a connection where some intermediate links are more constrained, you're going to adjust the tuning of the network stack.

 sysctl -a 

will show you a host of kernels settings you can tune. Settings depend on whether or not you're using IPv4 or IPv6 and what exactly you already do in your application but of interest may be:

  • net.ipv4.tcp_window_scaling=1 RFC 1323 - support for IPV4 TCP window sizes larger than 64K - generally needed on high bandwidth networks
  • net.ipv4.tcp_reordering=3 The maximum times an IPV4 packet can be reordered in a TCP packet stream without TCP assuming packet loss and going into slow start.
  • net.ipv4.tcp_low_latency=1 intended to give preference to low latency over higher throughput; setting =1 disables IPV4 tcp prequeue processing
  • net.ipv4.tcp_sack=0 setting to 1 enables selective acknowledgment for IPV4, which requires enabling tcp_timestamps and adds some packet overhead, which you don't need if you don't experience packetloss
  • net.ipv4.tcp_timestamps=0 Only advised in cases where sack is needed.
  • net.ipv4.tcp_fastopen=1 Enable to send data in the opening SYN packet.

Most if not all are documented better in the kernel source.

You can of course code raw TCP sockets and largely by-pass the kernel TCP/IP stack altogether.

Often highly tuned systems run in a trusted network and will have their local (iptables) firewalls disabled.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.