If you're going the route of high performance typically you'll want to run as few other (scheduled) processes as possible as they'll interfere with your application.
Linux, like the classical UNIX operating systems, is designed to run multiple applications concurrently in a fair way and tries to prevent resource starvation and you'll be aiming for the opposite, starve everything else except your application. Simple steps at the OS level are changing the nice level and real time priority of your application, changing the scheduler or going for a real-time kernel.
TCP/IP is typically tuned to prevent connection drops and make efficient use of the bandwidth available. To get the lowest latency possible out of a very fast link, rather than getting the highest bandwidth possible out of a a connection where some intermediate links are more constrained, you're going to adjust the tuning of the network stack.
will show you a host of kernels settings you can tune. Settings depend on whether or not you're using IPv4 or IPv6 and what exactly you already do in your application but of interest may be:
net.ipv4.tcp_window_scaling=1 RFC 1323 - support for IPV4 TCP
window sizes larger than 64K - generally needed on high bandwidth networks
net.ipv4.tcp_reordering=3 The maximum times an IPV4 packet can
be reordered in a TCP packet stream without TCP assuming packet loss
and going into slow start.
to give preference to low latency over higher throughput; setting =1
disables IPV4 tcp prequeue processing
net.ipv4.tcp_sack=0 setting to 1
enables selective acknowledgment for IPV4, which requires enabling
tcp_timestamps and adds some packet overhead, which you don't need if
you don't experience packetloss
net.ipv4.tcp_timestamps=0 Only advised in cases where sack is needed.
net.ipv4.tcp_fastopen=1 Enable to send data in the opening SYN packet.
Most if not all are documented better in the kernel source.
You can of course code raw TCP sockets and largely by-pass the kernel TCP/IP stack altogether.
Often highly tuned systems run in a trusted network and will have their local (iptables) firewalls disabled.