It all depends on how the app is coded, what it does and so on. Recent Linux versions have included new APIs to boost high throughput, such as splice/vmsplice, introduced in 2.6.17. But it obviously requires your app to be use them. All recent versions of modern distros will have at least that kernel anyway.
Splice basically allows you to pass data between file descriptors without copying it. You just tell the OS, take that many bytes from there and send them there. It reduces data copying and context switching since it all gets done in kernel.
Applications such as haproxy can use this call to fill up a dual 10 gigabit ethernet card on a low end server.
Now splice/vmsplice/tee are not going to make much difference if your app is not spending most of its time copying data around. But there are other syscalls you want to investigate to handle lots of simultaneous connecitons.
For instance make sure you use epoll instead of select/poll. Where the latter send a list of all the FDs they're interested in at every call, epoll_create builds a list in the kernel, and the epol_wait call just references that list. You can see the advantage when you're listening to thousands of sockets, and you call select/poll ten thousand times a second.
Beyond that you want to look at run-time sysctl settings. The defaults are usually way too conservative.
Finally, I would definitely recommend you use a subscription/support-based distro such as RHEL or SLES; the kind of support service they provide is exactly what you need, in that they will help you if there's a performance regression in the kernel, and they will be able to get the fix quickly in.