I am trying to set up a Wowza media server using a Dell R710 with 24GB memory and a 10Gbit connection and all the calculations say it should be able to handle around 20k connections however when I get to around 9000 connections, Wowza starts dropping packets because it deems them too late to deliver. I also notice a slowdown on the number of incoming connections.
At 9000 connections, the bandwidth out is about 3 - 4gbps. The stream is a live stream so there is no disk io.
The incoming connections are generated at a rate of 1 new connection every 3/4 second from 4 other servers in the same rack so there should be next to no latency and only a switch hooking them all up.
The one counter I notice starts to go up when the packets start dropping is
packets rejects in established connections because of timestamp
What does this counter mean and how can I fix it.
We have been through all the tcp settings and cannot seem to find the right combination to get more than 9000 connections without packet loss. We have also checked all the settings on the switch and router but cannot find the culprit.
OS is currently OpenSuse 11.4 and we have also tried Centos 5.6 previously with the same results.
We have also tried multiple instances of Wowza to spread the load between different JVM's but it is still the same result.