We are running a distributed system of java servers (os: linux) doing lot of computation with communication over TCP. While our traffic pattern in not bursty, on some machines we see a fluctuation in the network bandwidth usage as given below:
02:56:32 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
02:56:33 PM lo 61.00 61.00 8.69 8.69 0.00 0.00 0.00
02:56:33 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:56:33 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
02:56:34 PM lo 107.00 107.00 13.70 13.70 0.00 0.00 0.00
02:56:34 PM eth0 15514.00 15794.00 8036.93 7148.15 0.00 0.00 0.00
02:56:34 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
02:56:35 PM lo 59.00 59.00 8.85 8.85 0.00 0.00 0.00
02:56:35 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[pattern continues]
Essentially, the bandwidth usage fluctuates between 0-8MB/sec. We haven't had been able to figure out the cause of such fluctuations.
Any pointers/suggestions would be of great help.
Edit 1: We have TCPNODELAY set to true.
Edit 2: The Java ParNew GC runs every other second on these machines.
Edit 3: We are running only a single java process.
Edit 4: We are running with +XX:+DisableExplicitGC
sar -n DEV 1 10
. Is there another better tool.sar
command reads/proc/net/dev
, which should be accurate on any Linux kernel that isn't ancient.