We are running a distributed system of java servers (os: linux) doing lot of computation with communication over TCP. While our traffic pattern in not bursty, on some machines we see a fluctuation in the network bandwidth usage as given below:

    02:56:32 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
    02:56:33 PM        lo     61.00     61.00      8.69      8.69      0.00      0.00      0.00
    02:56:33 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

    02:56:33 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
    02:56:34 PM        lo    107.00    107.00     13.70     13.70      0.00      0.00      0.00
    02:56:34 PM      eth0  15514.00  15794.00   8036.93   7148.15      0.00      0.00      0.00

    02:56:34 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
    02:56:35 PM        lo     59.00     59.00      8.85      8.85      0.00      0.00      0.00
    02:56:35 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    [pattern continues] 

Essentially, the bandwidth usage fluctuates between 0-8MB/sec. We haven't had been able to figure out the cause of such fluctuations.

Any pointers/suggestions would be of great help.

Edit 1: We have TCPNODELAY set to true.

Edit 2: The Java ParNew GC runs every other second on these machines.

Edit 3: We are running only a single java process.

Edit 4: We are running with +XX:+DisableExplicitGC

  • What tool are you using to generate this logging? It may just be a quirk in the tool. (In fact, it is almost certainly is.) – David Schwartz Jan 28 '13 at 23:13
  • I am using sar -n DEV 1 10. Is there another better tool. – Sumit Jan 28 '13 at 23:35
  • The sar command reads /proc/net/dev, which should be accurate on any Linux kernel that isn't ancient. – David Schwartz Jan 29 '13 at 0:35
  • yes and we are using centos 6 – Sumit Jan 29 '13 at 1:03
  • Try NetHogs ( yum install nethogs ) - it will show you the bandwidth usage per process. – Daniel t. Jan 29 '13 at 20:10

I think this is your clue:

The Java ParNew GC runs every other second on these machines.

Your Java application communicates via the network. Every alternate second you are pausing the Java application to run Garbage Collection. You are only seeing network traffic every alternate second, implying the application is only running every alternate second.

Seems your network traffic pattern is exactly as-expected?

Running Garbage Collection every two seconds certainly isn't helping you. If the JVM is running this itself, then you need a larger heap size. If you are requesting this collection with System.gc() or Runtime.gc() then try not requesting it so often and see if the traffic pattern follows the GC pauses.

Getting into troubleshooting of your specific app's Garbage Collection is well beyond the scope of an answer on this site, but I'd suggest to pick some metric which is important to your application and measure that metric whilst trying out different heap sizes, different GC request intervals, and the different GCs in your JVM. There are plenty of articles out there on using garbagecat to analyse GC performance.

Don't worry too much about frequency of pauses or number of pauses or length of pauses. Use the Garbage Collector and settings which result in the best metrics for what matters to your application.

I think you'll find a lot of Java engineers who would say don't manually request garbage collection at all. JVM developers put heaps of effort into tuning the GCs to be automatically smart. They know more than mere mortals like you or I ever will. Trust them.

  • This could have explained the behavior if the duration of ParNew GC runs was 1sec but is it only 250ms, i.e, ParNew GC runs every other second but when it runs it runs only for 250ms. Since the ParNew GC is running for (substantially) less than a second why does the bandwidth usage drops to zero and not to say 75% of bandwidth usage in the previous second? – Sumit Aug 3 '13 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.