2

We are running a distributed system of java servers (os: linux) doing lot of computation with communication over TCP. While our traffic pattern in not bursty, on some machines we see a fluctuation in the network bandwidth usage as given below:

    02:56:32 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
    02:56:33 PM        lo     61.00     61.00      8.69      8.69      0.00      0.00      0.00
    02:56:33 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

    02:56:33 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
    02:56:34 PM        lo    107.00    107.00     13.70     13.70      0.00      0.00      0.00
    02:56:34 PM      eth0  15514.00  15794.00   8036.93   7148.15      0.00      0.00      0.00

    02:56:34 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
    02:56:35 PM        lo     59.00     59.00      8.85      8.85      0.00      0.00      0.00
    02:56:35 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    [pattern continues] 

Essentially, the bandwidth usage fluctuates between 0-8MB/sec. We haven't had been able to figure out the cause of such fluctuations.

Any pointers/suggestions would be of great help.

Edit 1: We have TCPNODELAY set to true.

Edit 2: The Java ParNew GC runs every other second on these machines.

Edit 3: We are running only a single java process.

Edit 4: We are running with +XX:+DisableExplicitGC

6
  • What tool are you using to generate this logging? It may just be a quirk in the tool. (In fact, it is almost certainly is.) Jan 28, 2013 at 23:13
  • I am using sar -n DEV 1 10. Is there another better tool.
    – Sumit
    Jan 28, 2013 at 23:35
  • The sar command reads /proc/net/dev, which should be accurate on any Linux kernel that isn't ancient. Jan 29, 2013 at 0:35
  • yes and we are using centos 6
    – Sumit
    Jan 29, 2013 at 1:03
  • Try NetHogs ( yum install nethogs ) - it will show you the bandwidth usage per process.
    – Daniel t.
    Jan 29, 2013 at 20:10

1 Answer 1

0

I think this is your clue:

The Java ParNew GC runs every other second on these machines.

Your Java application communicates via the network. Every alternate second you are pausing the Java application to run Garbage Collection. You are only seeing network traffic every alternate second, implying the application is only running every alternate second.

Seems your network traffic pattern is exactly as-expected?

Running Garbage Collection every two seconds certainly isn't helping you. If the JVM is running this itself, then you need a larger heap size. If you are requesting this collection with System.gc() or Runtime.gc() then try not requesting it so often and see if the traffic pattern follows the GC pauses.

Getting into troubleshooting of your specific app's Garbage Collection is well beyond the scope of an answer on this site, but I'd suggest to pick some metric which is important to your application and measure that metric whilst trying out different heap sizes, different GC request intervals, and the different GCs in your JVM. There are plenty of articles out there on using garbagecat to analyse GC performance.

Don't worry too much about frequency of pauses or number of pauses or length of pauses. Use the Garbage Collector and settings which result in the best metrics for what matters to your application.

I think you'll find a lot of Java engineers who would say don't manually request garbage collection at all. JVM developers put heaps of effort into tuning the GCs to be automatically smart. They know more than mere mortals like you or I ever will. Trust them.

1
  • This could have explained the behavior if the duration of ParNew GC runs was 1sec but is it only 250ms, i.e, ParNew GC runs every other second but when it runs it runs only for 250ms. Since the ParNew GC is running for (substantially) less than a second why does the bandwidth usage drops to zero and not to say 75% of bandwidth usage in the previous second?
    – Sumit
    Aug 3, 2013 at 5:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .