0

I've installed Ganglia on my Ubuntu 16.04 servers, recently. I basically followed the instructions here.

The problem is, rrd files for sets of the metrics for some servers are missing.

Most importantly, there are no metrics from the "master" node itself other than the GPU metrics from the Nvidia plugin.

Can somebody give me any advice on this issue?

Also, is there any way to force collecting all the metrics defined at least once immediately?

Thanks!

1 Answer 1

0

I figured out the issue and the solution.

man gmond.conf says:

If you have a large system with lots of metrics, you might experience UDP drops. This happens when gmond is not able to process the UDP fast enough from the network.
In this case you might consider changing your setup into a more distributed setup using aggregator gmond hosts.  Alternatively you can choose to create a bigger
receive buffer:

  udp_recv_channel {
    port = 6343
    buffer = 10485760
  }
B<buffer> is specified in bytes, i.e.: 10485760 will allow 10MB UDP
to be buffered in memory.

Note: increasing buffer size will increase memory usage by gmond

I increased the maximum size of UDP receive buffer to 100MB by adding net.core.rmem_max=104857600 to /etc/sysctl.conf (for permanent effect) and running sysctl -w net.core.rmem_max=104857600 which allowed me to modify the value without rebooting.

And then, I added a buffer attribute with a value of 104857600 to the udp_recv_channel block and restarted gmond.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .