I figured out the issue and the solution.
man gmond.conf
says:
If you have a large system with lots of metrics, you might experience UDP drops. This happens when gmond is not able to process the UDP fast enough from the network.
In this case you might consider changing your setup into a more distributed setup using aggregator gmond hosts. Alternatively you can choose to create a bigger
receive buffer:
udp_recv_channel {
port = 6343
buffer = 10485760
}
B<buffer> is specified in bytes, i.e.: 10485760 will allow 10MB UDP
to be buffered in memory.
Note: increasing buffer size will increase memory usage by gmond
I increased the maximum size of UDP receive buffer to 100MB by adding net.core.rmem_max=104857600
to /etc/sysctl.conf
(for permanent effect) and running sysctl -w net.core.rmem_max=104857600
which allowed me to modify the value without rebooting.
And then, I added a buffer
attribute with a value of 104857600 to the udp_recv_channel
block and restarted gmond
.