I figured out the issue and the solution.
man gmond.conf says:
If you have a large system with lots of metrics, you might experience UDP drops. This happens when gmond is not able to process the UDP fast enough from the network.
In this case you might consider changing your setup into a more distributed setup using aggregator gmond hosts. Alternatively you can choose to create a bigger
port = 6343
buffer = 10485760
B<buffer> is specified in bytes, i.e.: 10485760 will allow 10MB UDP
to be buffered in memory.
Note: increasing buffer size will increase memory usage by gmond
I increased the maximum size of UDP receive buffer to 100MB by adding
/etc/sysctl.conf (for permanent effect) and running
sysctl -w net.core.rmem_max=104857600 which allowed me to modify the value without rebooting.
And then, I added a
buffer attribute with a value of 104857600 to the
udp_recv_channel block and restarted