Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have a Graphite server to collect data through collectd, statsd, JMXTrans ... Since a few days, we frequently have holes in our data. Digging through the data we still have, we can see an increase in the carbon cache size (from 50K to 4M). We don't see an increase in the number of metrics collected (metricsReceived is stable at around 300K). We have an increase in the number of queries from 1000 to 1500 on average.

Strangely, the cpuUsage decreases slightly from 100% (we have 4 CPU) to 50% when the cache size increase.

Strangely again, we see an increase in the number if octets read from disk, and a decrease in the number of octets written.

We have carbon configure mostly with default values:

  • MAX_CACHE_SIZE = inf
  • MAX_UPDATES_PER_SECOND = 5000
  • MAX_CREATES_PER_MINUTE = 2000

Obviously, something has changed in our system, but we dont understand what, nor how we can find this cause ...

Any help ?

share|improve this question
    
I usually start from the the ground up approach to graphite issues; is there space on the disk to write to? Have the data directory permissions changed at all? Has there been a change in the daemon user collecting stats? Should there be no clear cause, it's entirely possible you have RRD corruption, and may need to find a way to export what you have, and start metric collection from scratch. –  Stephan Aug 28 '13 at 22:21
    
We checked disk space and permission, nothing strange there. No change in the daemon collecting data, maybe an increase in the number of metrics, but not that big. We're looking into WSP corruption. –  Guillaume Sep 1 '13 at 19:09

1 Answer 1

This is not a graphite stack's bug, but rather a IO bottleneck, most probably because your storage does not have the high enough IOPS. Because of this, the queue keeps building up, and overflows at 4M. At that point, You lose that much queued data, which is reflected later, as random 'gaps' in your graph. Your system cannot keep-up with the scale at which it is receiving metrics. It keeps filling up and overflowing.

Strangely, the cpuUsage decreases slightly from 100% (we have 4 CPU) to 50% when the cache size increase.

This is because your system begins swapping and the CPUs get a lot of 'idle time', because of the IO wait.

To add context, i have 500 provisioned IOPS at aws on a system on which i receive some 40K metrics. The queue is stable at 50K.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.