How can I see why my Riak cluster is not running the compaction phase on the machines when I expect?

Question

I have a cluster of Riak (V1.2.0) machines, each with about 50Gb of hard disk space. We have the bitcask storage configure in the following way:

%% Bitcask Config
{bitcask, [
         {data_root, "/var/lib/riak/bitcask"},
         {frag_merge_trigger, 40}, %% trigger merge if framentation is > 40% default is 60%
         {dead_bytes_merge_trigger, 67108864}, %% trigger if dead bytes for keys > 64MB default is 512MB
         {frag_threshold, 20}, %% framentation >= 20% default is 40
         {dead_bytes_threshold, 67108864} %% trigger if dead bytes for data > 64MB default is 128MB
       ]},

Our software has a pattern where the values are being rewritten frequently with the same keys. IE, the set of keys remain relatively stable while the values change at least once a day.

The pattern that we are seeing is that the available disk space shrinks each day until the other services on those machines stop operating because of out-of-space errors. Sometimes a restart on the box will reclaim some space.

Is there something wrong with my config? I'm not really sure how to set it up to reclaim the space 'automatically' enough that I don't run out of disk space within about a week.

How can I check?

• When the compaction algorithm determines to run or not and what it decides based on what criteria?
• How a compaction was actually run
• How much space was reclaimed

Answer 1

Add one line to your bitcask configuration

{log_needs_merge, true}

Whenever there is a merge, there will be an entry in the console.log file that shows which files met which triggers