Since last night I am seeing CPU load spikes on our cassandra boxes(Occasionally load goes up to 20, its a Amazon EC2 c1.xlarge with 300 iops EBS). After digging around a little I believe its related to heap memory and flushing memtables.
From logs: WARN 03:22:03,414 Heap is 0.7786981388910019 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN 03:22:03,415 Flushing CFS(Keyspace='XXX', ColumnFamily='XXXX') to relieve memory pressure
I have three nodes and only 2 of them are hitting this high load, moreover cluster is under extremely light load, no one is using it since yesterday and I still see this load.
I also observed that
top -H showed many threads in Sleep state and only a handful in R state.
nodetool cfstat showd the following for the ColumnFamily in the above stated cassandra logs:
Column Family: XXXX SSTable count: 8 Space used (live): 1479005837 Space used (total): 1479005837 Number of Keys (estimate): 2923008 Memtable Columns Count: 35375 Memtable Data Size: 7088479 Memtable Switch Count: 2393 Read Count: 2339668632 Read Latency: 3.042 ms. Write Count: 360448535 Write Latency: 0.079 ms. Pending Tasks: 0 Bloom Filter False Positives: 143197 Bloom Filter False Ratio: 0.73004 Bloom Filter Space Used: 7142048 Compacted row minimum size: 73 Compacted row maximum size: **785939** Compacted row mean size: 1957
Compacted Row maximum size for other ColumFamily is significantly less than this number. When starting this cluster we set
We are using cassandra 1.1.0 and open-6-jdk
Can any one please help me understand whey with no load on the system I am still seeing such high load on my machines.