14

I've got a one-node Cassandra cluster which is currently stuck in a major compaction process. After executing a nodetool compactit started the compaction and I see the tmp-sstable files with a size of 0 bytes. But that's all. No progress for hours.

I already stopped the Cassandra service, but after a restart it continued the compaction again without progress. A nodetool compactionstats shows 0,00% progress and a remaining time of six minutes. But I already waited 24 hours. In the meantime I stopped all readers and writers without seeing any difference.

The version I use is 2.1.2. I already tried with OpenJDK 7 and with Oracle's Java 7, but no difference.

The logs show no indication of OOM or any other exception or warning.

I'm not sure if it is of any help, but I currently migrate away from Cassandra as it doesn't fit my needs. So I delete a lot of data which has been migrated. There are many rows with several hundred up to a few thousand columns. But there are a handful rows with a million of columns. They now have a lot of tombstones in them due to the migration (deletion) process.

Any help in how to debug the issue are welcome.

  • just an idea, have you looked at /var/log/syslog or where would be logged something like to many files open/VM maxmem? – Antony Gibbs Dec 2 '14 at 0:11
  • @AntonyGibbs Nothing like that. It is an overpowered physical server. Other services run smoothly beside Cassandra. – mailq Dec 2 '14 at 19:10
  • 1
    Have you checked whether the process is CPU bound, I/O bound, or blocked waiting for something else? – kasperd Dec 3 '14 at 23:35
  • 1
    what are the settings of the launched java virtual machine? could be a powerfull server, if java instance is limited to 128M, it won't get more. The sound of nothing in logs makes me think memory limits... – Antony Gibbs Dec 6 '14 at 3:11
  • 1
    output of ulimit -a from a Cassandra-user console may be handy. – Antony Gibbs Dec 6 '14 at 3:19
1

I'am not a cassandra expert, but did you tried nodetool stop compaction and to set a threshold for compaction (setcompactionthreshold to 0), so the system will not retry it

  • nodetool stop -- COMPACTION It didnt help. Jobs are still going on. – Jigar Shah Jul 31 '17 at 14:07
  • @JigarShah Same question as the OP? you should maybe ask a new question to attract new answer. As maybe my solution dont help you, but helped the OP – yagmoth555 - GoFundMe Monica Jul 31 '17 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.