2

I had 3 nodes, all with ~550GB, and added 4 more. I hoped that data would migrate on its own from old servers to new ones. Unfortunately after one day, data seems to stay in old places.

In parallel, there is a constant stream of incoming data, and about 150GB more per node was added. So old nodes have currently 550+150 = ~700GB, new ones ~150GB.

Should I just wait? Is there a command or option that I should run to make them rebalance? Or it is a problem with replication 1 or inserting data while rebalancing?

--  Address   Load       Tokens  Owns   Host ID                               Rack
UN  x.x.x.x   702.48 GB  256     13.1%  3e9e7e79-c727-4a3f-86ad-0dbfb77a4540  rack1
UN  x.x.x.x   692.47 GB  256     15.5%  ac41b3df-301e-470a-b531-12f2aeba7328  rack1
UN  x.x.x.x   660.65 GB  256     14.2%  8b1bb70d-57fe-460a-b756-4ac82b24acb6  rack1
UN  x.x.x.x   156.7 GB   256     15.1%  d499acd3-8eae-4a4c-bba3-cd8e45bd1ce3  rack1
UN  x.x.x.x   140.99 GB  256     13.6%  f266de96-8547-4e68-8d31-50a13e6c6628  rack1
UN  x.x.x.x   138.44 GB  256     13.5%  cbf53e0d-398d-4fc7-befd-b83d173a4e67  rack1
UN  x.x.x.x   155.17 GB  256     15.0%  dddbcb68-9730-456a-8429-de06bcf2f2bb  rack1

I'm using Cassandra 2.09. Murmur 3 partitioner, 256 vnodes per node, replication 1 (no copy).

3
2

You need to run a cleanup on the old nodes, and perhaps a repair -- it won't move the data around of its own accord. See the documentation here for adding new nodes to a cluster.

8
  • Thanks. By default, the repair command takes a snapshot of each replica - does that mean that I need double the space for repair? Oct 3 '14 at 11:19
  • 1
    Repair will use more space, but not double the space, as it works on one replica at a time. So it will use additional space equal to the size of your largest replica.
    – Mike Scott
    Oct 3 '14 at 11:20
  • What about -par options for parallel repair? Where are intermediate data stored in such scenario? Documentation is very vague. Oct 3 '14 at 11:22
  • Sorry, I've not used the -par option, so I can't comment on that. But having looked at the documentation, I wouldn't recommend using that option on a live database.
    – Mike Scott
    Oct 3 '14 at 11:39
  • I've stopped inserting and tried to run repair. But nodetool repair <keyspace> returns almost immediately on each node and do not change anything: Starting repair command #6, repairing 256 ranges for keyspace xxx Oct 3 '14 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.