2

I had 3 nodes, all with ~550GB, and added 4 more. I hoped that data would migrate on its own from old servers to new ones. Unfortunately after one day, data seems to stay in old places.

In parallel, there is a constant stream of incoming data, and about 150GB more per node was added. So old nodes have currently 550+150 = ~700GB, new ones ~150GB.

Should I just wait? Is there a command or option that I should run to make them rebalance? Or it is a problem with replication 1 or inserting data while rebalancing?

--  Address   Load       Tokens  Owns   Host ID                               Rack
UN  x.x.x.x   702.48 GB  256     13.1%  3e9e7e79-c727-4a3f-86ad-0dbfb77a4540  rack1
UN  x.x.x.x   692.47 GB  256     15.5%  ac41b3df-301e-470a-b531-12f2aeba7328  rack1
UN  x.x.x.x   660.65 GB  256     14.2%  8b1bb70d-57fe-460a-b756-4ac82b24acb6  rack1
UN  x.x.x.x   156.7 GB   256     15.1%  d499acd3-8eae-4a4c-bba3-cd8e45bd1ce3  rack1
UN  x.x.x.x   140.99 GB  256     13.6%  f266de96-8547-4e68-8d31-50a13e6c6628  rack1
UN  x.x.x.x   138.44 GB  256     13.5%  cbf53e0d-398d-4fc7-befd-b83d173a4e67  rack1
UN  x.x.x.x   155.17 GB  256     15.0%  dddbcb68-9730-456a-8429-de06bcf2f2bb  rack1

I'm using Cassandra 2.09. Murmur 3 partitioner, 256 vnodes per node, replication 1 (no copy).

Improve this question
3
  • Did you balance the tokens? See: datastax.com/dev/blog/balancing-your-cassandra-cluster
    – user299898
    Commented Jul 17, 2015 at 4:02
  • Please include the full content of your advice in the post.
    – Deer Hunter
    Commented Jul 17, 2015 at 5:15
  • @GeorgeWilliamHerbert That post is from 2012, for a much older version of Cassandra. Not sure it still applies.
    – internetdotcom
    Commented Aug 7, 2015 at 16:11

1 Answer 1

Reset to default
2

You need to run a cleanup on the old nodes, and perhaps a repair -- it won't move the data around of its own accord. See the documentation here for adding new nodes to a cluster.

Improve this answer
8
  • Thanks. By default, the repair command takes a snapshot of each replica - does that mean that I need double the space for repair?
    – Jacek Kaniuk
    Commented Oct 3, 2014 at 11:19
  • 1
    Repair will use more space, but not double the space, as it works on one replica at a time. So it will use additional space equal to the size of your largest replica.
    – Mike Scott
    Commented Oct 3, 2014 at 11:20
  • What about -par options for parallel repair? Where are intermediate data stored in such scenario? Documentation is very vague.
    – Jacek Kaniuk
    Commented Oct 3, 2014 at 11:22
  • Sorry, I've not used the -par option, so I can't comment on that. But having looked at the documentation, I wouldn't recommend using that option on a live database.
    – Mike Scott
    Commented Oct 3, 2014 at 11:39
  • I've stopped inserting and tried to run repair. But nodetool repair <keyspace> returns almost immediately on each node and do not change anything: Starting repair command #6, repairing 256 ranges for keyspace xxx
    – Jacek Kaniuk
    Commented Oct 3, 2014 at 14:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .