-1

My production cluster had the repair service enabled since april 16th with the default 9 days time to completion and repairs would complete properly. However, since may 22nd, it is being disabled automatically by Opscenter:

From /var/log/opscenter/opscenterd.log:

[...]
2014-06-03 21:13:47-0400 [zs_prod] ERROR: Repair task (<Node 10.1.0.22='6417880425364517165'>, (-4019838962446882275L, -4006140687792135587L), set(['zs_logging', 'OpsCenter'])) timed out after 3600 seconds.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: Repair task (<Node 10.1.0.22='6417880425364517165'>, (-4006140687792135587L, -4006140687792135586L), set(['zs_logging', 'OpsCenter'])) timed out after 3600 seconds.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: More than 100 errors during repair service, shutting down repair service
2014-06-03 22:16:44-0400 [zs_prod]  INFO: Stopping repair service
[...]

From /var/log/opscenter/repair_service/zs_prod.log:

[...]
2014-06-03 22:16:44-0400 [zs_prod] ERROR: Repair task (<Node 10.1.0.22='6417880425364517165'>, (-4006140687792135587L, -4006140687792135586L), set(['zs_logging', 'OpsCenter'])) timed out after 3600 seconds.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: Task (<Node 10.1.0.22='6417880425364517165'>, (-4006140687792135587L, -4006140687792135586L), set(['zs_logging', 'OpsCenter'])) has failed 1 times.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: 101 errors have ocurred out of 100 allowed.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: More than 100 errors during repair service, shutting down repair service
2014-06-03 22:16:44-0400 [zs_prod]  INFO: Stopping repair service

On the nodes on which the repair fails, from /var/log/cassandra/system.log:

ERROR [RMI TCP Connection(93502)-10.1.0.22] 2014-06-03 20:12:28,858 StorageService.java (line 2560) Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range but is not fully contained in one; this would lead to i
mprecise repair
        at org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:164)
        at org.apache.cassandra.repair.RepairSession.<init>(RepairSession.java:128)
        at org.apache.cassandra.repair.RepairSession.<init>(RepairSession.java:117)
        at org.apache.cassandra.service.ActiveRepairService.submitRepairSession(ActiveRepairService.java:97)
        at org.apache.cassandra.service.StorageService.forceKeyspaceRepair(StorageService.java:2620)
        at org.apache.cassandra.service.StorageService$5.runMayThrow(StorageService.java:2556)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

These errors, which only occurs if the repair service is running, are the only errors these nodes experience. Outside of the repair task, the Cassandra cluster works perfectly.

I am running Opscenter 4.1.2 with a 6 nodes DSE 4.0.2 cluster installed on linux virtual machines. The nodes run a vanilla installation of Ubuntu Server 12.04 64-bit and DSE was installed and secured according to the provided installation documentation.

I have been experiencing that problem on my development cluster for a while too (with DSE 4.0.0, 4.0.1 and 4.0.2), but I thought this was because of some configuration error on my part. The problem has appeared spontaneously at some point too.

The Cassandra cluster has been working very smoothly with a good write throughput. It is very stable and has enough resources to work with. We did not notice any problems with the applications that depend on it.

1 Answer 1

2

It’s a known bug in OpsCenter that was fixed in 4.1.3 version (see http://www.datastax.com/documentation/opscenter/4.1/opsc/release_notes/opscReleaseNotes413.html, the last issue)

I don’t think there is a workaround other than upgrading OpsCenter (which should be pretty easy to do)

2
  • Thank you. I have upgraded to 4.1.3 on Monday because I thought OPSC-2806 could have to do with it. I did not see the error yet. However, it now warns me that it "cannot complete without adversely affecting the cluster" if I set it under my gc_grace_seconds value of 9 days, which is very strange, because a full repair takes about 1.5 days.
    – jlemire-zs
    Jun 25, 2014 at 16:16
  • I am getting the "Requested range intersects a local range but is not fully contained in one; this would lead to imprecise repair" error in OpsCenter 6.1.0, so it seems like it might still be an issue.
    – Shannon
    Jun 7, 2017 at 16:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .