Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

My production cluster had the repair service enabled since april 16th with the default 9 days time to completion and repairs would complete properly. However, since may 22nd, it is being disabled automatically by Opscenter:

From /var/log/opscenter/opscenterd.log:

[...]
2014-06-03 21:13:47-0400 [zs_prod] ERROR: Repair task (<Node 10.1.0.22='6417880425364517165'>, (-4019838962446882275L, -4006140687792135587L), set(['zs_logging', 'OpsCenter'])) timed out after 3600 seconds.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: Repair task (<Node 10.1.0.22='6417880425364517165'>, (-4006140687792135587L, -4006140687792135586L), set(['zs_logging', 'OpsCenter'])) timed out after 3600 seconds.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: More than 100 errors during repair service, shutting down repair service
2014-06-03 22:16:44-0400 [zs_prod]  INFO: Stopping repair service
[...]

From /var/log/opscenter/repair_service/zs_prod.log:

[...]
2014-06-03 22:16:44-0400 [zs_prod] ERROR: Repair task (<Node 10.1.0.22='6417880425364517165'>, (-4006140687792135587L, -4006140687792135586L), set(['zs_logging', 'OpsCenter'])) timed out after 3600 seconds.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: Task (<Node 10.1.0.22='6417880425364517165'>, (-4006140687792135587L, -4006140687792135586L), set(['zs_logging', 'OpsCenter'])) has failed 1 times.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: 101 errors have ocurred out of 100 allowed.
2014-06-03 22:16:44-0400 [zs_prod] ERROR: More than 100 errors during repair service, shutting down repair service
2014-06-03 22:16:44-0400 [zs_prod]  INFO: Stopping repair service

On the nodes on which the repair fails, from /var/log/cassandra/system.log:

ERROR [RMI TCP Connection(93502)-10.1.0.22] 2014-06-03 20:12:28,858 StorageService.java (line 2560) Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range but is not fully contained in one; this would lead to i
mprecise repair
        at org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:164)
        at org.apache.cassandra.repair.RepairSession.<init>(RepairSession.java:128)
        at org.apache.cassandra.repair.RepairSession.<init>(RepairSession.java:117)
        at org.apache.cassandra.service.ActiveRepairService.submitRepairSession(ActiveRepairService.java:97)
        at org.apache.cassandra.service.StorageService.forceKeyspaceRepair(StorageService.java:2620)
        at org.apache.cassandra.service.StorageService$5.runMayThrow(StorageService.java:2556)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

These errors, which only occurs if the repair service is running, are the only errors these nodes experience. Outside of the repair task, the Cassandra cluster works perfectly.

I am running Opscenter 4.1.2 with a 6 nodes DSE 4.0.2 cluster installed on linux virtual machines. The nodes run a vanilla installation of Ubuntu Server 12.04 64-bit and DSE was installed and secured according to the provided installation documentation.

I have been experiencing that problem on my development cluster for a while too (with DSE 4.0.0, 4.0.1 and 4.0.2), but I thought this was because of some configuration error on my part. The problem has appeared spontaneously at some point too.

The Cassandra cluster has been working very smoothly with a good write throughput. It is very stable and has enough resources to work with. We did not notice any problems with the applications that depend on it.

share|improve this question

1 Answer 1

up vote 2 down vote accepted

It’s a known bug in OpsCenter that was fixed in 4.1.3 version (see http://www.datastax.com/documentation/opscenter/4.1/opsc/release_notes/opscReleaseNotes413.html, the last issue)

I don’t think there is a workaround other than upgrading OpsCenter (which should be pretty easy to do)

share|improve this answer
    
Thank you. I have upgraded to 4.1.3 on Monday because I thought OPSC-2806 could have to do with it. I did not see the error yet. However, it now warns me that it "cannot complete without adversely affecting the cluster" if I set it under my gc_grace_seconds value of 9 days, which is very strange, because a full repair takes about 1.5 days. –  jlemire-zs Jun 25 '14 at 16:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.