2

I cannot generate a cluster report in OpsCenter. The server and clients are running on CentOS 5.9, 64bit. Software versions:

  • opscenter-3.2.2-1
  • opscenter-agent-3.2.2-1

Here is the error reported in the web browser:

There was an error generating the cluster report: u'/dev/mapper/Staging3SSTables_vol1'

That is the file system on the third cassandra node in this cluster. That underscore should be a hyphen:

$ ls -la /dev/mapper/Staging3SSTables_vol1
ls: /dev/mapper/Staging3SSTables_vol1: No such file or directory

$ ls -la /dev/mapper/Staging3SSTables-vol1
brw-rw---- 1 root disk 253, 0 Oct  7 13:56 /dev/mapper/Staging3SSTables-vol1

$ df -h /dev/mapper/Staging3SSTables-vol1
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/Staging3SSTables-vol1
                      551G  199M  522G   1% /var/lib/cassandra/data

I don't get this error for the other nodes in the cluster.

What could be causing this?

Here is the full output of the error from opscenterd.log:

2013-10-10 08:48:37-0700 [] ERROR: There was an error while generating a cluster report. Traceback (most recent call last):
          File "/usr/lib/python2.6/site-packages/opscenterd/StatsReporter.py", line 132, in generateReport
          File "/usr/share/opscenter/lib/py-redhat/2.6/shared/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/share/opscenter/lib/py-redhat/2.6/shared/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python2.6/site-packages/opscenterd/StatsReporter.py", line 148, in createClusterReport
          File "/usr/share/opscenter/lib/py-redhat/2.6/shared/amd64/twisted/internet/defer.py", line 1018, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/share/opscenter/lib/py-redhat/2.6/shared/amd64/twisted/python/failure.py", line 349, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python2.6/site-packages/opscenterd/StatsReporter.py", line 109, in format_node
          File "/usr/share/opscenter/lib/py-redhat/2.6/shared/amd64/twisted/internet/defer.py", line 542, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python2.6/site-packages/opscenterd/Cluster.py", line 484, in sumDiskSpace
        KeyError: u'/dev/mapper/Staging3SSTables_vol1'

2013-10-10 08:48:37-0700 [] ERROR: Error generating the cluster report: Traceback (most recent call last):
          File "/usr/lib/python2.6/site-packages/opscenterd/WebServer.py", line 2243, in ClusterReportController
        KeyError: u'/dev/mapper/Staging3SSTables_vol1'
2

What's happening here is opscenterd is looping over partitions containing any configured data directories from cassandra.yaml, and comparing those to the output of df --print-type --no-sync --block-size=1G --local. OpsCenter seems to think that /dev/mapper/Staging3SSTables_vol1 is one of those partitions, which is obviously wrong.

We do some escaping of special characters in partition names, but - is in the list of allowed, non-escaped characters. So unless that dash isn't really a dash, or there is simply a bug in OpsCenter code, that shouldn't be causing any issues.

I don't get this error for the other nodes in the cluster.

Once this error occurs, the cluster report will stop processing. So if the offending node is the first to be processed, the other nodes wouldn't have a chance to succeed or fail.

| improve this answer | |
  • Thank you very much for your response, although I'm just now seeing it a year later. This problem disappeared after upgrading OpsCenter sometime after noticing the problem. I don't know what version brought the fix, but things are working fine now. – Gene Oct 4 '14 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.