Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've looked through multiple forum posts, I've turned on debug et al, and I still cannot for the life of me figure out why the disk space isn't working. ALL other information is filled--except for the disk space.

On the Agent logs, I see requests for /os-metric/disk-space, but it always returns an empty value. There are NO errors on start-up, and DEBUG logging doesn't reveal anything. The output in the log is:

INFO [qtp728395819-23] 2014-03-20 16:49:19,636 HTTP: :get /os-metric/disk-space {} - 200

There are, also, no errors on the Opscenter server.

I initially had errors where it could not find the right cassandra.yaml file, but I have corrected these, and there are now zero errors in the startup-logs. This occurs with the latest version of Opscenter 4.1.1, and Cassandra 1.2.15. Are there any more suggestions on what to look for? For example, does it use a non-standard utility (i.e., not df)?

EDIT: Adding output from //nodes; this is common for each node.

{
    "data_held": 18001238572.0,
    "dc": "DC1",
    "devices": {
        "commitlog": "sdb",
        "data": [
            "sdc"
        ],
        "other": [
            "sda1"
        ],
        "saved_caches": "sdb"
    },
    "has_jna": true,
    "last_seen": 0,
    "load": 0.13,
    "mode": "normal",
    "network_interfaces": [
        "lo",
        "eth0",
        "sit0"
    ],
    "node_ip": "{snip}",
    "node_name": "{snip}",
    "node_version": {
        "cassandra": "1.2.15",
        "dse": null,
        "jobtracker": null,
        "search": null,
        "tasktracker": null
    },
    "num_procs": 8,
    "os": "linux",
    "partitions": {
        "commitlog": "/dev/sdc",
        "data": [
            "_"
        ],
        "other": [
            "/dev/sdb",
            "/dev/sda1"
        ],
        "saved_caches": "/dev/sdc"
    },
    "rack": "DEFAULT-RACK",
    "rpc_ip": "{snip}",
    "streaming": {},
    "task_progress": {},
    "token": "-5829041219150380184",
    "vnodes": true
}

cassandra.yaml data directories:

# Directories where Cassandra should store data on disk.  Cassandra
# will spread data evenly across them, subject to the granularity of
# the configured compaction strategy.
data_file_directories:
    - /var/lib/cassandra/data

$ cat /proc/mounts
[snip]
/dev/sdc /var/lib/cassandra/data ext3 rw,data=ordered 0 0
share|improve this question
    
Can you try restarting opscenterd, and then restarting each of your agents to see if that resolves the issue? There is a known race condition we're working to fix around restarting opscenterd without restarting the agents as well. –  mbulman Mar 24 at 16:43
    
I have done this multiple times. In fact, I have taken down the entire Cassandra cluster and started from scratch 2-3 times already. The agents are communicating all the other information just fine--just not the disk-usage metric. –  Andrew M. Mar 24 at 16:55
    
Next step is to see if opscenterd is aware of the Cassandra data partitions, but looking at the node information from the API: datastax.com/documentation/opscenter/4.1/api/docs/… , as well as hitting the API route you pasted in the original message on the agent to see if you're getting accurate information back. –  mbulman Mar 24 at 23:35
    
Thanks, @mbulman, I went ahead and added the call to /nodes to the API, since this seemed to show the relevant information (i.e., the devices and partitions). –  Andrew M. Mar 25 at 2:47
    
We ran into the same issue. If your backend storage is not a block device then the storage will not show up in OpsCenter. I have been told by Datastax that there is a open issue for this. –  Dan Dentinger Mar 25 at 14:10
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.