1

We are facing one question about monitoring volume storage.

We use metrics "kubelet_volume_stats_used_bytes" and "kubelet_volume_stats_capacity_bytes". On our lab we have > 100 volumes mounted to pods, and among these volumes there are 14 volumes created by Glusterfs storage, the rest volumes are provided by Ceph.

$ kubectl get sc
NAME                            PROVISIONER                    AGE
default                         ceph.rook.io/block             166d
glusterfs-storageclass          kubernetes.io/glusterfs        166d
local-storage                   kubernetes.io/no-provisioner   166d
non-repl                        ceph.rook.io/block             166d
rook-ceph-block-rep (default)   ceph.rook.io/block             166d

We try to curl metrics from Kubernetes API

# curl -k -s -H "Authorization: Bearer <token>" https://kubernetes.default.svc:443/api/v1/nodes/<hostname>/proxy/metrics 2>&1 |grep kubelet_volume_stats_used_bytes

And unfortunately among all nodes we only get info from those 14 Glusterfs volumes - no metrics from the rest (created by Ceph)

So, is it possible to get metrics from those Ceph created volumes? If possible, how can we do to make Kubernetes provide them?

Thanks in advance.

BTW - we are not sure about Ceph because we also noticed on Kubernetes doc there is a chapter storage provisioner and this is what it says,

Volume Plugin   Internal Provisioner    Config Example
CephFS          -                       -
GlusterFS       ✓                       Glusterfs

Don't know if this is the reason. Kubernetes version 1.12.3

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", 
GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", 
BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", 
Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", 
GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", 
BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", 
Platform:"linux/amd64"}
0

This seems to work on Kubernetes v 1.13+.

Here is a GitHub issue for it ceph: add metrics for flexvolume driver #3128.

And here is a comment describing how to set it up.

Also you can use Prometheus for it. This is well documented and described in rook docs

  • Thanks for your reply:) – Lijing Zhang Sep 25 '19 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.