0

first of all my highlevel Setup:

  • Proxmox 3 Node Cluster Virtual Environment 5.2-6
    • on some VMs I use write back caching.
  • ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) running on 3 Nodes:
    • Rook rook/ceph:v0.9.3
    • Kubernetes Major:"1", Minor:"10", GitVersion:"v1.10.1"
    • 24 Disks/Osds (8 on each host)
    • Each host has:
    allocatable:
     cpu: "8"
     ephemeral-storage: "52931845238"
     hugepages-2Mi: "0"
     memory: 32649880Ki
     pods: "110"
    capacity:
     cpu: "8"
     ephemeral-storage: 57434728Ki
     hugepages-2Mi: "0"
     memory: 32752280Ki
     pods: "110"
    

The Problem

So my problem is, that I have huge Problems with the Rook/Proxmox combination. Bot systems Rook/Ceph and Proxmox are on different Systems in the same network in the same Datacenter.

Everytime I want to schedule a Host down (which I tried with setting all OSDs to down and add-noup) or as happened right now, that one host staled and I had to restart the host as it was not responsive, all or at least one VM breaks its filesystem completely.

I set the failure domain to Host so as far as I understand, Ceph should be able to handle this failover, but it does not.

What am I missing? Thought already that the add-noup option may break something but it also happend with a "natural" failure of one host.

Rook Setup

Rook Operator Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: 2019-03-18T18:15:47Z
  generation: 2
  labels:
    chart: rook-ceph-v0.9.3
    operator: rook
    storage-backend: ceph
  name: rook-ceph-operator
  namespace: rook-ceph-system
  resourceVersion: "37343691"
  selfLink: /apis/extensions/v1beta1/namespaces/rook-ceph-system/deployments/rook-ceph-operator
  uid: da6c994a-49a9-11e9-a470-001b21bcaf58
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rook-ceph-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rook-ceph-operator
        chart: rook-ceph-v0.9.3
    spec:
      containers:
      - args:
        - ceph
        - operator
        env:
        - name: ROOK_LOG_LEVEL
          value: INFO
        - name: ROOK_ENABLE_SELINUX_RELABELING
          value: "true"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: ROOK_MON_HEALTHCHECK_INTERVAL
          value: 45s
        - name: ROOK_MON_OUT_TIMEOUT
          value: 9999s
        image: rook/ceph:v0.9.3
        imagePullPolicy: IfNotPresent
        name: rook-ceph-operator
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: rook-ceph-system
      serviceAccountName: rook-ceph-system
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-03-18T18:15:47Z
    lastUpdateTime: 2019-03-20T08:59:52Z
    message: ReplicaSet "rook-ceph-operator-555d586566" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: 2019-03-21T13:40:36Z
    lastUpdateTime: 2019-03-21T13:40:36Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Ceph Cluster CRD

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"ceph.rook.io/v1","kind":"CephCluster","metadata":{"annotations":{},"name":"rook","namespace":"rook-ceph"},"spec":{"backend":"ceph","cephVersion":{"allowUnsupported":false,"image":"ceph/ceph:v13.2.4-20190109"},"dataDirHostPath":"/var/lib/rook","mon":{"allowMultiplePerNode":false,"count":3},"network":{"hostNetwork":true},"storage":{"config":{"osdsPerDevice":"1","storeType":"bluestore"},"deviceFilter":null,"location":null,"metadataDevice":null,"useAllDevices":true,"useAllNodes":true}}}
  clusterName: ""
  creationTimestamp: 2019-03-18T18:16:26Z
  finalizers:
  - cephcluster.ceph.rook.io
  generation: 1
  name: rook
  namespace: rook-ceph
  resourceVersion: "37344201"
  selfLink: /apis/ceph.rook.io/v1/namespaces/rook-ceph/cephclusters/rook
  uid: f17441c3-49a9-11e9-b135-001b21bcaf4e
spec:
  cephVersion:
    image: ceph/ceph:v13.2.4-20190109
  dashboard: {}
  dataDirHostPath: /var/lib/rook
  mon:
    allowMultiplePerNode: false
    count: 3
  network:
    hostNetwork: true
  rbdMirroring:
    workers: 0
  storage:
    config:
      osdsPerDevice: "1"
      storeType: bluestore
    useAllDevices: true
    useAllNodes: true
status:
  state: Created

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"ceph.rook.io/v1","kind":"CephBlockPool","metadata":{"annotations":{},"name":"proxmox","namespace":"rook-ceph"},"spec":{"crushRoot":"default","failureDomain":"host","replicated":{"size":3}}}
  clusterName: ""
  creationTimestamp: 2019-03-18T18:59:15Z
  generation: 1
  name: proxmox
  namespace: rook-ceph
  resourceVersion: "36934084"
  selfLink: /apis/ceph.rook.io/v1/namespaces/rook-ceph/cephblockpools/proxmox
  uid: ed1ef78d-49af-11e9-b135-001b21bcaf4e
spec:
  crushRoot: default
  failureDomain: host
  replicated:
    size: 3

Reproduce

In order to reproduce my scenario: * Installing 3 Node Proxmox Cluster * Installing Kubernetes * Install Rook on Kubernetes * Creating Ceph cluster and pool * create user for proxmox with access to that pool * Connect Proxmox to Ceph * Create VM with some synthetical write load and xfs * Shut down one ceph node * Bring the ceph node up again (* sometimes it seems that this just happens when I create a snapshot)

Then the current result is, that the filesystem is broken. On the KVM screen there are then messages like: enter image description here

If thats not enough to reproduce retry. From 6 VMs 4 are most likely crashing for me, while Gitlab seems to crash every time.

Expected result would be, that the vm is still running.

Update 1: Added Picture of Proxmox Console

Update 2: Maybe taking Snapshots is evil

I did now further testing. Maybe it is because we were running snapshots via the admin user at the rook-tools pod before maintanance or after an outage (as I paniced). At least I could reproduce the behaviour without bringing nodes or OSDs down. I will do further testing.

Taking snapshots via the Proxmox GUI seems to work (as it may try to get a consistent IO state)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.