We've faced weird issue with one of volumeMounts of one of our deployments. Here are relevant manifests' snippets:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: xxx-media-pv-resource-claim spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: cephfs ... apiVersion: extensions/v1beta1 kind: Deployment ... volumeMounts: - mountPath: /opt/media/logs name: xxx-media-log-storage - mountPath: /shared-media-files name: xxx-media-resource-storage ... volumes: - name: xxx-media-log-storage persistentVolumeClaim: claimName: xxx-media-pv-log-claim - name: xxx-media-resource-storage persistentVolumeClaim: claimName: xxx-media-pv-resource-claim ...
We get read/write error: "Operation not permitted" when trying to, e.g. cat/echo from/to files in
/shared-media-files although: a) can successfully touch/mkdir/chmod/rm there; b) can do everything in
/opt/media/logs which is backed up by identical PVC (except name). Cephfs storageclass and corresponding Ceph cluster is a storage for whole k8s cluster and all k8s entities that use it except
xxx-media-resource-storage behave as expected.
k8s master: v1.14.1
k8s workers: v1.16.0
We checked everything obvious like Ceph cluster health, free storage space, file permissions and ACLs, mount mode, PV/PVC descriptions, logs and didn't find anything special. As
xxx-media-pv-resource-claim PVC accessMode is RWX, we tried to use it with another test pod and got same errors. We tried to google the issue as well but with no luck.
Please help to troubleshoot the issue.