1

I'm trying to use OpenShift with PersistentStorage on a GusterFS cluster.

I'm starting one of the default templates : mysql-persistent

I've installed a GlusterFS cluster and create a volume gv_mysql_01

I've added the glusterfs endpoints in openshift :

oc get endpoints
NAME                ENDPOINTS                                 AGE
glusterfs-cluster   10.100.134.26:24007,10.100.134.28:24007   1h

I've the create the PersistentVolume on openshift :

cat gluster-mysql_01-storage.yaml 
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "mysql" 
spec:
  capacity:
    storage: "512Mi" 
  accessModes:
    - "ReadWriteOnce"
  glusterfs: 
    endpoints: "glusterfs-cluster" 
    path: "gv_mysql_01" 
    readOnly: false
  persistentVolumeReclaimPolicy: "Recycle"

oc create -f gluster-mysql_01-storage.yaml

The PersistentStorage is bind to the container :

oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM        REASON    AGE
mysql     <none>    512Mi      RWO           Bound     test/mysql             53m

oc get pvc
NAME      LABELS                               STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
mysql     template=mysql-persistent-template   Bound     mysql     512Mi      RWO           1h

On on host of the cluster the volume is mount :

10.100.134.26:gv_mysql_01 on /var/lib/origin/openshift.local.volumes/pods/c111c480-8ec7-11e5-8405-0a57f8bdd6b3/volumes/kubernetes.io~glusterfs/mysql type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

but in the container logs :

docker logs b8cd5bb3b0be
Running mysql_install_db ...
mkdir: cannot create directory '/var/lib/mysql/data': Permission denied
chmod: cannot access '/var/lib/mysql/data/mysql': Permission denied
mkdir: cannot create directory '/var/lib/mysql/data': Permission denied
chmod: cannot access '/var/lib/mysql/data/test': Permission denied
151119 14:30:20 [Warning] Can't create test file /var/lib/mysql/data/mysql-1-q2yxh.lower-test
151119 14:30:20 [Warning] Can't create test file /var/lib/mysql/data/mysql-1-q2yxh.lower-test
/opt/rh/mysql55/root/usr/libexec/mysqld: Can't change dir to '/var/lib/mysql/data/' (Errcode: 13)
151119 14:30:20 [ERROR] Aborting

I've try to change owner,group,rights of /var/lib/origin/openshift.local.volumes/pods/c111c480-8ec7-11e5-8405-0a57f8bdd6b3/volumes/kubernetes.io~glusterfs/mysql with the uid:gid of the mysql user in the container, but it still doesn't work.

Am I missing something ?

1

Try these SELinux settings

setsebool -P virt_use_fusefs 1
setsebool -P virt_sandbox_use_fusefs 1
| improve this answer | |
1

I was having the same error and I was able to get it working by changing the default scc restricted policy strategy. Since you are running the openshift mysql image that runs as a specific user (27), you need to change the restricted scc to allow the id to run

[root@ose1 ceph]# oc edit scc restricted

then edit the file (with vi) and change the fsGroup value from RunAsAny to MustRunAs

    fsGroup:
        type: MustRunAs

You can see this:

[root@ose1 ceph]# oc get scc restricted
NAME         PRIV      CAPS      HOSTDIR   SELINUX     RUNASUSER        FSGROUP     SUPGROUP   PRIORITY
restricted   false     []        false     MustRunAs   MustRunAsRange   MustRunAs   RunAsAny   <none>

You also need to change the default UID range in the namespace you are using:

[root@ose1 ceph]# oc edit ns default  (sub your namespace for 'default')

openshift.io/sa.scc.uid-range: 25/10000

I made start at 25 as an example, but this will allow the openshift mysql image to run it's user of 27

| improve this answer | |
  • the fsGroup is already MustRunAs. I've change the uid-range but I still get the permission denied. For the moment the only solution I get is to disable selinux, and chown 26:26 the mysql glusterfs mountpoint, and chmod 777. Without all of that MySQL doesn't start. – mick Nov 23 '15 at 15:17
  • @mick - I'm trying to reproduce your problem, but am not able to, I did notice that you are using id 26. The mysql image is hard coded with id 27. See my pod definition below – screeley Nov 24 '15 at 19:07
  • ` apiVersion: v1 kind: Pod metadata: name: mysql-gluster labels: name: mysql-gluster spec: containers: - image: openshift3/mysql-55-rhel7 name: mysql-gluster volumeMounts: - name: glustervol mountPath: /var/lib/test/mysql env: - name: MYSQL_ROOT_PASSWORD value: foopass securityContext: runAsUser: 27 privileged: false volumes: - name: glustervol persistentVolumeClaim: claimName: glusterfs-claim` – screeley Nov 24 '15 at 19:13
  • @mick - formatting is terrible, sorry, but see answer below, if you have SELinux enforcing you may need to set those SELinux booleans, I have SELinux enforcing, so your pod does run but you are not able to access the gluster volume? – screeley Nov 24 '15 at 19:21
  • I've try what you say. In a first time it works, and then I try to understant why it is not working with the openshift default template mysql-persistent. Adding runAsUser: 27 in the template didn't change anything. I've then see that in your pod definition, you mount the persistentVolume in the bad directory, /var/lib/test/mysql instead of /var/lib/mysql (I'm using the openshift/mysql-55-centos7 image). The database is starting because not using the persistent volume. In /var/lib/test/mysql I still get permission denied. – mick Nov 25 '15 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.