Ceph now includes (experimental) support for multiple filesystems within a single storage cluster; but the mount options don't seem to allow specifying which filesystem to mount.

I have configured two testing filesystems, each with their own mds and backing pools; but can't get a client node to mount both of them; as the mount options only specify a monitor, which is the same for both.

root@ws:~# ceph status
cluster 92ed87a5-559c-4b79-90c7-6b381374d431
 health HEALTH_OK
 monmap e7: 1 mons at {mon-jacopo=}
        election epoch 17, quorum 0 mon-jacopo
  fsmap e59: burninatorfs-1/1/1 up mediafs-1/1/1 up {[burninatorfs:0]=mds-jacopo=up:active,[mediafs:0]=mds-media=up:active}
 osdmap e404: 2 osds: 2 up, 2 in
        flags sortbitwise
  pgmap v36088: 288 pgs, 5 pools, 48411 bytes data, 40 objects
        10315 MB used, 5540 GB / 5554 GB avail
             288 active+clean

I can't find any options for fstab that allow me to mount one ceph fs or the other. The result of any combination of mount paths and options I've tried has been an unknown selection of the two filesystems being mounted at both points.

root@jacopo:/media# cat /etc/fstab | grep ceph
mon-jacopo.burninator.net:/ /media/burninator   ceph        noshare,noatime,_netdev,name=burninator,secret=XXXX==   0 0
mon-jacopo.burninator.net:/ /media/media    ceph        noshare,noatime,_netdev,name=media,secret=YYYY==    0 0

Strangely, even when I isolate the permissions for both filesystems by different identities and secrets, they mount as though they are the same.

    key: XXXX==
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw pool=burninatorfs-data,allow rw pool=burninatorfs-metadata
    key: YYYY==
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw pool=mediafs-data,allow rw pool=mediafs-metadata

Changes are allowed to whichever filesystem is actually mounted (haven't found a way to determine this) - and both mount points reflect the result as though it were a bind mount. I'm pretty sure last part is a bug, as the client connection shouldn't be shared (noshare mount option) and the mount credentials should only be allowed to modify at most the mount that is pointing to the 'correct' filesystem.

root@jacopo:~# mount | grep ceph
mon-jacopo.burninator.net:/ on /media/burninator type ceph (rw,noatime,name=burninator,secret=<hidden>,noshare,acl,_netdev)
mon-jacopo.burninator.net:/ on /media/media type ceph (rw,noatime,name=media,secret=<hidden>,noshare,acl,_netdev)
root@jacopo:~# cd /media
root@jacopo:/media# find burninator/
root@jacopo:/media# find media/
root@jacopo:/media# mkdir media/newdir
root@jacopo:/media# touch media/newdir/1
root@jacopo:/media# touch media/newdir/2
root@jacopo:/media# rm burninator/newdir/1
root@jacopo:/media# find burninator/
root@jacopo:/media# find media/

All nodes are running Ubuntu 16.04 LTS, at or near kernel 4.4.0-45-generic, ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) installed by ceph-deploy 1.5.32.


Reading this commit, you can use the mount option mds_namespace.

# mount -t ceph -o mds_namespace=burninatorfs /mnt/burninatorfs
# mount -t ceph -o mds_namespace=mediafs /mnt/mediafs
  • Thanks very much for the info - unfortunately as I was unable to find this when I was evaluating cephfs I am now not in a position to test it. – rvalue Mar 27 '17 at 4:20

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.