This is a follow-up question from: How do I mount an rbd device from fstab Which I asked some time back.

How do I mount rbd devices with the fuse driver? (I can't use the kernel driver from docker without opening up lots of privileges)

As far as I can tell I create an entry into the fstab file with something like:

id=client.admin /mnt fuse.ceph 0 0

However I can't see how to specify the rbd block device. If I run this mount, I see a file in /mnt/mariadb1 (mariadb1 is the name of my rbd device created with rbd create --size 250000 mariadb1)

How do I actually mount it as a filesystem?

Note: Yes I'd really like to use the kernel rbd driver and mount it from the host and then expose it to my docker container, but.... I'm using CoreOS.

If it is possible to mount filesystem in the host mount namespace I'd settle for that, but don't see a way forward yet. The best I've managed is to mount it from a container that has lots of privileges. But I can't share that container with --volumes-from. I don't really fancy having this particular container with full access to the host.

  • Don't do this. Mount it in the host and then point your Docker data volumes at it. – Michael Hampton Jul 9 '15 at 3:42
  • 1
    Sure, if you could tell me how to do that on CoreOS I would. – Matt Jul 9 '15 at 3:45
  • Ok, this might be a possibility. I'll have to write my own script. cephnotes.ksperis.com/blog/2014/01/09/… – Matt Jul 9 '15 at 4:00
  • 1
    Hopefully, docker volume plugins (we have a prototype Ceph implementation) will do this in the future. – Alistair A. Israel Jul 10 '15 at 4:31
  • @aisrael - that sounds useful. Then I could bring the container up anywhere such as with fleet and not care about host mounts or require admin permissions to set it up. That's why I'm interested in testing the fuse version out. – Matt Jul 10 '15 at 6:56

I have to assume that you are referring to RBD-fuse when talking about this because ceph-fuse is a cephfs utility and nothing to do with RBD.

However the premise of your question is further broken because your understanding of what rbd-fuse actually does is incomplete.

librbd provides you with an image file that is stored in Ceph. Most people (through rbd map) use that image to provide a pseudo block device in the form of /dev/rbd*.

The only difference with rbd-fuse is that it exposes its own pseudo file-system and provides the images as plain POSIX compliant files as opposed to Block devices.

Hence rbd-fuse has no more knowledge about the actual contents of the image file than rbd map does.

When invoking rbd-fuse mountpoint any RBD images inside the pool will be visible within mountpoint. It takes command options, notably -p if you use a different pool in ceph other than rbd and -c if you use a different configuration file other than /etc/ceph/ceph. You can also expose individual images using -r.

What you do with those files is up to you. They can have a file-system written to them (beware Linux will grumble as they aren't specialist block devices). You can mount them as loop devices if a file-system is present.

$ rbd create test
# mkdir -p /rbd_images
# rbd-fuse /rbd_images
# mkfs /rbd_images/test
# mkdir -p /mnt/rbd_test
# mount /rbd_images/test /mnt/rbd_test

Should present something along the lines as follows:

$ mount
rbd-fuse on /rbd_images type fuse.rbd-fuse (rw,nosuid,nodev)
/rbd_images/rbd_test on /mnt/rbd_test type ext4 (rw)

Whether this is the correct way (for your usage) to do this though is a different matter. However it can allow you access to RBD images that would be unsupported otherwise by your systems krbd version.

  • 1
    indeed, but there is precious little documentation out there on this utility, so I thought I'd articulate it for posterity... – Stuart Harland Nov 2 '16 at 13:37

The ceph.fuse filesystem type (and its manual equivalent ceph-fuse) are used to mount cephfs volumes. Those are fundamentally different from RADOS block devices (RBD).

On the one hand an RBD represents a block device on which a host system can run a file system, much like it would with a physical block device. Ceph doesn't know anything about the structure of the file system inside the RBD. On the other hand, cephfs is a file system in its own right, i.e. ceph manages individual files and metadata. One of the consequences is that you can generally mount an RBD only once, because two hosts accessing the same block device don't have any provisions to synchronize accesses to it, whereas you can mount a cephfs many times much like a network file system.

  • 1
    Actually I think you mount it as a file and then mount the file using a loopback device. ceph.com/docs/master/man/8/rbd-fuse It's not quite what I was hoping for so I went a different route. – Matt Aug 9 '15 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.