3

I have built an experimental ceph cluster - 12 nodes, 50 osds, 3 mons, 3 mds, for which I'm trying to run a samba gateway. It seems that when writing lots of small files, the fsync() system calls from samba will routinely block, presumably at the frequency of the journal flush interval. I'm a developer, and not really a sysadmin, and would appreciate some background on how to minimise the impact of fsyncs with ceph. I have removed the fsync calls from samba for now, which helps massively, but i still think performance with lots of small files should be much better. Power loss integrity isn't a worry. Also, with large files, the cluster will saturate then 10G link. My journal disks are certainly not optimal - they are mechanical disks, each shared between a few osds. Is there a way to prevent journal writes(?) from blocking for so long on fsync? Is ceph waiting the until the next journal commit when it hits an fsync call? I don't really have budget for ssd journals, so impact minimisation would be the only option. Also, with the ceph kernel client, the performance is much better than going via the samba gateway - so this apparently isn't being bounded by network bandwidth.

Servers used are old compute nodes that have been repurposed: 4x Xeon 5160 with 16Gb RAM in each node, with 1G bonded network interfaces, and 10G Infiniband for the cluster network.

Each OSD node has a single local 10K SAS disk for the journals, and multiple OSDs using a large Dell PERC RAID enclosure utilised in single disk per OSD mode.
Pausing can vary between nothing, and about 5 secs, which is the journal flush interval, so I guess it depends where the fsync() occurs relative to the time pending to journal commit.

I haven't tried Bluestore yet, but it would be the default choice in future if/when this goes into production.

4
  • For how much time does fsync block? What are hardware specifications?
    – shodanshok
    Jan 26, 2017 at 21:23
  • 1
    Do you use the "classic" on-disk format? If yes, have you tried bluestore?
    – maxf
    Jan 26, 2017 at 22:30
  • 1
    Further progress: performance certainly appears to be bound by synchronous flush of journals. Mounting with nobarrier improves significantly, but on 10 year old hardware, who knows what the condition of disk controller batteries will be like. I would like to know how I can optimise journal performance properly, without resorting to nobarrier on the xfs journal partitions.
    – kdm
    Jan 27, 2017 at 17:49
  • 1
    Spinning disks can only do ~50 fsyncs per second. Put the journal (xfs's journal, or bluestore's journal if you are using it by now) onto an SSD, and you'll get > 5000 fsyncs per second easily. (Only use enterprise SSDs with capacitors, otherwise you'll get only ~250 fsyncs/s, see here). There is no better safe solution; the only other alternative is to change the programs themselves to not do fsync in case where it isn't necessary; for example, in many cases you can batch fsync after writing a bunch of files.
    – nh2
    Jun 24, 2018 at 0:37

1 Answer 1

0

Is ceph waiting the until the next journal commit when it hits an fsync call?

Yes, mostly. But it acts a little differently depending on the backend.

Under FileStore, there's a small journal buffer that can act as a small write-burst cache, but it's tiny. And yes, once it fills, it blocks to flush - across the entire cluster or PG.

Under BlueStore, there's no such buffer. And yes, bluestore blocks on every write to fsync to the Journal - all Journals in the PG. This is how BlueStore can remain very consistent and predictable in IOPS and writes. Under Bluestore, you want to move at least the write-ahead-log (WAL) off to an Enterprise SSD - because BlueStore will move the Journal and DB off to the same WAL partition - if there's enough room (you don't even have to specify them, just the WAL).

Enterprise SSDs as WAL/DB/Journals because they ignore fsync

But the real issue in this cluster is that you are using sub-optimum HDDs as Journals that are blocking on very slow fsyncs when they get flushed.

Even Consumer-grade SSDs have serious issues with Ceph's fsync frequency as journals/WAL, as consumer SSDs only have transactional logs and no real power-backup.

It's Enterprise SSDs that have large capacitors that allow the drive to continue operating after a power-loss. Thus, they can guarantee a successful write under a power-loss event.

The added benefit is that Enterprise SSDs typically ignore fsync commands from the OS! Because they can guarantee the success of the write, they immediately return the fsync request from the OS.

Thus, you get major performance gains when using an Enterprise-grade SSD as your WAL/DB/Journal.

Under FileStore, you'll see those delays go away, but you'll see inconsistent bursts of cache and then back down.

This is where BlueStore comes in, as BlueStore will guarantee a consistent IOPS and write across the board. But, you need WAL/DB/Journal on an Enterprise SSD to ignore those fsyncs.

At this time, Intel S3700s can be had in the used market for about $40/ea. Tiny investment for massive performance gains of unblocking fsyncs.

Some quotes (https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Bluestore_vs_Filestore):

Filestore writes everything to the journal and only starts to flush it to the data device when the journal fills up to the configured percent. This is very convenient because it makes journal act as a «temporary buffer» that absorbs random write bursts.

Bluestore can’t do the same even when you put its WAL+DB on SSD. It also has sort of a «journal» which is called «deferred write queue», but it’s very small (only 64 requests) and it lacks any kind of background flush threads. So you actually can increase the maximum number of deferred requests, but after the queue fills up the performance will drop until OSD restarts.

And: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/

The BlueStore journal will always be placed on the fastest device available, so using a DB device will provide the same benefit that the WAL device would while also allowing additional metadata to be stored there (if it will fit). This means that if a DB device is specified but an explicit WAL device is not, the WAL will be implicitly colocated with the DB on the faster device.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .