2

I have built an experimental ceph cluster - 12 nodes, 50 osds, 3 mons, 3 mds, for which I'm trying to run a samba gateway. It seems that when writing lots of small files, the fsync() system calls from samba will routinely block, presumably at the frequency of the journal flush interval. I'm a developer, and not really a sysadmin, and would appreciate some background on how to minimise the impact of fsyncs with ceph. I have removed the fsync calls from samba for now, which helps massively, but i still think performance with lots of small files should be much better. Power loss integrity isn't a worry. Also, with large files, the cluster will saturate then 10G link. My journal disks are certainly not optimal - they are mechanical disks, each shared between a few osds. Is there a way to prevent journal writes(?) from blocking for so long on fsync? Is ceph waiting the until the next journal commit when it hits an fsync call? I don't really have budget for ssd journals, so impact minimisation would be the only option. Also, with the ceph kernel client, the performance is much better than going via the samba gateway - so this apparently isn't being bounded by network bandwidth.

Servers used are old compute nodes that have been repurposed: 4x Xeon 5160 with 16Gb RAM in each node, with 1G bonded network interfaces, and 10G Infiniband for the cluster network.

Each OSD node has a single local 10K SAS disk for the journals, and multiple OSDs using a large Dell PERC RAID enclosure utilised in single disk per OSD mode.
Pausing can vary between nothing, and about 5 secs, which is the journal flush interval, so I guess it depends where the fsync() occurs relative to the time pending to journal commit.

I haven't tried Bluestore yet, but it would be the default choice in future if/when this goes into production.

  • For how much time does fsync block? What are hardware specifications? – shodanshok Jan 26 '17 at 21:23
  • 1
    Do you use the "classic" on-disk format? If yes, have you tried bluestore? – maxf Jan 26 '17 at 22:30
  • 1
    Further progress: performance certainly appears to be bound by synchronous flush of journals. Mounting with nobarrier improves significantly, but on 10 year old hardware, who knows what the condition of disk controller batteries will be like. I would like to know how I can optimise journal performance properly, without resorting to nobarrier on the xfs journal partitions. – kdm Jan 27 '17 at 17:49
  • Spinning disks can only do ~50 fsyncs per second. Put the journal (xfs's journal, or bluestore's journal if you are using it by now) onto an SSD, and you'll get > 5000 fsyncs per second easily. (Only use enterprise SSDs with capacitors, otherwise you'll get only ~250 fsyncs/s, see here). There is no better safe solution; the only other alternative is to change the programs themselves to not do fsync in case where it isn't necessary; for example, in many cases you can batch fsync after writing a bunch of files. – nh2 Jun 24 '18 at 0:37

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.