4

We've set up a mirroring pair of GlusterFS servers. No special tuning, whatever came "out of the box" with GlusterFS-3.5.1 in the official RHEL6 RPM, that's what we have.

The cluster works, but the performance is pretty awful. For example, extracting a large tarball (firefox-31.0.source.tar.bz2) via GlusterFS on localhost takes a whopping 44 minutes here. Extracting the same file directly -- on the same disk -- takes less than 2. There is a similar disparity in removing the created trees (takes 10 minutes via gluster)...

Of course, it is to be expected, that the mirroring needs to take place, etcaetera, a network-using filesystem will be slower -- but 30 times slower? Simply copying the large file over is fast -- so it is not the bandwidth we are lacking. While the untar-ing is running, I see both the glusterfs (client) and the glusterfsd (server) processes consuming a lot of CPU (about 10% each), but the system remains about 70% idle -- both gluster-processes are a lot busier than the extracting bzip2 and tar are... What are they doing?

Is there some tuning I can do to dramatically improve performance here? Or should I try ceph (or gfarm?) instead of gluster? Or are they all terrible with a large number of small files? Thank you!

3

Gluster is terrible at large numbers of small files. At least, as of version 3.6. They claim that the beta version of 3.7 that's already been released greatly improves this issue. I'd think that this version will be released as stable soon, since the Beta was released on April 28th, and they planned on the stable version being released earlier this month.

5
  • 1
    Note: Gluster v3.7 was released on May 18, 2015. No word on whether it fixes the small file issue yet. – Ernie May 19 '15 at 21:08
  • Any update on this one? – weynhamz Dec 25 '17 at 3:02
  • It seems to be a problem inherent to the architecture, and quite possibly, all distributed filesystems. – Ernie Jan 8 '18 at 22:33
  • We are seeing this behavior with small files in Gluster v3.11. – isalgueiro Feb 14 '18 at 12:55
  • 2
    @Ernie any news on this? 5 years have passed.. – FooBar Sep 24 '19 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.