We are using GlusterFS to provide a distributed file system on 2 web servers which use a shared media directory and cache. So i have created 2 gluster volumes (media and var) and have mounted these in /var/www/site/media and /var/www/site/var/.

Each server is running the gluster server and client so that we have some redundancy and get to keep the data replicated. The var volume is written and read very heavily.

The problem we are encountering is that randomly the mounts will break and when you perform a ls -lah on the directory it is showing as d???????. To resolve the issue all we have to do is umount the directory and remount it.

I have reviewed the glusterfs log files and can see when the mount disappeared

[2013-05-02 11:32:02.105021] I [client3_1-fops.c:502:client3_1_unlink_cbk] 0-site-media-client-1: remote operation failed: No such file or directory

[2013-05-02 11:32:02.105270] I [client3_1-fops.c:502:client3_1_unlink_cbk] 0-site-media-client-0: remote operation failed: No such file or directory

[2013-05-02 11:32:02.105299] W [fuse-bridge.c:911:fuse_unlink_cbk] 0-glusterfs-fuse: 11806336: UNLINK() /catalog/product/cache/1/image/1000x1000/9df78eab33525d08d6e5fb8d27136e95/v/e/some-stuff-0915740$

[2013-05-02 11:32:02.378497] I [client3_1-fops.c:502:client3_1_unlink_cbk] 0-site-media-client-0: remote operation failed: No such file or directory

[2013-05-02 11:32:02.378625] I [client3_1-fops.c:502:client3_1_unlink_cbk] 0-site-media-client-1: remote operation failed: No such file or directory

We would like to know what is causing these problems and to resolve them to prevent these interruptions to the service.

If you require any more information feel free to ask and I'll provide what i can.

The requested additional information is below, the two servers are identical:

Ubuntu 12.04.2

Linux VDED-XXX-XXX 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28 00:28:53 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

glusterfs 3.2.5 built on Jan 31 2012 07:39:59

VMWare ESX Servers

Errors From The Brick Logs at the time of the crash/disconnection/problem:

site-media brick log:

[2013-05-02 11:32:00.849296] I [server3_1-fops.c:964:server_unlink_cbk] 0-site-media-server: 9109306: UNLINK /catalog/product/cache/1/image/1000x1000/9df78eab33525d08d6e5fb8d27136e95/v/e/some-stuff-091574183930-box.jpg (0) ==> -1 (No such file or directory)

[2013-05-02 11:32:02.86607] I [server3_1-fops.c:964:server_unlink_cbk] 0-site-media-server: 9109345: UNLINK /catalog/product/cache/1/image/1000x1000/9df78eab33525d08d6e5fb8d27136e95/v/e/some-stuff-091574098692.jpg (0) ==> -1 (No such file or directory)

[2013-05-02 11:32:02.105131] I [server3_1-fops.c:964:server_unlink_cbk] 0-site-media-server: 12553441: UNLINK /catalog/product/cache/1/image/1000x1000/9df78eab33525d08d6e5fb8d27136e95/v/e/some-stuff-091574097992-box.jpg (0) ==> -1 (No such file or directory)

[2013-05-02 11:32:02.485694] W [inode.c:1044:inode_path] (-->/usr/lib/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve+0xf8) [0x7f4534639418] (-->/usr/lib/glusterfs/3.2.5/xlator/protocol/server.so(server_resolve_inode+0x70) [0x7f4534639290] (-->/usr/lib/glusterfs/3.2.5/xlator/protocol/server.so(resolve_loc_touchup+0x105) [0x7f4534638425]))) 0-/var/gluster/wwrd-media/inode: no dentry for non-root inode 184269351: 11a65ece-7b4b-4364-a28c-63df686f5648

site-var brick log doesnt seem to contain any errors

  • Please post which distro, kerenel version, gluster version you are using. Are servers virtual? if yes: ESX, KVM, ??? Thanks May 3 '13 at 16:05
  • Ive added the requested information @DanilaLadner May 3 '13 at 16:34
  • Please need brick logs as well May 3 '13 at 16:48
  • I mean or anything unusual in them. May 3 '13 at 16:51
  • Ive added an exert from the site-media brick log which looks fishy and matches with the time of the crash the last line there looks like a possible cause and the others are just symptoms of the crash. The site-var brick log intrigues me as that doesnt seem to have any errors but the mount was broken just the same as the other one @DanilaLadner May 4 '13 at 13:38

Looking at the logs. It definitely seems like some kind of race condition as there is an intense writing to that folder. There were some bugs reported for these on 3.2.* Gluster Version. I suggest you to upgrade to 3.3.1, which is fully supported on your OS. A lot of bugs and performance issues were resolved. You can also do the upgrade from your version.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.