I have a 2-node HA GlusterFS setup. About 15 Virtual machines mount the GlusterFS volumes. The GlusterFS servers are running on Debian 5 Lenny; the client are running on Debian 6 Squeeze; all are running GlusterFS 3.0.5 (because it is available in the Debian repositories).
I have noticed that a few hours after editing them, several files keep getting deleted or replaced with previous versions of the files.
Sometimes, but not always, I find an entry in a logfile in one of the clients:
[2011-07-31 06:25:16] W [fuse-bridge.c:493:fuse_entry_cbk] glusterfs-fuse: LOOKUP(/puppet/support_modules/sudo/manifests/init.pp) inode (ptr=0x7fded445aa00, ino=13896088, gen=5592218346205480808) found conflict (ptr=0x12b5eb0, ino=13896088, gen=5592218346205480808)
The time of day for these messages is always the same - 06:25h. At that time, one of the VMs is running a backup (using backup2l) which includes the mounted GlusterFS volume. Could this be relevant?
It is so frustrating. At the moment, I'm seriously considering kicking out GlusterFS, which otherwise works actually fine.
EDIT: I can confirm that during a backup, a couple of error messages end up in the logfile:
[2011-08-09 12:18:23] E [afr.c:107:afr_set_split_brain] mirror-0: invalid argument: inode
