Today we faced a strange error while using gluster.
The gluster log was full of
[2016-04-18 03:54:32.164587] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 128: LOOKUP() /jobs/infrastructure/config.xml => -1 (Input/output error)
While the permissions, user and group were full of question marks.
Interestingly enough the setup was partially functional as I was able to create files and directories from one client and see them on the other, but not the other way around.
We suspect this was sort of a split-brain condition which might have been caused by out of sync clocks. Notably the servers were on different time zones, so the difference was in the hours. Running the split brain detection commands on the server didn't produce any interesting result.
Any suggestion on what might have caused this?