I have been using a suite of filesystem testing tools to benchmark and abuse a GlusterFS volume. The volume is a replica 3 volume spread out over 6 hosts.
Fio, iozone, and Bonnie indicate to me that Gluster is working just fine and the bandwidth is roughly equal to that of the client and server network adapters, so performance can't really be improved. Most of my test cases operated on 32gb files, apart from iozone and Bonnie.
I have gotten reports of split brain occurring for certain files which are being concurrently written to by multiple clients. All of the documentation I have read seems to indicate that split brain largely occurs when network partitions happen, and this is clearly not the case, judging from the logs.
Unfortunately, this split brain seems to occur only when using a certain hosted service, and I have zero introspection into how that service operates, what version of Gluster client it has, etc. The servers are running the latest 4.0 release.
Judging from the failure case I have been presented with ("split brain happens when two containers are writing to the same file at the same time"), I need a test that will reproduce a similar situation.
I could definitely write my own test case in C or Rust, but is there something out there which will test this exact case without having to write anything?
I do have access (but not introspection) into this hosted service, so I will probably test that too. I'm also scratching my head at the actual problem: what is the desired outcome when two programs write different data to the same file at the same time?
EDIT: The servers are running the latest CentOS 7 release. My testing client server is also running the same. The underlying filesystem is XFS.
Is there a specific test case that I can use to try to recreate the problem?