We are running GlusterFS replication server on 2 nodes and we have 2 clients. Self healing daemon is enabled. Each client is connected to a different server using the Gluster client. We have a lot of very small files on the Gluster volume.
We use GlusterFS 3.9.1 from the official GlusterFS APT Repositories on a Debian Jessie system.
The issue we are facing with is that one of the servers is at 0.1-0.5 load average while the other is at 200. The server with the high load also has a huge amount of data being streamed constantly to both the client nodes. This data stream continues even when the clients are not reading writing data.
The following values are measured by
server: outgoing 35-40 MB/s
Two clients: incoming 17-20 MB/s
Our performance on the Gluster client is very poor. An
ls can take up to 10 second to complete, and our app works extremely slow.
Servers and clients are connecting over an internal data-center network and can handle much more bandwidth so this is not a limiting factor.
My two main questions are :
1: Are these differences in server load normal behavior for GlusterFS and what causes this?
2: Why is there such a high constant data stream to clients form one of the servers.
I cannot seem to find any information concerning this in the Gluster Documentation or on the Internet.