I am trying to make a 2 node linux servers with high available apache using glusterfs 3.7.6 for data replication and pacemaker+corosync as resource manager. However i'm seeing a problem with gluster in the specific scenario when both nodes get shutdown and one of them comes online first. Even though there's a brick on that node and gluster service is running, there's no brick process.

[root@node1 ~]# gluster volume status data 
Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data                   N/A       N/A        N       N/A  
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on localhost                     N/A       N/A        N       N/A  

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

And when I start the other node, everything seems to work and I can mount the volume.

[root@node1 ~]# gluster volume status data
Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data                  49152      0          Y       2674 
Brick node2:/gluster_data                  49152      0          Y       3086 
NFS Server on localhost                     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       2796 
NFS Server on node2                         N/A       N/A        N       N/A  
Self-heal Daemon on node2                   N/A       N/A        Y       3085 

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

And at this point if I shut down node2, node1 brick process remains active so atleast I can mount and use it.

[root@node1 ~]# gluster volume status data

Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data                   49152     0          Y       2674 
NFS Server on localhost                     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       2796 

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

So my observation is that for the gluster volume to work, both nodes need to be online atleast for a moment so the bricks can start and then if one node goes down, it won't affect the volume operation. So how can I make it work when one of the nodes come online from a total blackout ?

up vote 7 down vote accepted

The problem any cluster-node has when coming up from a full stop is:

Do I have the latest state, or not? I don't want to claim latest if I'm behind the other down nodes.

This is why clustering very often includes some kind of quorum mechanism, so existing nodes can vote on state and converge on a consensus. Two node clusters can't use this mechanism, since there will never be a 'majority' partition. In the 3.7 release, Gluster gained a quorum feature.

http://gluster.readthedocs.org/en/release-3.7.0beta1/Features/server-quorum/

In that document, they state that 2-node clusters can't use it for the very reason I described above.

In your case, you may want to consider creating some management-only nodes in your Gluster setup. These would be peers that are probed into the cluster, but don't host any storage. Their entire reason to exist would be to maintain cluster-state. These can exist in different racks, datacenters, power phases, to try and ensure they are in a different fault-domain than the storage bricks. This will increase the number of members in a cluster, and will improve your chances of having a majority partition if one of the storage-bricks comes up without the other one.

Unfortuonately, the behavior you're seeing is working as designed and can't be changed without adding more servers into the cluster.

  • 1
    Thanks a million for the excellent explanation !! This is the best kind of answer with both explanation of the underlying reason and suggestion on how to avoid it. Thanks a lot again. – Anindya Feb 3 '16 at 10:24

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.