1

I just installed GlusterFS on two nodes it seems working fine the status is up and also the replication is working fine.

But when I stop the node1 volume cluster volume stop glustervol1` or shut down the server to simulate a failure, the second node doesn't take over.

I followed these instructions to install GlusterFS HERE

EDITED

I am using VMware with 7 Vm and CentOS 7.

App-Master : LampStack, Remote DB in Mysql-Master, Storage in Gluster 1.

gluster1:/glustervol1       /home/wordpress/  glusterfs   defaults,_netdev  0  0

App-Slave : Lampstack, Remote DB in Mysql-Slave, Storage in Gluster 2.

gluster2:/glustervol1       /home/wordpress/  glusterfs   defaults,_netdev  0  0

Mysql-Master : Mysql.

Mysql-Slave : Mysql (Replication).

Gluster 1 : GlusterFS server where the volume is created.

Gluster 2 : GlusterFS the Replica of Gluster 1.

LoadBalancer : Nginx Load Balancing between App-master & App-Slave

*****INFO*****

172.16.172.147 gluster1
172.16.172.148 gluster2
172.16.172.146 appslave
172.16.172.143 appmaster

Logs From App-Master server

After Switching Off Gluster1

[2016-11-21 16:36:18.532124] W [socket.c:642:__socket_rwv] 0-glusterfs: readv on 172.16.172.147:24007 failed (Connection timed out)
[2016-11-21 16:36:18.532125] W [socket.c:642:__socket_rwv] 0-glustervol1-client-0: readv on 172.16.172.147:49152 failed (Connection timed out)
[2016-11-21 16:36:18.532323] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-glustervol1-client-0: disconnected from glustervol1-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-11-21 16:36:31.965015] E [socket.c:2332:socket_connect_finish] 0-glusterfs: connection to 172.16.172.147:24007 failed (No route to host)
[2016-11-21 16:36:31.965141] E [socket.c:2332:socket_connect_finish] 0-glustervol1-client-0: connection to 172.16.172.147:24007 failed (No route to host)

After Turning On Gluster1

[2016-11-21 16:39:02.258175] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-11-21 16:39:02.258595] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-glustervol1-client-0: changing port to 49152 (from 0)
[2016-11-21 16:39:02.262348] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-glustervol1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-21 16:39:02.299637] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-glustervol1-client-0: Connected to glustervol1-client-0, attached to remote volume '/bricks/brick1/brick'.
[2016-11-21 16:39:02.299714] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-glustervol1-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-11-21 16:39:02.300513] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-glustervol1-client-0: Server lk version = 1

After Switching Off Gluster2

[2016-11-21 16:41:33.394122] C [rpc-clnt-ping.c:161:rpc_clnt_ping_timer_expired] 0-glustervol1-client-1: server 172.16.172.148:49152 has not responded in the last 42 seconds, disconnecting.
[2016-11-21 16:41:33.394943] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fd4ad63c906] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd4ad40792e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd4ad407a3e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fd4ad4093fc] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fd4ad409c08] ))))) 0-glustervol1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-11-21 16:40:50.706048 (xid=0x59e)
[2016-11-21 16:41:33.394973] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-glustervol1-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2016-11-21 16:41:33.395188] E [rpc-clnt.c:362:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fd4ad63c906] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd4ad40792e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd4ad407a3e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7fd4ad4093fc] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fd4ad409c08] ))))) 0-glustervol1-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-11-21 16:40:50.706053 (xid=0x59f)
[2016-11-21 16:41:33.395201] W [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-glustervol1-client-1: socket disconnected
[2016-11-21 16:41:33.395211] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-glustervol1-client-1: disconnected from glustervol1-client-1. Client process will keep trying to connect to glusterd until brick's port is available

After Turning On Gluster2

[2016-11-21 16:41:45.255081] E [socket.c:2332:socket_connect_finish] 0-glustervol1-client-1: connection to 172.16.172.148:24007 failed (No route to host)

NB : The Failover worked even even if Gluster1 or Gluster2 was down the App kept working.

Improve this question

2 Answers 2

Reset to default
2

This isn't exactly how GlusterFS works, as it doesn't "fail over". No single node acts as a master, as GlusterFS is masterless. The client is responsible for connecting to all Gluster peers, and it's the client that must maintain network visibility to servers in order to maintain sane volume activity. Inter-node communication is usually only used during volume heal operations and for negotiating peer trust.

If you're following that guide, I'm assuming you've created a replica 2 gluster volume from two bricks. What does your configuration look like, and what does the overall network look like that's connecting these Gluster nodes together, as well as what's connecting them to the clients? Most problems with Gluster have to do with client to server communication problems.

Improve this answer
6
  • information added up
    – Dolomats
    Nov 19, 2016 at 19:59
  • What do you mean by saying your app master has storage in Gluster 1 while your app-slave is in Gluster 2? They should both be in Gluster 1 and 2, and should be accessing them simultaneously. It's the clients that maintain volume replicas. Gluster 2 is not a replica of Gluster 1, they are replicas of each other and are read / written completely simultaneously. A two node replica 2 Gluster volume doesn't have any concept of quorum, automatic self-heal, or read-only service freeze on peer loss. This isn't great for consistency, but it does stay up on a node failure.
    – Spooler
    Nov 20, 2016 at 2:47
  • Also, what exactly do you mean when you say that the second Gluster node doesn't take over? What is the exact failure, and what do your logs look like during this failure?
    – Spooler
    Nov 20, 2016 at 2:48
  • I added the fstab entry for each server to know where is each one storage. All the write is done on Appmaster/Gluster1 and AppSlave/Gluster2 is the replica of the AppMaster To loadBalance the charge between them and in case 1 is down the second take the relay. I mean by "Gluster node doesn't take over" that when I switch off a node the second doesn't replace it like when u have LoadBalancing if u switch 1 server the second keep taking all the charge.
    – Dolomats
    Nov 20, 2016 at 13:37
  • You're still not describing your failure beyond stating that you have one. What do your logs read? They're in /var/log/gluster/, and in dmesg. Also, GlusterFS should not be behind a load balancer, so be sure you're not using load balancer VIPs for it. Gluster expects its IP addresses to not change or float.
    – Spooler
    Nov 20, 2016 at 14:08
2

Obviously you forgot a parameter

you should add backupvolfile-server for mounts

example:

gluster1:/glustervol1       /home/wordpress/  glusterfs   defaults,_netdev,backupvolfile-server=gluster2  0  0
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.