gluster libgfapi HA

Question

I was exploring KVM_qemu with gluster distributed storage. My goal was to build a test cluster with complete HA . i.e HA for VMs as well as gluster storage . I'm planning to setup a 3 node gluster cluster and use qemu gluster driver with libgfapi instead of old FUSE method.But then I notice that in vm xml file or qemu image create command we can only specify one gluster server IP !! So if that server goes down storage will be unavailable ? Then What should be the approach for providing HA while use qemu libgfapi method ? Assigning virtual floating IP to gluster servers with keeplived ?

Please advice .. Thanks

Answer 1

<driver name='qemu' type='qcow2' cache='none'/>
     <source protocol='gluster' name='kvm/test.img'>
        <host name='192.168.60.11' port='24007'/>
        <host name='192.168.60.12' port='24007'/>
        <host name='192.168.60.13' port='24007'/>
     </source>
      <target dev='vda' bus='virtio'/>      
</disk> 

You can install gluster in kvm node and make gluster peer probe kvm host. After this kvm host become a member of the cluster, and xml config

<driver name='qemu' type='qcow2' cache='none'/>
     <source protocol='gluster' name='kvm/test.img'>
        <host name='127.0.0.1' port='24007'/>
     </source>
      <target dev='vda' bus='virtio'/>      
</disk> 

Answer 2

This is less of a concern than it at first seems.

First of all, when specify a Gluster host IP during client mount (whether with FUSE or libgfapi), you're only using that for peer probing. Once that probing is completed, libfgapi handles communication between all probed peers, on the IP(s) that they provide when probed.

This is all great until the host that your probing by explicit IP goes down. There are a few things you can do to keep this from happening. Keepalived or ctdb can provide a floating IP that you would use only for client to server probing. Alternatively, a DNS round-robin will accomplish this with names.

This name or floating IP won't be used for actually sending data to the Gluster volumes, but will only be to ensure that probing can always happen even in a degraded environment.

With libvirt, this ends up looking like this:

<driver name='qemu' type='qcow2' cache='none'/>
 <source protocol='gluster' name='kvm/test.img'>
    <host name='gluster-probe' port='24007'/>
 </source>
  <target dev='vda' bus='virtio'/>      
</disk> 

In this case, I've implemented a DNS round-robin outside of libvirt so that I can be sure this name always resolves to a host in the gluster peer pool, provided that at least one is up. After it contacts one, it gets the full list, connects to the hosts in that list via IP, and then happily moves along.