Hello Serverfault community,
I'm currently designing an OpenStack cluster. The part where I'm currently stuck is the Storage Architecture.
I thought of building two redundant Ceph clusters in different racks with a different fuse and UPS. So far so good. The thing is, Openstack Cinder communicates via the network with Ceph. Even when I have redundant switches in place what if the router/gateway of the Ceph network goes down. So in order to achieve full redundancy I need two redundant gateways, am I right?
I have made an abstract image of what I'm trying to do: Storage Architecture
On the left you can see the OpenStack Compute nodes, everyone connected via two separate 10GE Links to each switch. Those switches are connected via 2x 10GE trunk to each of the dedicated Ceph cluster switches who are connected with each other, so Ceph nodes can communicate with each other for redundancy. Note, that even when I have used different colors this whole network topology is in one VLAN.
From my point of view, everything can fail and the cluster would still be operating normally. But what if the gateway of the storage network fails? Then I guess it would work as long as the switch has cached MAC ARP entries but when they expire the cluster would go down. Do you have any suggestions for me how to make the cluster fully redundant and optimize it?
I am only focusing on the storage component of OpenStack, therefore the Compute nodes are not connected to the network node or management network yet.