Hello Serverfault community,
I'm currently designing an OpenStack cluster. The part where I'm currently stuck is the Storage Architecture.

I thought of building two redundant Ceph clusters in different racks with a different fuse and UPS. So far so good. The thing is, Openstack Cinder communicates via the network with Ceph. Even when I have redundant switches in place what if the router/gateway of the Ceph network goes down. So in order to achieve full redundancy I need two redundant gateways, am I right?

I have made an abstract image of what I'm trying to do: Storage Architecture

On the left you can see the OpenStack Compute nodes, everyone connected via two separate 10GE Links to each switch. Those switches are connected via 2x 10GE trunk to each of the dedicated Ceph cluster switches who are connected with each other, so Ceph nodes can communicate with each other for redundancy. Note, that even when I have used different colors this whole network topology is in one VLAN.

From my point of view, everything can fail and the cluster would still be operating normally. But what if the gateway of the storage network fails? Then I guess it would work as long as the switch has cached MAC ARP entries but when they expire the cluster would go down. Do you have any suggestions for me how to make the cluster fully redundant and optimize it?

I am only focusing on the storage component of OpenStack, therefore the Compute nodes are not connected to the network node or management network yet.


  • Is there a specific reason why you want two separate CEPH clusters? – maxf Aug 24 '16 at 12:08
  • I don't want two separate Ceph Clusters. It should only show that I don't want the replica to be in the same rack as if for example the fuse for the rack would have an issue the replica would also be down. – Phinitris Aug 25 '16 at 15:34

For the CEPH part of your question: CEPH uses the CRUSH map to decide where objects are stored, which you can customize. Marking OSDs with the rack they belong to and preventing copies of the same object to be stored in the same rack is actually quite common. For more on this, look here. Therefore you don't need two separate CEPH clusters. For network configuration, CEPH nodes should be in a "Cluster Network" thats accessible from your Cinder nodes and have a private Cluster network thats used for replication between OSDs.

As for Cinder: Yes, if Cinder goes down you loose access. Therefore you'll want to deploy Cinder in a HA configuration.

  • Thanks. The point I was trying to make is that the Cluster Network needs to be redundant too through an external device/HA router configuration, am I right? I would indeed deploy Cinder on HA controller nodes. – Phinitris Aug 26 '16 at 16:47
  • The cluster network is normally deployed using e.g. redundant switches and LACP. It's not routed anywhere since it's only used by CEPH internally, so I'm not sure what you mean? – maxf Aug 26 '16 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.