How can I create a small cluster and include non dedicated servers?

Question

We want to set up a small cluster, witch will store some databases and testing services at first. The cluster will initially launched on 8 dedicated servers (HP G3/G5). 6 of the servers in in the same rack, and two of them is on individual separate locations. Outside this cluster we have 2-4 powerfull servers that mostly is not using all of their own resources. This got me thinking that these could be added to the cluster and perform cluster operations when they are not using 100% of it's resources. Another feature I want the cluster to have is to run docker containers as cluster services and of course that the cluster is HA and load balanced. I've been searching a lot on this subject, but can't really find a good answer.

The option that is strongest now is to install Ubuntu 14.04 and Apache Mesos and add in the non-dedicated servers as slaves in the cluster after Mesos have implemented dynamic slave attributes

Answer 1

Did you tried to look at the availability zones ? Maybe it could help you. About the containers, HA and LoadBalancing, could you explain a little more your need? Mesos is by design already redundant as it take care of tasks health and status, so you don't really need HA as you could see it on Virtualization platforms. The load balancing is also not really implemented as you think to use it on usual infrastructure as mesos cluster will automatically schedule your tasks depending of offers and demands it send / receive, the scheduler will look at the global available resources and make offers for your task dynamically.

In fact in mesos you rather think of redundancy and load balancing on your application design rather than on your infrastructure (even if you can build mesos master and slave on HA mode) but it's not really interesting.

I rather implement failures as a feature over there to be able to manage any problems or failures with the infrastructure. For example, if you want something relatively safe you will need a minimum of 3 servers for the mastering part plus 6 servers for the slave part.

On my infrastructure, we do use small Linux plus docker (often referenced as atomic distributions now a day) on masters and minimal Linux (CentOS/RHEL 7 with strictly essentials binaries and tools, but not atomic distributions) plus mesos-slave on top.

This allows us to secure our infrastructure as if anyone of the slaves or master died, there will be a detection of the failure by another member and the attached services or tasks would be invalidated, then restarted on another node depending of your application failure and recovery policies.

But hey, once again, this is my setup and there is probably way better implementation out there.