A client requests me to make a robust system to run containerized microservices within their LAN.
Restriction: They give me 2 machines and 6 data-disks. No more hardware. I have to build "the best I can" here.
My worries are the storage and availability. Speed/performance is not an issue.
I'm thinking of:
- In each machine build a RAID-5 using 3 data-disks, yielding in one data-volume in each machine. ZFS for example.
- Tie the 2 machines with a distributed filesystem. GlusterFs for example.
- Then use Kubernetes to create a cluster of 2 nodes pointing their Persistent Volumes to the distributed FS.
- The fact the kubernetes cluster runs in the same hardware than the distributed filesystem is a mere casuality.
Question is: Any better solution given the restrictions of the client?
Read the context to understand more.
I'm designing a server architecture for running a bunch of 30 microservices locally for a radio station. No AWS, no cloud. We are talking about on-premises server.
For the whole scenario speed is not an issue (traffic is low). The business drivers here are:
- Data persistance (minimize the risk of loosing data).
- High availability (minimize the risk of a downtime).
If at any point they are incompatible, avoiding data-loose takes precedence over high-availability: In the limit, I can tell the journalists to stop working for a few minutes; but we can't loose the interview that was recorded earlier this morning.
What they currently run (don't cry, please!!)
They run now all in one server with no containers and no data redundancy beyond backups. They experienced a disaster on 2018 and took them 2 full days to recover. The radio had to stop all the employees working, re-install the full OS, reinstall all the applications by hand, recover all the data from the backups, test everything then thell the journalists "back to write news".
Of course this is not acceptable in those days (even it was not acceptable in 2018).
The hardware they have for this project
To overcome this, they recently bought 2 servers with 1 system disk + 3 data disks on each (total 6 data-disks). The initial idea they had for the data-disks is to make a local software RAID-5 across the 3 data-disks within each server.
Call the servers
beta. The two machines are the same in cpu, ram and system-disk as well as the 3 data-disks. So the computers are exact clones. They will both run Ubuntu linux.
Master-slave (client's idea) vs cluster (my proposal)
The idea they had is to use
alpha as the main server and make
beta be a "clone" of
alpha so if it dies they can switch over the clients to
beta in half an hour by manually reconfiguring the clients to point to another IP.
Nevertheless I think that the current technologies should enable me to create some sort of cluster so they are both alive and fully-in-sync so if anyone of those break, the clients experience zero-downtime.
The radio station initially requested me to build a RAID on
alpha, via ZFS and then another RAID on
beta via ZFS. Set a bunch of dockers on
--restart=always and then point the clients of the services (running in their journalists' respective PCs) to
alpha (think services as news writing, image uploading, audio recording, program scheduling, web publishing, media transcoding, local live stream feed to the cloud, etc.).
About the storage, their initial though was:
- For photos and audios, make regular backups from
- For MySQL have a master-master in
betais mostly acting as a slave but ready to be used in case of
Then if alpha breaks, switch all the clients to
I'm specifically interested in the storage part of the story.
What I'm thinking
Instead of "manually switching clients" I was thinking of using kubernetes to make a cluster of 2 worker nodes. As I can't have separate hardware to act as "kubernetes master" I was thinking of making also
beta be both redundant kubernetes masters of themselves as workers.
alpha would be a kubernetes master for
beta nodes, as well as
beta would be a redundant master of kubernetes also for both
Up to here no problem.
Until we arrive to the storage.
Storage, persistent volumes, GlusterFs and ZFS
When it comes to
Persistent Volumes in kubernetes, the users of kubernetes launching their pods/containers need to rely that the data will not be lost. Some system administrator (in this case me) needs to "build" the redundancy below to ensure the volume "is" there with the proper data.
So this is what I was thinking:
1. A local ZFS layer
In the operating system of
alpha(native to the system, forget kubernetes for a second) use ZFS to make a RAID across the 3 data-disks (equal in size). So if each disk is, say 1TB, there are 3TB of which 2TB will be available in the data volume and 1TB is under the hood for redundancy. Let's call the disks A1, A2 and A3. Let's call the ZFS volume A.
beta, replicate the structure. Disks B1, B2, B3. Let's call the ZFS volume B.
Up to here I'd have have 2 independent servers each protected against a single failure of a single disk. No protection against 2-disk simultaneus failure. No protection against a full-node down.
2. A distributed GlusterFs layer
Then create GlusterFs across
betaon top of the ZFS volumes. I understand that GlusterFs has to give me some sort of
mirroringconfiguration, so the ZFS volumes A and B are one mirror of the other. So if A is 2TB and B is 2TB the "total available storage" is also 2TB for use.
So adding up GlusterFs and ZFS at this moment, from the 6TB in total hardware capacity, 2TB are available for users and therefore 4TB are acting as redundancy.
Up to here, I should have a "distributed disk" that has much more redundancy and allows failure of 2 disks and also node-failure.
I see protection of 2 disks failing in the following manner:
- If the two disks pertain to different volumes (say fails A2 and B3) then each NFS separately protects against that and both ZFS volumes A and B are not disrupted (GlusterFs sees no changes).
- If the 2 disks failling belong to the same node, then the full volume is failing. For example a failure in A2 and A1 makes A broken. But GlusterFs should be able to balance to use "only 1 node" until the other becomes available (in this case "use only B until A comes back again").
3. Kubernetes container runtime for service cluster + Kubernetes Persistent Volumes.
Finally, use Kubernetes Persistent Volumes would point to the GlusterFs volumes.
If I had 4 machines, probably I'd use 2 as kubernetes nodes and 2 for storage acting as a networked storage to the cluster.
But we only have 2 physical machines so kubernetes will point "persistent volumes" to "GlusterFs" exactly as if they were "in another remote machine" making it agnostic that the volumes are physically in the same nodes.
I have never physically built a construct like this before. Over the paper, it works. I wonder if the reality is different.
Given the constraints (2 machines, 6 data-disks), question is:
- Is this topology the best way to create a mini-cluster with zero-downtime and data-redundancy for the client?
- If not, what changes should I apply and why.