For a given small HPC cluster (~16 nodes) a master node is used as a front-end for users to login and interact with SLURM, and not as a computing node. The master node is currently a bare-metal server. Since the cluster is so small, the idea came up of migrating the master node into a virtual machine instead of a bare-metal node, in order to have one extra computing node.
What would be the pros and cons of having a master node as a virtual machine for a HPC cluster?
In an HPC environment you don't want things that may slowdown your computing performance, but with that in mind you could have the headnode virtualized, and in fact this is a good approach if you can handle the downsides of it.
The advantages are the same as virtualizing servers, you can leverage snapshots, issue backups easily, grow the VM as needed and other good things that you usually do with VM's.
For the disadvantages here we go:
Usually you want the login node (or the headnode if users login on the headnode) to have the same CPU architecture of you HPC cluster. Why is that? So the users can easily compile software on the login node with optimization flags like -march=native and enable things like vectorization techniques like AVX2 and AVX-512 (if you are on latest Intel processors). On a VM you'll probably need to educate your users to specify the targeted CPU of your cluster, or of your queues, if you're running in a mixed CPU environment.
If there's any parallel filesystem, like Lustre and BeeGFS and those are only available on Infiniband, it may be difficult to provide access to it on a VM, you'll probably need to have IOMMU enabled to passthrough the Infiniband interface to the VM, and keep in mind that storage performance will be worse than on bare metal.
On the Infiniband topic, usually login nodes and headnodes can have a low latency fabrics interface, commonly Infiniband. That's not a requirement, but usually we add those cards just for easier operation. So you'll have the entire OpenFabrics stack on the login node where users submit jobs in the queue system with things like PMI/PMIx and the required libraries to compiler the software with correct MPI (Message Passing Interface) libraries. You can add those without the hardware, but it's easier to just have it.
The last point applies to visualization too. Some HPC environment have visualization nodes with Quadro GPUs for data visualization. You'll probably need to passthrough the GPUs to have proper support on data visualization software. It's possible, just more work is needed than just running it on bare metal.
Finally the network, perhaps the most challenging aspect of it is regarding the network. On bare metal headnodes and login nodes you usually just add network interface as is and try to use basic networking. Inside a VM you'll probably need to configure VLAN's inside the VM to reach the switch where your compute nodes are. There's the question regarding the out-of-band management, like IPMI/BMC/iDRAC/iLO/etc that should be reachable from the headnode if you want to properly control your compute nodes. So some networking planning is advised when running your headnode in a VM.
For the compute nodes you probably want to be running stateful instead of stateless, since stateless will create too much traffic on the headnode instead of just running the nodes stateful. On baremetal we usually use stateless because it's easier to maintain and we have a lot of idle resources on the headnode, but this may not be true if the headnode is on a VM.
That's the experience that I got managing HPC for some years now. I still have some headnodes virtualised but I prefer to keep them baremetal.
