How to make KVM image like in EC2 AMI

Question

We have few baremetal servers(with debian os), where we run multiple(100+) KVM machines - mostly debian and centos. Those machines are on LVM or distributed file system(moosefs&glusterfs)

Is it possible to create some environment to automate creation of machines where we can specify:

• hostname
• disk size
• other (cpu, network, ram) are optional

Now we are creating new machines every time by hand and its boring because we have to make fresh install every time. We would like to make something similar to Amazon Machine Images

I installed proxmox for a second, but i havent seen functionality similar to AMI. But maybe i am wrong...

Answer 1

My standard procces (by hand)

1. I create a base image.
   • From the perspective of the host machine is only a .img or .qcow2 file.
2. I start duplicating the base image
3. Then I change the size of the image if needed
   • For .qcow2 I use qemu-img convert utility.
   • For .img I use dd to append more zeros in the end of the image.
4. Then I go to virt-manager interface and create a new virtual machine "using existing image". On that I set how much processing and memory I need.
5. I boot the machine image and then
   • I change the IP addr
   • I change the Hostname
   • I remove the sshd keys to force sshd generate new ones
   • I resize the / partition to use the whole disk (this is unecessary if the guest image uses btrfs)

All of this can be automated by a simple shell script. The key is to learn how to use the virsh utility for the creation of the vms, make sure your networks interfaces are bridges, use btrfs as guest filesystem. Then you can duplicate the image, mount the image, change the configurations of the mounted image, unmount the image, add the vm with virsh and run.

I did not do that yet because I manage few VMs... actually I use KVM to partition a huge hardware in two or three very large ones. I am a bit autistic in that part. I preffer to make my own scripts to have a high degree of control in those aspects.

More automated and user friendly

What you are looking for is called openstack, that as I understood is very user friendly and can use KVM, VirtualBox, LXD, Xen, VMWare as hypervisors. One interface to bind them all.

I know it exists. It seems good. Ubuntu server has it. Redhat also. But I never touched one because virt-manager is enough to me.

Snapshots

The same thing Iain suggested to be done by ZFS, can be done with standard KVM. QCow2 supports snapshots. Meaning that for clusters you can make a base image, create the snapshot, mount the snapshot, change the configurations (IP, hostname), unmount and start. The best part is that only the changed sectors will be written to the snapshot.

Notes

My experience is with 160 logical processor / 2048 GiB virtualization hosts divided in up to 10 virtual machines. All of these are linux servers (apache/php/postgres/java). I have no experience whatsoever in hundreds of small virtual machines in a big hardware.

By the way. I am migrating from Debian and RedHat to Ubuntu. Most of what I have is running on Ubuntu Server for years now. The Legacy Debians will die this year. The legacy RedHat next year. There is no much difference between Debian and Ubuntu from the server perspective, I choose ubuntu because new things gets packaged first on ubuntu.

ZFS

I tested ZFS on linux couple years ago and do not fell that was good enough to run serious servers on it. I choose to wait for BTRFS and have notthing to complaint. I am using it on some of my servers and my workstations. I will move my bigger servers to it early next year to improve my backup.

Conclusion

If your setting is standarized, maybe is worth to take the time and make your own scripts because it will be very automated. However, if you need several kinds of appliances and deal with different client needs you may consider learning an solution like openstack.

Answer 2

I created a base image then took a ZFS snapshot of it. I then use a ZFS clone of the snapshot for each VM. I also whipped up a script to do this and change various parameters for the system build, change the hostname etc.

Takes just a few seconds to have a VM up and running.