Getting Ubuntu 16.04 on a ZFS root on a Hetzner dedicated server

Question

By now, it's well possible to get Ubuntu 16.04 running on a ZFS root-fs. Ubuntu 16.04 has ZFS in the default package manager, and with guides like this, it's not hard to get started.

However, all guides I've seen require being able to boot from a Ubuntu installation image. For a Hetzner dedicated server, this is an uncommon installation procedure, as it requires engineers to visit the server and plug-in a remote KVM.

By default, the dedicated servers boot into a rescue system, which allows to install a variety of Linux distributions through their 'installiamge' script. However, this script does not support ZFS yet.

How to get a Hetzner dedicated server running on a ZFS root?

Answer 1

The basic idea is to get Ubuntu installed on a small partition on the harddrive, partition the hard-drive to use the remainder of the space for ZFS, and then copy the installation over. I'm mainly using this guide on instructions how to do that.

Lazy, and experience with Ansible? I wrote a little stack of scripts to automate these steps. They are available on: https://github.com/tijszwinkels/hetzner-ubuntu-16.04-zfs-root-ansible/blob/master/hetzner-ubuntu-16.04.yml Be careful, these scripts assume that the host is booted into the Hetzner rescue system, and they will wipe your drives as a very first step. Use at your own risk!

# SSH into the host.

# Wipe the drives. Assuming SSDs on 'sda' and 'sdb'.
/sbin/blkdiscard /dev/sda
/sbin/blkdiscard /dev/sdb

# Install Ubuntu 16.04 on a 4G partition using the Hetzner 'installimage' script
/root/.oldroot/nfs/install/installimage -a -n my-hostname -r yes -l 1 -p /:ext4:4G -K /root/.ssh/robot_user_keys -i /root/.oldroot/nfs/install/../images/Ubuntu-1604-xenial-64-minimal.tar.gz

# Reboot the system.
/sbin/shutdown -r now

# Wait for the host to come back up, and SSH in.

# Install the 'parted' parition editor
apt-get update && apt-get install -y parted

# Run parted on the first drive, create a partition in all remaining space. (UNTESTED!)
sudo parted /dev/sda
(parted) mkpart primary 4097MB -1s
(parted) quit

# Run parted on the second drive, create a partition in all remaining space. (UNTESTED!)
sudo parted /dev/sdb
(parted) mkpart primary 4097MB -1s
(parted) quit

# Install required ZFS packages
apt-get install -y zfs-dkms zfs-initramfs

# Create a ZFS pool named 'tank'
# Please note that I'm using the /dev/disk/by-id interface. This is more resilient than /dev/sda and /dev/sdb
zpool create -f -o ashift=13 -O atime=off -O dedup=off -O compression=lz4  tank mirror `ls /dev/disk/by-id/ata-*-part2` 

# Create OS partiton
zfs create tank/os

# Rsync the current system to the new partition.
rsync -a --one-file-system / /tank/os/

# Chroot into the system
cd /tank/os
mount --bind /dev dev
mount --bind /proc proc
mount --bind /sys sys
mount --bind /run run
chroot .

# Install GRUB into the drives
export ZPOOL_VDEV_NAME_PATH=YES
update-grub
grub-install /dev/sda
grub-install /dev/sdb

Now, you should have a Hetzner dedicated server that happily boots into Ubuntu 16.04 with a ZFS root fs. Good luck!

Answer 2

Magnificent guide, thank you @TinkerTank.

If you are using one of the newer NVMe Servers and following this guide, you might want to use these steps and device names instead:

# Boot into rescue from Robot
# SSH into the host.

# Wipe the drives. Assuming SSDs on 'sda' and 'sdb'.
# For SSD Servers
#/sbin/blkdiscard /dev/sda
#/sbin/blkdiscard /dev/sdb
# For NVMe Servers
/sbin/blkdiscard /dev/nvme0n1
/sbin/blkdiscard /dev/nvme1n1

# Install Ubuntu 16.04 on a 16G partition using the Hetzner 'installimage' script
# For Ubuntu 18.04 it should be this, but FAILS TO BOOT
#/root/.oldroot/nfs/install/installimage -a -n my-hostname -r yes -l 1 -p /:ext4:16G -K /root/.ssh/robot_user_keys -i /root/.oldroot/nfs/install/../images/Ubuntu-1804-bionic-64-minimal.tar.gz
# For Ubuntu 16.04:
/root/.oldroot/nfs/install/installimage -a -n my-hostname -r yes -l 1 -p /:ext4:16G -K /root/.ssh/robot_user_keys -i /root/.oldroot/nfs/install/../images/Ubuntu-1604-xenial-64-minimal.tar.gz
# Press x to continue immediately or wait a few seconds...

# Reboot the system.
#/sbin/shutdown -r now

reboot

# Wait for the host to come back up, and SSH in.
# Update the server
apt update
apt upgrade

# Create a partition on first disk with all the remaining space.
fdisk /dev/nvme0n1
# Press n then accept all defaults and save with w 

# Create a partition on the second disk with all remaining space.
fdisk /dev/nvme1n1
Press n then accept all defaults and save with w 

reboot

# Install required ZFS packages
apt install zfsutils-linux

# Create a ZFS pool named 'tank'
# Please note that I'm using the /dev/disk/by-id interface. This is more resilient than /dev/sda and /dev/sdb
# For SSD servers @hetznet: zpool create -f -o ashift=13 -O atime=off -O dedup=off -O compression=lz4  tank mirror `ls /dev/disk/by-id/ata-*-part2` 
# For NVMe servers:
zpool create -f -o ashift=13 -O atime=off -O dedup=off -O compression=lz4  tank mirror nvme0n1p2 nvme1n1p2

# Create OS partition
zfs create tank/os

# Rsync the current system to the new partition.
rsync -a --one-file-system / /tank/os/

# Chroot into the system
cd /tank/os
mount --bind /dev dev
mount --bind /proc proc
mount --bind /sys sys
mount --bind /run run
chroot .

# Install GRUB into the drives
export ZPOOL_VDEV_NAME_PATH=YES
update-grub
# For SSD Servers:
#grub-install /dev/sda
#grub-install /dev/sdb
# For NVMe servers:
grub-install /dev/nvme0n1p2
grub-install /dev/nvme1n1p2

exit
reboot

Now, you should have a Hetzner dedicated server that happily boots into Ubuntu 16.04 with a ZFS root fs. Good luck!

Works seamlessly on Ubuntu 16.04, fails to boot on Ubuntu 18.04. Anyone knows why and how to solve it?

Notice: instead of 4GB of HDD Space specified in the original tutorial, I launch the initial OS on a 16GB partition. Newer servers come with at least 2 x 512GB NVMe, so you can convert those 2 x 16 partitions onto 2 x 16GB SWAP partitions that Linux kernel will stripe by itself (faster than using one single Raid partition).