5

I have a virtual server running in qemu on proxmox. It has two physical disks attached to the host using a virtio: directive in 101.cfg as folllows

virtio2: /dev/disk/by-id/ata-vol1,backup=0 
virtio3: /dev/disk/by-id/ata-vol2,backup=0

These contain a mirrored ZFS volume from which my files are served using nextcloud. This all works fine because these disks are never removed from the guest.

I also have a rotation of three other physical disks to which I back up (the physical controller is an LSI raid controller in JBOD configuration, the physical hardware is an HP Proiliant DL380e Gen8). I do this by zfs sending an updated snapshot from the storage array to whatever backup disk is present on the system. I had added these to the VM as follows:

virtio4: /dev/disk/by-id/ata-backup1-volname,backup=0
virtio5: /dev/disk/by-id/ata-backup2-volname,backup=0
virtio6: /dev/disk/by-id/ata-backup3-volname,backup=0

This sort of works but the problem is that none of those disks are hot swappable, so if I remove and insert disks in rotation, the guest stops being able to see them. I have disk hotplug enabled in my qemu config.

My question is what is the correct way to dynamically attach and detach these disks to my VM as they are inserted and removed?

I have been able to work out a partial solution using "qm monitor" to attach and detach the disk I am taking out by attaching and detaching virtio-scsi "pci cards" as follows:

To attach

qm> drive_add 0 file=/dev/disk/by-id/ata-<disk-id>,if=none,id=backup_vol,cache=none,detect-zeroes=on
qm> device_add virtio-blk-pci,drive=backup_vol,id=backup_scsi_controller

To detach

qm> device_del backup_scsi_controller

(Source: https://blog.chrishowie.com/2019/09/19/hot-swapping-virtio-disks-on-qemu/ via Google cache)

I was thinking I could potentially add these to my backup scripts, but it doesn't seem to be possible however to run qm monitor commands from a shell script, and qm monitor will not take arguments or input from stdin. From reading aroud, it might be possible to create a socket into qemu-agent on the guest and inject this using JSON, but I know nothing about JSON and am really hoping there is an easier way.

2

I have been somewhat around the houses on this one, and indeed did get to the point where I was able to send QMP commands to a guest via a UNIX socket. This is supported by the default Proxmox config using a socket at /var/run/qemu-server/<server-id>.qmp. I can then interactively send QMP JSON messages using

rlwrap -C qmp socat - UNIX:/var/run/qemu-server/101.qmp

(rlwrap is not installed by default, but can be installed using "apt install rlwrap")

However, the QMP message format was as opaque as I thought it was going to be and has no direct equivalent to "drive_add" for reasons of ideological purity (which are perhaps understandable but nonetheless hugely unhelpful in the context of my present difficulty). There is a "blockdev_add" command that may well have done what I wanted assuming I could work out the correct syntax.

I therefore took a step back and came to the conclusion that "qm monitor" probably wasn't accepting input from stdin because of readline support. This lead me to think of "expect". A quick "apt install expect" later, and I was able to create scripts to attach and detach drives as follows:

qemu-drive-attach <vm-id> <device-name> <path-to-block-device>

with qemu-drive-attach script (device-name is arbitrary - I use "virtio8" for consistency with the Proxmox naming scheme)

!/usr/bin/expect
set vm_id               [lindex $argv 0];
set device_name         [lindex $argv 1];
set device_file         [lindex $argv 2];

spawn qm monitor $vm_id
expect "qm> "
send "drive_add 0 file=$device_file,if=none,id=drive-$device_name,cache=none,detect-zeroes=on\r"

expect "OK" {
    expect "qm> "
    send "device_add virtio-blk-pci,drive=drive-$device_name,id=$device_name\r"
}

expect "qm> "
send "quit\r"

send_user "\n"

And for detaching

qemu-drive-detach <vm-id> <device-name>

with qemu-drive-detach script

#!/usr/bin/expect
set vm_id               [lindex $argv 0];
set device_name         [lindex $argv 1];

spawn qm monitor $vm_id
expect "qm> "
send "device_del $device_name\r"

expect "qm> "
send "quit\r"

send_user "\n"

Error checking with these scripts leaves something to be desired, particularly as device_add and device_del are silent on success, but this will enable external block device drives to be hotswapped into the VM from a script.

| improve this answer | |
0

virtio-blk does not support hot swapping.

To use hot-swappable disks, you need to use a driver with such support, as SATA or virtio-scsi (which implement the SCSI protocol). Then you can use something as virsh (edit: as you correctly noted, it is not supported on Proxmox) or Qemu HMP (universally supported, but potentially more error prone) to hot add/remove disks.

| improve this answer | |
  • Thanks for the suggestion, but I'm afraid that doesn't get me home. qemu fails to start up if you try to attach a non-present scsi device, and only one of the disks will ever be in the system at the time. Nor will plugging a drive into the host cause it to automatically be attached to the guest. I think what's needed therefore is qm monitor commands for attaching and detaching virtio scsi devices and a way of running those commands from a script. Virsh is not available on proxmox, and I am not certain how it would interact if installed. – Christi May 21 at 17:58
  • I just checked and virsh is not compatible with proxmox because it does not use libvirt. – Christi May 21 at 18:25
  • virsh is only a means to interact with qemu, you can attach/detach disks to VM using qemu monitor/HMP. You can't boot with a missing disks; however, you can boot with the minimal disks set and attach/detach other disks on demand (if using a bus driver with hotplug support, as SATA or virtio-scsi). That said, if you want an out-of-the-box, automatic method to attach a specific host disk to a running VM, I think you are out of luck: while possible, you probably need to script it yourself. – shodanshok May 21 at 18:56
  • As I said in the question, it looks like it might be possible to use qm monitor commands or via a socket connected to qm-agent using JSON. I do not understand JSON or how qemu-agent parses JSON, so I'm hopong someone will be able to help out there. I tried converting the qm-monitor commands above to use virtio-scsi, but device_add then fails to recognise the drive= parameter, which is a bit odd. Still the commands above work as expected if I am able to automate them in some way. – Christi May 22 at 9:06
  • Please note that this answer is not correct in that virtio-blk does support hotswapping if you use the method described in the question. It may make more sense to use virtio-scsi in the same way, but I have not got that to work yet. – Christi May 28 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.