Creating VMs in separate ZFS filesystems

I’m running my VMs on a plain Ubuntu server with KVM/QEMU, with VMs stored on ZFS.

Up until now, I’ve created my VMs as QCOW2 files in a single ZFS filesystem; e.g. all VMs have lived in /ssdpool/vmimages. The vmimages directory wasn’t a child ZFS file system to ssdpool, but a plain directory created by VirtManager once upon a time.

There are a couple good reasons to deepen the structure, though:

First of all, it simplifies rollbacks from snapshots as each VM lives in its own ZFS filesystem.

Second, VM storage can be assigned its own ZFS properties that fit with the type of data stored in it - though honestly that’s less of an issue for me, as I generally just use QCOW2 images, as I mentioned.

Third, it makes it possible to purge unused disk images from my backups before the end of my backup retention, serving as an improvement to my storage utilization in cases where I’m sure I won’t need the data again.

So how do I go about changing this behavior, then?

Creating ZFS file systems

The first step is to create a new ZFS file system under my ssdpool. I set up the existing structure when I had less experience with ZFS, and the vmimages substructure is actually just a directory in ssdpool, rather than being its own file system, as is the way I would set it up today.

I will create a new pool, ssdpool/vdisks. Then I will create separate ZFS file systems in this structure for each VM and migrate them into their new home.

As described by Jim Salter in a Klara Systems article on ZFS recordsize tuning, the QCOW2 images I use for my virtual machines should work best with a recordsize parameter that coincides with the native block size of the QCOW2 image; 64KB.

sudo zfs create ssdpool/vdisks -o recordsize=64K

We can verify that it went well:

$ sudo zfs get recordsize ssdpool/vdisks
NAME            PROPERTY    VALUE    SOURCE
ssdpool/vdisks  recordsize  64K      local
$

At this point it is prudent to set the correct permissions on the new file system for KVM:

sudo chown -R libvirt-qemu:kvm /ssdpool/vdisks

Child file systems will, by default, inherit ZFS properties from their parents, so to create a filesystem for one of my virtual machines shouldn’t require me to explicitly define the record size. Let’s test that hypothesis:

sudo zfs create ssdpool/vdisks/testsrv1
sudo zfs get recordsize ssdpool/vdisks/testsrv1
NAME                        PROPERTY    VALUE    SOURCE
ssdpool/vdisks/testsrv1     recordsize  64K      inherited from ssdpool/vdisks
$ 

Perfect. this simplifies our future VM creation.

Migrating KVM virtual machines

Changing the storage path for a VM seems to require the following steps:

• Shut down the VM
• Move the VM files to their new location
• Edit the VM to find its files in their new location
• Start the VM back up

Let’s do this.

sudo virsh shutdown testsrv1

Then we wait until the machine has shut down:

$ sudo virsh list --all
 Id   Name                      State
------------------------------------------
 -    testsrv1                  shut off

We move the VM virtual disk(s) to its new home, and ensure the permissions are correct:

sudo mv testsrv1.qcow2 /ssdpool/vdisks/testsrv1/
sudo chown libvirt-qemu:kvm /ssdpool/vdisks/testsrv1/testsrv1.qcow2 

We edit the virtual machine to update its disk location:

virsh edit testsrv1

In the file, we locate the disk block(s) and update the path found in the source file XML block, and then save and exit the editor.

Finally we start the VM again, and hopefully we haven’t messed anything up:

virsh start testsrv1

Of course migrating the VM is only the most visible step of this change. We’re not done until we have confirmed our backups. Let’s revisit that in the next post.