On Thu, Apr 23, 2020 at 05:15:03PM +0000, Andy Smith wrote:
Ubuntu 20.04 is scheduled to be released at some point
things stand its kernel won't boot at BitFolk.
I think it's due to some bugs in Xen's LZ4 decompression. These
problems have since been fixed but it would mean a forced reboot to
deploy them and I'm not going to do that, for this.
Some updates and some questions for customers about this.
Firstly, there WAS a bug in Xen's LZ4 compression but that was only
relevant for direct kernel load. That's when the guest's kernel
exists as a file on the host's filesystem and is referenced in the
config file for the guest.
That's not how we boot guests at BitFolk, so that's not relevant; it
isn't the source of the problem here.
There's a detailed description of the boot process of a BitFolk
The short version is: BitFolk has a grub binary that is compiled as
a Xen PV guest kernel. This grub binary acts as your kernel, so it's
the first thing a guest does when it's booted. It then looks for
grub configuration in your block devices and if it finds it then it
loads it. From your configuration it should find the correct actual
Linux kernel and initramfs and chain load to them at which point
booting proceeds normally.
I don't yet fully understand how or why grub as a Xen PV kernel
behaves differently to normal grub, but it seems like the Xen PV
part of grub requires support for every kind of kernel compression,
and no one has yet implemented LZ4 decompression for it.
The "not a Xen kernel" error message comes from grub.
My enquiries about this on the grub-devel mailing list have so far
met with silence:
At the moment it doesn't look like anyone is interested in adding
LZ4 kernel decompression to grub's Xen PV target. If you are capable
of doing so (or know someone who is) but would need financial
compensation to do so, please contact me directly; we may be able to
work something out.
Something that was pointed out to me is that when grub is compiled
as a PVH guest it does handle LZ4-compressed kernels. I do want to
move from Xen PV mode to Xen PVH mode but am not yet quite ready to
do so. It does provide some reassurance though that there is a
future where grub can continue to boot these things.
A current workaround is to decompress the kernel. This is described
Someone on the xen-users list pointed out that an apt hook can be
used to automatically decompress kernels at kernel upgrade time:
The basic concept seems good though I do think their supplied script
needs some security hardening (see my reply).
My question to you is, if I made the BitFolk Xen Shell installer for
Ubuntu 20.04 install such an apt hook so that it automatically
decompressed LZ4 kernels, would that be an acceptable state to
consider Ubuntu 20.04 officially supported at BitFolk?
What would you be thinking the first time you did a kernel upgrade
and saw apt call an extra script to decompress the kernel?
At the moment Ubuntu 20.04 is not listed on the BitFolk web site as
available, nor is there an installer for it in the Xen Shell,
because of the need to decompress its kernel. It is only possible to
install it by upgrade from an earlier release or by use of the
Rescue VM - and then you need to take care of the kernel yourself.
Put another way, is automating the decompression of LZ4 kernels at
kernel upgrade time a good enough solution to allow Ubuntu 20.04 to
be listed as available on the web site and in the Xen Shell?
If the consensus is "no" then I think Ubuntu 20.04 will have to
remain as not officially supported here until either there's a
pvgrub that supports LZ4 kernels or BitFolk introduces PVH mode.
-- No-nonsense VPS hosting