22 Jan
2020
22 Jan
'20
12:48 p.m.
I've just done a pretty standard update on my VPS, which is still on a
32 bit install running Ubuntu 18.04. Unfortunately for the first time I
can think of there was an issue with the kernel version. It upgraded to
the linux-image-5.3.0-26-generic package (I'm on the HWE path) and it
failed to reboot.
Not an issue for me as it was easy enough to use the console to boot off
the previous kernel, but I thought it worth noting in case anyone else
was impacted. Another good reason for me to get sorted with my upgrade
to 64 bit!
--
Paul Tansom | Aptanet Ltd. | https://www.aptanet.com/ | 023 9238 0001
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
22 Jan
22 Jan
1:10 p.m.
Hi Paul,
On Wed, Jan 22, 2020 at 12:48:44PM +0000, Paul Tansom wrote:
I've just done a pretty standard update on my VPS,
which is still on a
32 bit install running Ubuntu 18.04. Unfortunately for the first time I
can think of there was an issue with the kernel version. It upgraded to
the linux-image-5.3.0-26-generic package (I'm on the HWE path) and it
failed to reboot.
That is rather unfortunate. Do you happen to know if the initramfs
is LZ4 compressed? Ubuntu switched to LZ4 for these in 20.04 and I
am aware of issues there with Xen and some versions of grub.
Possibly the HWE kernel switched too?
Not an issue for me as it was easy enough to use the
console to boot off
the previous kernel, but I thought it worth noting in case anyone else
was impacted. Another good reason for me to get sorted with my upgrade
to 64 bit!
If it is the LZ4 thing then unfortunately nothing is going to help.
I think I actually have to roll out a new version of hypervisor
(which means reboots for everyone).
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
1:19 p.m.
On Wed, Jan 22, 2020 at 01:10:18PM +0000, Andy Smith wrote:
If it is the LZ4 thing then unfortunately nothing is
going to help.
I think I actually have to roll out a new version of hypervisor
(which means reboots for everyone).
… or you can make it go back to gzip:
$ sudo sed -i 's/lz4/gzip/' /etc/initramfs-tools/initramfs.conf
$ sudo update-initramfs -v -u -k all
(if that is the issue)
I shall experiment.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2:13 p.m.
Hi,
TL;DR: The HWE kernel on 64-bit Ubuntu 18.04 does seem to work and
does not use LZ4 compression for its initramfs - it's still gzip.
I've just installed Ubuntu 18.04 64-bit and then installed
linux-generic-hwe-18.04 which gets me:
$ ls -la /boot/*5.3*
-rw-r--r-- 1 root root 235811 Dec 18 15:01 /boot/config-5.3.0-26-generic
-rw-r--r-- 1 root root 65897765 Jan 22 13:39 /boot/initrd.img-5.3.0-26-generic
-rw------- 1 root root 4487587 Dec 18 15:01 /boot/System.map-5.3.0-26-generic
-rw------- 1 root root 9146616 Dec 18 15:20 /boot/vmlinuz-5.3.0-26-generic
This does boot 64-bit and I don't believe it's using LZ4 compression for
anything:
$ grep COMPRESS /etc/initramfs-tools/initramfs.conf
# COMPRESS: [ gzip | bzip2 | lzma | lzop | xz ]
COMPRESS=gzip
Though "file" can't seem to tell what it is:
$ file /boot/initrd.img-*
/boot/initrd.img-4.15.0-74-generic: ASCII cpio archive (SVR4 with no CRC)
/boot/initrd.img-5.3.0-26-generic: ASCII cpio archive (SVR4 with no CRC)
I think this is because the first bit of the initramfs is an
uncompressed cpio containing a CPU microcode update:
$ cpio -ivt < /boot/initrd.img-5.3.0-26-generic
drwxr-xr-x 3 root root 0 Nov 28 2018 .
drwxr-xr-x 3 root root 0 Nov 28 2018 kernel
drwxr-xr-x 3 root root 0 Nov 28 2018 kernel/x86
drwxr-xr-x 2 root root 0 Nov 28 2018 kernel/x86/microcode
-rw-r--r-- 1 root root 30546 Nov 28 2018
kernel/x86/microcode/AuthenticAMD.bin
62 blocks
This uncompressed cpio blob can be stripped by skipping 62 blocks as
noted above:
$ dd if=/boot/initrd.img-5.3.0-26-generic of=/tmp/initrd.img bs=512 skip=62
128644+1 records in
128644+1 records out
65866021 bytes (66 MB, 63 MiB) copied, 0.715398 s, 92.1 MB/s
But then:
$ file /tmp/initrd.img
/tmp/initrd.img: ASCII cpio archive (SVR4 with no CRC)
$ cpio -ivt < /tmp/initrd.img
drwxr-xr-x 2 root root 0 Oct 15 13:00 kernel
drwxr-xr-x 2 root root 0 Oct 15 13:00 kernel/x86
drwxr-xr-x 2 root root 0 Oct 15 13:00 kernel/x86/microcode
drwxr-xr-x 2 root root 0 Oct 15 13:00
kernel/x86/microcode/.enuineIntel.align.0123456789abc
-rw-r--r-- 1 root root 2960384 Oct 15 13:00
kernel/x86/microcode/GenuineIntel.bin
5784 blocks
So that's another uncompressed cpio blob. Looks like first is AMD
microcode and second is Intel microcode. Okay so try stripping both
(5784+62=5846 blocks):
$ dd if=/boot/initrd.img-5.3.0-26-generic of=/tmp/initrd.img bs=512 skip=5846
122860+1 records in
122860+1 records out
62904613 bytes (63 MB, 60 MiB) copied, 0.793158 s, 79.3 MB/s
$ file /tmp/initrd.img
/tmp/initrd.img: gzip compressed data, last modified: Wed Jan 22 13:39:21 2020, from Unix
Well alrighty then…
$ zcat /tmp/initrd.img | cpio -ivt | head
drwxr-xr-x 12 root root 0 Jan 22 13:39 .
-rwxr-xr-x 1 root root 6682 Oct 7 11:53 init
drwxr-xr-x 4 root root 0 Jan 22 13:39 usr
drwxr-xr-x 3 root root 0 Jan 22 13:39 usr/share
drwxr-xr-x 3 root root 0 Jan 22 13:39 usr/share/plymouth
drwxr-xr-x 4 root root 0 Jan 22 13:39 usr/share/plymouth/themes
drwxr-xr-x 2 root root 0 Jan 22 13:32
usr/share/plymouth/themes/ubuntu-text
-rw-r--r-- 1 root root 209 Apr 4 2019
usr/share/plymouth/themes/ubuntu-text/ubuntu-text.plymouth.in
-rw-r--r-- 1 root root 205 Jan 22 13:32
usr/share/plymouth/themes/ubuntu-text/ubuntu-text.plymouth
lrwxrwxrwx 1 root root 59 Jan 22 13:39
usr/share/plymouth/themes/text.plymouth ->
/usr/share/plymouth/themes/ubuntu-text/ubuntu-text.plymouth
"lsinitramfs" should do all this much simpler but I couldn't find a
way to have it tell me which decompression method it was using. If
you do know a simpler way of telling then please do let me know!
I will now reinstall that using 32-bit to verify that it does not
work.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2:42 p.m.
On Wed, Jan 22, 2020 at 02:13:45PM +0000, Andy Smith wrote:
I will now reinstall that using 32-bit to verify that
it does not
work.
Yeah so that kernel crashes during boot on 32-bit:
[ 0.747755] BUG: unable to handle page fault for address: fffffffc
[ 0.747780] #PF: supervisor write access in kernel mode
[ 0.747801] #PF: error_code(0x0003) - permiss
It may be worth reporting a bug on launchpad. I'm not going to do so
but will help provide information if anyone else does.
The environment is Xen PV mode; Paul experienced this on Xen
hypervisor version 4.10.1 and I replicated it on version 4.12.1.
Finding a way to build a recent upstream kernel release might also
be worthwhile in order to rule out a bug in the actual upstream
kernel.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
29 Jan
29 Jan
6:10 p.m.
** Andy Smith <andy(a)bitfolk.com> [2020-01-22 14:42]:
On Wed, Jan 22, 2020 at 02:13:45PM +0000, Andy Smith
wrote:
** end quote [Andy Smith]
Just waiting for a quite time to remember to reboot and test the kernel update
that has rolled out. It has fixed the nVidia issues on my old card on a 64 bit
desktop, but that doesn't mean anything when it comes to a 32 bit server!
--
Paul Tansom | Aptanet Ltd. | https://www.aptanet.com/ | 023 9238 0001
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
I will now reinstall that using 32-bit to verify
that it does not
work.
Yeah so that kernel crashes during boot on 32-bit:
[ 0.747755] BUG: unable to handle page fault for address: fffffffc
[ 0.747780] #PF: supervisor write access in kernel mode
[ 0.747801] #PF: error_code(0x0003) - permiss
It may be worth reporting a bug on launchpad. I'm not going to do so
but will help provide information if anyone else does.
The environment is Xen PV mode; Paul experienced this on Xen
hypervisor version 4.10.1 and I replicated it on version 4.12.1.
Finding a way to build a recent upstream kernel release might also
be worthwhile in order to rule out a bug in the actual upstream
kernel.
1550
days inactive
1557
days old
5 comments
2 participants
participants (2)
-
Andy Smith -
Paul Tansom