Hi,
TL;DR: The ~30% of you still running 32-bit PV guests are going to
have your config changed in a month. We've tested that on many
different configurations and haven't had a problem yet but it's
always possible something could go wrong, and if so you'll only find
out at the next boot. If affected we recommend you instead make the
change yourself at a time convenient to you.
This email is only relevant to you if you're still running in 32-bit
PV mode. Most customers run 64-bit. If you type "uname -m" in your
VM then it will say "amd64" for 64-bit and "i686" for 32-bit. It
also says it on the summary page of:
https://panel.bitfolk.com/account/
You can stop reading if you're already running as 64-bit, or in PVH
mode.
We haven't got a simple way to check if you are PVH mode because the
intention is that eventually will be a detail you don't have to care
about (all VMs will be PVH and that has been the default for over a
year now). You can for now log in to the Xen Shell and type
"virtmode" and it will tell you. So if that says "PVH" you can also
stop reading.
For several years now we have been trying to encourage customers
running 32-bit PV mode guests to switch to 64-bit and / or PVH mode.
There are many reasons for this but the most pressing one is that
it's not possible to fully protect 32-bit PV guests against the
various already known speculation attacks (nor probably new ones
that will be discovered).
About 30% of the customer base still runs 32-bit PV mode guests even
though the default has been 64-bit since about 2012. We are clearly
not going to be able to force everyone to switch in a timely manner
so we have been testing a different way of running legacy 32-bit PV
mode guests.
That testing has gone well - there haven't been any issues - so
we're going to convert all remaining 32-bit PV mode guests to that
configuration on Tuesday 18 January 2022.
Since it's not possible to test every permutation of installed guest
though, we can't rule out there being a problem, and that problem
will only manifest at your next boot.
If you'd like to make the config change ahead of time here is how:
1. Log in to your Xen Shell.
More info: https://tools.bitfolk.com/wiki/Xen_Shell
2. Make sure the version in the "help" command is at least this:
xen-shell> help
xen-shell v1.48bitfolk66
The Xen Shell stays running after you disconnect so it is
possible to be running an older version. If it is older, "exit"
out of every window until it logs you out, then log in again.
3. Use the "arch" and "virtmode" commands to confirm that you are
currently running in 32-bit PV mode:
xen-shell> arch
Your current install architecture is: i686
xen-shell> virtmode
Your current virtualisation mode is: PV
4. Use the "arch i686" command to force a switch to i686 (32-bit)
architecture again. This will update your config to use pvshim.
5. Use the "shutdown" command to shut your guest down.
6. Use the "boot" command to boot it again.
It should boot pretty much the same as before. If it does not, then
you will likely not be able to get it to boot again yourself and
will need to put in a support ticket.
This change will be made for all remaining 32-bit PV mode guests on
Tuesday 18 January, without further testing, as that would involve
forcible reboot.
If you do want to take some action about this here are some things
you could do, in order of best choices to worst choices:
a) Ask for a new "migration VPS" which would be an empty account
that you can do a new install into (which would be 64-bit PVH as
that's the default):
https://tools.bitfolk.com/wiki/Migrating_to_a_new_VPS
b) Upgrade your kernel past 4.19.0 and make sure you're running
grub-pc (not legacy Grub) as bootloader, with a
/boot/grub/grub.cfg file, then switch to PVH mode.
c) If running at least Debian 7 (wheezy) or comparable age Ubuntu
you can install an amd64 (64-bit) kernel even while everything
else is 32-bit. That turns your VM into a 64-bit PV guest. Follow
these CrossGrading instructions only as far as installing and
booting into the new kernel:
https://wiki.debian.org/CrossGrading
d) Do nothing and let us switch you to using pvshim. Your guest is
still insecure and running with reduced performance compared to
64-bit but this only then affects you.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
Hi,
A reminder that if you have a 32-bit Debian guest that you are
keeping up to date:
The Linux kernel removed support for 32-bit PV guests at version
5.9, so it will not be possible for you to upgrade from Debian
10 (buster) to 11 (bullseye) without taking action.
This was mentioned before, but since then there have been a few
casualties anyway (seemingly-unbootable guests).
https://lists.bitfolk.com/lurker/message/20210930.104643.2ab5f9c0.en.html
As you can see, as long as you are running a kernel above 4.19.0 and
are using grub-pc to boot, you can just switch to PVH mode.
This isn't an issue for Ubuntu because no 32-bit support there at
all for some time, nor for CentOS because no upgrades between major
releases there either.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
Hello,
I am currently using a PowerDNS Authoritative server in my Bitfolk VPS
alongside Bitfolk's secondary DNS servers.
At home, I also have a self hosted NAS setup which involves the use of
Traefik alongside docker containers.
I have been trying to generate a wildcard Let's Encrypt certificate
using ACME via the Traefik container, authenticating via RFC2136.
However, while ACME was successfully able insert a TXT record into the
zone, it hasn't updated the Secondary DNS and reports back with the
following error:
> unable to generate a certificate for the domains [m6wiq.uk *.m6wiq.uk]: error: one or more domains had a problem:\n[*.m6wiq.uk] time limit exceeded: last error: NS a.authns.bitfolk.co.uk. did not return the expected TXT record [fqdn: _acme-challenge.m6wiq.uk., value:
Through my research on PowerDNS, I have ensured that SOA-EDIT-DNSUPDATE
is set to 'INCREASE' and that FORWARD-DNSUPDATE and NOTIFY-DNSUPDATE are
enabled. Is there anything else that I need to configure on PowerDNS to
ensure RFC2136 updates inform the secondary DNS servers?
Best Regards,
William
--
William Wright
Callsign: M6WIQ
Mail: william(a)m6wiq.uk
Hi,
As you may recall we've been reminding you over the years that
32-bit guests need to be made a thing of the past for a variety of
reasons - primarily security.
Unfortunately, 30% of customers are still running a 32-bit PV mode
guest, so there is no prospect of swiftly getting people to upgrade
/ reinstall¹.
Luckily there is another mode called "pvshim", which runs a 64-bit
PVH guest that chain loads the unmodified 32-bit PV guest. In our
limited testing that works without any change on the guest
administrator's side.
So, if you're currently running a 32-bit PV mode VM please would you
consider volunteering to let us test booting it under pvshim? If so,
please reply to me off-list letting me know what sort of time of day
would be acceptable to try it out.
It would involve us shutting down your VM and then booting it again.
It would be best if we did it, not you, as if there were a problem
we'd just revert the configuration and boot your VM again.
Once we're comfortable that it works on a wide variety of guests
then we'd move all 32-bit PV guests to that setup which they would
pick up at their next boot.
If you don't actually know whether you're running 32-bit…
- Typing "uname -m" will say "i686" for 32-bit and "x86_64" for
64-bit.
- It also says it on the summary page of
https://panel.bitfolk.com/account/
Cheers,
Andy
¹ This pvshim thing is a last ditch option for people who can't
upgrade their OS for whatever reason. It would in all cases be
better to run an up to date version of Linux instead.
The Linux kernel removed 32-bit Xen PV support at release 5.9.0,
but anything newer than 4.19 should support being run in PVH mode,
so that's what you'd do there.
A 64-bit install running in PVH mode would be best. The default
for a new install at BitFolk was switched to 64-bit PV in April
2013. It was switched to 64-bit PVH in November 2020.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Some time in the last 6 months the kernel-ml kernel and associated packages from
EPEL (used to work under Xen on CentOS 7, 8 and later and other
RHEL-like) seems to not include the module xen-blkfront in the
initramfs that it creates. As a result you don't see any block
devices at boot.
I don't know if this is intentional. I don't exactly know how it's
meant to work. I *think* that dracut is supposed to be able to work
out what drivers are required for your root filesystem and include
those without you having to do anything special.
Whatever the case it is not doing it right now. It's easy to fix
though.
# lsinitrd /boot/initramfs-5.15.6-1.el7.elrepo.x86_64.img | grep xen-
xen-netfront
# # note: no xen-blkfront
# cat > /etc/dracut.conf.d/xen.conf <<End-of-script
add_drivers+=" xen-blkfront "
End-of-script
# dracut -f --kver "5.15.6-1.el7.elrepo.x86_64" /boot/initramfs-5.15.6-1.el7.elrepo.x86_64.img
# lsinitrd /boot/initramfs-5.15.6-1.el7.elrepo.x86_64.img | grep xen-
xen-netfront
xen-blkfront
If your VM is currently unbootable you'll need to do this from the
rescue VM:
user@rescue:~$ sudo -s
root@rescue:/home/user# mount /dev/xvda1 /mnt
root@rescue:/home/user# cd /mnt
root@rescue:/mnt# mount -t proc /proc proc/
root@rescue:/mnt# mount --rbind /sys sys/
root@rescue:/mnt# mount --rbind /dev dev/
root@rescue:/mnt# chroot /mnt /bin/bash
[root@rescue /]# # This is your CentOS install, proceed as above
Our installer for CentOS 8 takes care of the above for you right
now, but if you have a pre-existing CentOS 7 or 8 (or whatever)
system using kernel-ml from EPEL then you may want to confirm that
your initramfs still has xen-blkfront inside of it otherwise you
will get a nasty surprise at next boot.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
