BitFolk Users December 2021

users@mailman.bitfolk.com

3 participants
5 discussions

Remaining 32-bit PV guests will be switched to pvshim on Tuesday 18 January

by Andy Smith

Hi, TL;DR: The ~30% of you still running 32-bit PV guests are going to have your config changed in a month. We've tested that on many different configurations and haven't had a problem yet but it's always possible something could go wrong, and if so you'll only find out at the next boot. If affected we recommend you instead make the change yourself at a time convenient to you. This email is only relevant to you if you're still running in 32-bit PV mode. Most customers run 64-bit. If you type "uname -m" in your VM then it will say "amd64" for 64-bit and "i686" for 32-bit. It also says it on the summary page of: https://panel.bitfolk.com/account/ You can stop reading if you're already running as 64-bit, or in PVH mode. We haven't got a simple way to check if you are PVH mode because the intention is that eventually will be a detail you don't have to care about (all VMs will be PVH and that has been the default for over a year now). You can for now log in to the Xen Shell and type "virtmode" and it will tell you. So if that says "PVH" you can also stop reading. For several years now we have been trying to encourage customers running 32-bit PV mode guests to switch to 64-bit and / or PVH mode. There are many reasons for this but the most pressing one is that it's not possible to fully protect 32-bit PV guests against the various already known speculation attacks (nor probably new ones that will be discovered). About 30% of the customer base still runs 32-bit PV mode guests even though the default has been 64-bit since about 2012. We are clearly not going to be able to force everyone to switch in a timely manner so we have been testing a different way of running legacy 32-bit PV mode guests. That testing has gone well - there haven't been any issues - so we're going to convert all remaining 32-bit PV mode guests to that configuration on Tuesday 18 January 2022. Since it's not possible to test every permutation of installed guest though, we can't rule out there being a problem, and that problem will only manifest at your next boot. If you'd like to make the config change ahead of time here is how: 1. Log in to your Xen Shell. More info: https://tools.bitfolk.com/wiki/Xen_Shell 2. Make sure the version in the "help" command is at least this: xen-shell> help xen-shell v1.48bitfolk66 The Xen Shell stays running after you disconnect so it is possible to be running an older version. If it is older, "exit" out of every window until it logs you out, then log in again. 3. Use the "arch" and "virtmode" commands to confirm that you are currently running in 32-bit PV mode: xen-shell> arch Your current install architecture is: i686 xen-shell> virtmode Your current virtualisation mode is: PV 4. Use the "arch i686" command to force a switch to i686 (32-bit) architecture again. This will update your config to use pvshim. 5. Use the "shutdown" command to shut your guest down. 6. Use the "boot" command to boot it again. It should boot pretty much the same as before. If it does not, then you will likely not be able to get it to boot again yourself and will need to put in a support ticket. This change will be made for all remaining 32-bit PV mode guests on Tuesday 18 January, without further testing, as that would involve forcible reboot. If you do want to take some action about this here are some things you could do, in order of best choices to worst choices: a) Ask for a new "migration VPS" which would be an empty account that you can do a new install into (which would be 64-bit PVH as that's the default): https://tools.bitfolk.com/wiki/Migrating_to_a_new_VPS b) Upgrade your kernel past 4.19.0 and make sure you're running grub-pc (not legacy Grub) as bootloader, with a /boot/grub/grub.cfg file, then switch to PVH mode. c) If running at least Debian 7 (wheezy) or comparable age Ubuntu you can install an amd64 (64-bit) kernel even while everything else is 32-bit. That turns your VM into a 64-bit PV guest. Follow these CrossGrading instructions only as far as installing and booting into the new kernel: https://wiki.debian.org/CrossGrading d) Do nothing and let us switch you to using pvshim. Your guest is still insecure and running with reduced performance compared to 64-bit but this only then affects you. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

3 years, 1 month

A reminder for those upgrading 32-bit Debian guests

by Andy Smith

Hi, A reminder that if you have a 32-bit Debian guest that you are keeping up to date: The Linux kernel removed support for 32-bit PV guests at version 5.9, so it will not be possible for you to upgrade from Debian 10 (buster) to 11 (bullseye) without taking action. This was mentioned before, but since then there have been a few casualties anyway (seemingly-unbootable guests). https://lists.bitfolk.com/lurker/message/20210930.104643.2ab5f9c0.en.html As you can see, as long as you are running a kernel above 4.19.0 and are using grub-pc to boot, you can just switch to PVH mode. This isn't an issue for Ubuntu because no 32-bit support there at all for some time, nor for CentOS because no upgrades between major releases there either. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

3 years, 1 month

PowerDNS Secondary DNS for ACME Let's Encrypt Wildcard certificates

by William Wright

Hello, I am currently using a PowerDNS Authoritative server in my Bitfolk VPS alongside Bitfolk's secondary DNS servers. At home, I also have a self hosted NAS setup which involves the use of Traefik alongside docker containers. I have been trying to generate a wildcard Let's Encrypt certificate using ACME via the Traefik container, authenticating via RFC2136. However, while ACME was successfully able insert a TXT record into the zone, it hasn't updated the Secondary DNS and reports back with the following error: > unable to generate a certificate for the domains [m6wiq.uk *.m6wiq.uk]: error: one or more domains had a problem:\n[*.m6wiq.uk] time limit exceeded: last error: NS a.authns.bitfolk.co.uk. did not return the expected TXT record [fqdn: _acme-challenge.m6wiq.uk., value: Through my research on PowerDNS, I have ensured that SOA-EDIT-DNSUPDATE is set to 'INCREASE' and that FORWARD-DNSUPDATE and NOTIFY-DNSUPDATE are enabled. Is there anything else that I need to configure on PowerDNS to ensure RFC2136 updates inform the secondary DNS servers? Best Regards, William -- William Wright Callsign: M6WIQ Mail: william(a)m6wiq.uk

3 years, 2 months

Volunteers sought for running 32-bit guests under pvshim

by Andy Smith

Hi, As you may recall we've been reminding you over the years that 32-bit guests need to be made a thing of the past for a variety of reasons - primarily security. Unfortunately, 30% of customers are still running a 32-bit PV mode guest, so there is no prospect of swiftly getting people to upgrade / reinstall¹. Luckily there is another mode called "pvshim", which runs a 64-bit PVH guest that chain loads the unmodified 32-bit PV guest. In our limited testing that works without any change on the guest administrator's side. So, if you're currently running a 32-bit PV mode VM please would you consider volunteering to let us test booting it under pvshim? If so, please reply to me off-list letting me know what sort of time of day would be acceptable to try it out. It would involve us shutting down your VM and then booting it again. It would be best if we did it, not you, as if there were a problem we'd just revert the configuration and boot your VM again. Once we're comfortable that it works on a wide variety of guests then we'd move all 32-bit PV guests to that setup which they would pick up at their next boot. If you don't actually know whether you're running 32-bit… - Typing "uname -m" will say "i686" for 32-bit and "x86_64" for 64-bit. - It also says it on the summary page of https://panel.bitfolk.com/account/ Cheers, Andy ¹ This pvshim thing is a last ditch option for people who can't upgrade their OS for whatever reason. It would in all cases be better to run an up to date version of Linux instead. The Linux kernel removed 32-bit Xen PV support at release 5.9.0, but anything newer than 4.19 should support being run in PVH mode, so that's what you'd do there. A 64-bit install running in PVH mode would be best. The default for a new install at BitFolk was switched to 64-bit PV in April 2013. It was switched to 64-bit PVH in November 2020. -- https://bitfolk.com/ -- No-nonsense VPS hosting

3 years, 2 months

EPEL kernel-ml boot problems - no block devices

by Andy Smith

Hi, Some time in the last 6 months the kernel-ml kernel and associated packages from EPEL (used to work under Xen on CentOS 7, 8 and later and other RHEL-like) seems to not include the module xen-blkfront in the initramfs that it creates. As a result you don't see any block devices at boot. I don't know if this is intentional. I don't exactly know how it's meant to work. I *think* that dracut is supposed to be able to work out what drivers are required for your root filesystem and include those without you having to do anything special. Whatever the case it is not doing it right now. It's easy to fix though. # lsinitrd /boot/initramfs-5.15.6-1.el7.elrepo.x86_64.img | grep xen- xen-netfront # # note: no xen-blkfront # cat > /etc/dracut.conf.d/xen.conf <<End-of-script add_drivers+=" xen-blkfront " End-of-script # dracut -f --kver "5.15.6-1.el7.elrepo.x86_64" /boot/initramfs-5.15.6-1.el7.elrepo.x86_64.img # lsinitrd /boot/initramfs-5.15.6-1.el7.elrepo.x86_64.img | grep xen- xen-netfront xen-blkfront If your VM is currently unbootable you'll need to do this from the rescue VM: user@rescue:~$ sudo -s root@rescue:/home/user# mount /dev/xvda1 /mnt root@rescue:/home/user# cd /mnt root@rescue:/mnt# mount -t proc /proc proc/ root@rescue:/mnt# mount --rbind /sys sys/ root@rescue:/mnt# mount --rbind /dev dev/ root@rescue:/mnt# chroot /mnt /bin/bash [root@rescue /]# # This is your CentOS install, proceed as above Our installer for CentOS 8 takes care of the above for you right now, but if you have a pre-existing CentOS 7 or 8 (or whatever) system using kernel-ml from EPEL then you may want to confirm that your initramfs still has xen-blkfront inside of it otherwise you will get a nasty surprise at next boot. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

3 years, 2 months

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

BitFolk Users December 2021