BitFolk Announcements

announce@mailman.bitfolk.com

259 discussions

Ubuntu 22.04 (Jammy Jellyfish) now available for new installs and self-installs

by Andy Smith

Hi, Ubuntu 22.04 (Jammy Jellyfish) is now available for new installs and self-installs using our Xen Shell: https://tools.bitfolk.com/wiki/Using_the_self-serve_net_installer In a change to previous releases, Ubuntu has switched away from the Debian installer and recommends that the official Ubuntu Cloud Image is used for installs on public clouds. So, when you kick off an install of Ubuntu 22.04 it will: - ask you for your chosen hostname and password - overwrite your disk with the image for Ubuntu 22.04 - boot it and customise it automatically using the hostname and password you provided and information held in your BitFolk account. For more information please see the article about Ubuntu at BitFolk: https://tools.bitfolk.com/wiki/Ubuntu#22.04_.28Jammy_Jellyfish.29_and_beyond Upgrades from 20.04 to 22.04 can be done as usual using "do-release-upgrade", although do note that it's normal Ubuntu policy to not offer an upgrade until the first point release of the LTS, so at the moment you would need to force that with "do-release-upgrade -d". We are not aware of any particular issues with upgrading to or installing Ubuntu 22.04 at BitFolk. As always you should read Ubuntu's own release notes for general things to be aware of. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years

A reminder that CentOS 8 is now EOL and they are not kidding about that

by Andy Smith

Hi, For those using CentOS 8, a reminder that it went EOL on 31 December 2021 and as of 31 January 2022 they removed all the public mirrors as well, so you can't get updates or install anything any more. This apparently took a number of projects by surprise… including some Red Hat ones! https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/Y4HU5BQHZNGVLXLUUA… If you are using CentOS 8 I think we would recommend that you switch it to CentOS Stream 8 at least in the short term, because that is the only way you get security updates. I understand that there is a simple script to do this without reinstall and at least 1 BitFolk customer has done it. In terms of BitFolk supporting CentOS Stream 9, we're waiting for ELRepo to gain an EL9 kernel-ml package, then that should be viable. I asked you all about your plans post-CentOS 8 and didn't get a huge response. Some interest was expressed in Rocky Linux and Oracle Linux. So, those will likely be available soon. A customer also put together instructions for installing OpenSUSE Leap, which I have run through and seems to work fine: https://tools.bitfolk.com/wiki/User:Moggers87/Installing_Opensuse Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 3 months

Remaining 32-bit PV guests will be switched to pvshim on Tuesday 18 January

by Andy Smith

Hi, TL;DR: The ~30% of you still running 32-bit PV guests are going to have your config changed in a month. We've tested that on many different configurations and haven't had a problem yet but it's always possible something could go wrong, and if so you'll only find out at the next boot. If affected we recommend you instead make the change yourself at a time convenient to you. This email is only relevant to you if you're still running in 32-bit PV mode. Most customers run 64-bit. If you type "uname -m" in your VM then it will say "amd64" for 64-bit and "i686" for 32-bit. It also says it on the summary page of: https://panel.bitfolk.com/account/ You can stop reading if you're already running as 64-bit, or in PVH mode. We haven't got a simple way to check if you are PVH mode because the intention is that eventually will be a detail you don't have to care about (all VMs will be PVH and that has been the default for over a year now). You can for now log in to the Xen Shell and type "virtmode" and it will tell you. So if that says "PVH" you can also stop reading. For several years now we have been trying to encourage customers running 32-bit PV mode guests to switch to 64-bit and / or PVH mode. There are many reasons for this but the most pressing one is that it's not possible to fully protect 32-bit PV guests against the various already known speculation attacks (nor probably new ones that will be discovered). About 30% of the customer base still runs 32-bit PV mode guests even though the default has been 64-bit since about 2012. We are clearly not going to be able to force everyone to switch in a timely manner so we have been testing a different way of running legacy 32-bit PV mode guests. That testing has gone well - there haven't been any issues - so we're going to convert all remaining 32-bit PV mode guests to that configuration on Tuesday 18 January 2022. Since it's not possible to test every permutation of installed guest though, we can't rule out there being a problem, and that problem will only manifest at your next boot. If you'd like to make the config change ahead of time here is how: 1. Log in to your Xen Shell. More info: https://tools.bitfolk.com/wiki/Xen_Shell 2. Make sure the version in the "help" command is at least this: xen-shell> help xen-shell v1.48bitfolk66 The Xen Shell stays running after you disconnect so it is possible to be running an older version. If it is older, "exit" out of every window until it logs you out, then log in again. 3. Use the "arch" and "virtmode" commands to confirm that you are currently running in 32-bit PV mode: xen-shell> arch Your current install architecture is: i686 xen-shell> virtmode Your current virtualisation mode is: PV 4. Use the "arch i686" command to force a switch to i686 (32-bit) architecture again. This will update your config to use pvshim. 5. Use the "shutdown" command to shut your guest down. 6. Use the "boot" command to boot it again. It should boot pretty much the same as before. If it does not, then you will likely not be able to get it to boot again yourself and will need to put in a support ticket. This change will be made for all remaining 32-bit PV mode guests on Tuesday 18 January, without further testing, as that would involve forcible reboot. If you do want to take some action about this here are some things you could do, in order of best choices to worst choices: a) Ask for a new "migration VPS" which would be an empty account that you can do a new install into (which would be 64-bit PVH as that's the default): https://tools.bitfolk.com/wiki/Migrating_to_a_new_VPS b) Upgrade your kernel past 4.19.0 and make sure you're running grub-pc (not legacy Grub) as bootloader, with a /boot/grub/grub.cfg file, then switch to PVH mode. c) If running at least Debian 7 (wheezy) or comparable age Ubuntu you can install an amd64 (64-bit) kernel even while everything else is 32-bit. That turns your VM into a 64-bit PV guest. Follow these CrossGrading instructions only as far as installing and booting into the new kernel: https://wiki.debian.org/CrossGrading d) Do nothing and let us switch you to using pvshim. Your guest is still insecure and running with reduced performance compared to 64-bit but this only then affects you. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 3 months

Automated card payments now available

by Andy Smith

Hello, There is now support for saving payment card details for future automated charging. To save card details please see: https://panel.bitfolk.com/account/billing/ If you already have automated payments set up through Direct Debit or PayPal and want to switch to card payment, you will need to cancel that arrangement first, which you can also do from the above page. We have accepted one-off card payments through Stripe since October 2013, and it has been a regular request to enable the saving of card details so that they don't have to be input each time there is an invoice. This has finally now been done. This is a minimally working implementation. I expect once it sees greater use there will be some suggestions for improvement. Please make these on our feature tracker at: https://tools.bitfolk.com/redmine/issues/202 One-off card payments will remain available for those who prefer to pay that way. More about supported payment methods: https://tools.bitfolk.com/wiki/Payment_methods Thanks, Andy BitFolk Ltd -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 4 months

A reminder for those upgrading 32-bit Debian guests

by Andy Smith

Hi, A reminder that if you have a 32-bit Debian guest that you are keeping up to date: The Linux kernel removed support for 32-bit PV guests at version 5.9, so it will not be possible for you to upgrade from Debian 10 (buster) to 11 (bullseye) without taking action. This was mentioned before, but since then there have been a few casualties anyway (seemingly-unbootable guests). https://lists.bitfolk.com/lurker/message/20210930.104643.2ab5f9c0.en.html As you can see, as long as you are running a kernel above 4.19.0 and are using grub-pc to boot, you can just switch to PVH mode. This isn't an issue for Ubuntu because no 32-bit support there at all for some time, nor for CentOS because no upgrades between major releases there either. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 4 months

We are again starting up our efforts to move customers from servers that need updates

by Andy Smith

Hi, You may recall that all through the first half of 2021 we've been moving customer services off of certain servers in order to upgrade the servers and put them back in service. That effort ground to a halt in June because of other more pressing concerns. We're now starting that up again to finish the job. We sent notification emails to everyone who would be affected, but this was back in June so you may have forgotten. These went to customers on servers "hen" and "paradox", which are the last two servers that need upgrade. That notification email asked you to let us know if you need more than 5 minutes of notice for the work to be done. If you did reply to that, don't worry, we still have records of that and will give you the amount of notice you asked for. If you didn't reply then we are still assuming that 5 minutes of notice at any time of day is fine and that's how we'll be proceeding over the next couple of weeks. If that situation has changed then you should look for the original notification email and reply to it with your needs. The last batch of notification emails were sent out to customers on "hen" and "paradox" on Saturday 5 June 2021 with subject line: We need to move your BitFolk VPS '$accountname' to other hardware If you can't find it but still need to let us know, just email support(a)bitfolk.com to open a support ticket. Again, this only affects customers on servers "hen" and "paradox". Here's how to work out which server your service is on: https://bitfolk.com/customer_information.html#toc_3_Which_piece_of_actual_h… Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 7 months

Linux ended 32-bit PV guest support with v5.9, which means no Debian 11 (bullseye)

by Andy Smith

Hi, Yesterday evening and again this morning we've had two customers try to upgrade their 32-bit Debian 10 VMs to Debian 11 and end up with something that doesn't boot. This is because the Linux kernel stopped supporting 32-bit Xen PV domains at version 5.9. The quick workaround for those on Debian 10: xen shell> virtmode pvh xen shell> boot We have talked a lot about this over on the "users" list over the years, and for a while now the default at BitFolk has been 64-bit, PVH mode guests, but we can't switch existing customers over to PVH mode because it requires at least kernel version 3.19 and we don't know what kernels you're running. So existing customers have been left to switch on their own. Switching to PVH mode will for now allow you to continue to run 32-bit VMs. However, aside from this, 32-bit Linux has been in decline for some time and it's know to be less performant and less secure than 64-bit. So the time has already passed where you should be planning your switch to 64-bit. == Just switching your kernel == Most of the advantage is to be gained by just switching the kernel, so those running Debian could do that as Debian has good support for this. 1. Upgrade to Debian 10 (buster) 2. Follow these instructions only up to and including the "Install a kernel that supports both architectures in userland" step. 3. Connect to your Xen Shell 4. Shut down, boot, select the new amd64 kernel xen shell> shutdown xen shell> boot If for any reason this does not work, just boot again and select your previous i686 kernel again. We suggest doing this in the Xen Shell so you can interact with the boot process because the new amd64 kernel may not be listed first in your bootloader. 5. Once satisfied that your amd64 kernel works you can remove the i686 kernel packages. Debian will take care of providing you with amd64 kernel updates in future. If you haven't already done so you should consider switching to PVH mode now as well. We do not recommend trying to fully cross-grade your operating system to 64-bit unless you are an expert. == Reinstall == You can do a reinstall in place yourself: https://tools.bitfolk.com/wiki/Using_the_self-serve_net_installer Don't forget to first switch your architecture to 64-bit and your virtmode to PVH: xen shelll> arch x86_64 xen shell> virtmode pvh as these are the modern defaults. We can also offer a new account free for two weeks for you to install into and move your things over. https://tools.bitfolk.com/wiki/Migrating_to_a_new_VPS == PVH mode? == Which virtualisation mode you use is rather something we don't expect customers to have to worry too much about, so new customers have been in PVH mode for some time and haven't had to think about it, but existing customers will need to make the change at some point. Anyone with a kernel that's 4.19 or newer should be able to switch to it. Here's more info: https://tools.bitfolk.com/wiki/PVH Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 7 months

All customers have been opted back out of suspend+restore

by Andy Smith

Hi, TL;DR: We turned off suspend+restore for everyone. We think it is okay for you to re-enable it as long as you use kernel 4.2 or newer (released 6 years ago), but can't tell what kernel you're running so erred on the side of caution. We continue to use it for our own VMs. More detail: We've just opted you all out of suspend+restore because of the filesystem corruption that afflicted 2 customer VMs during the maintenance in August. There were 83 customer VMs that previously had opted in. While investigating that we did of course not do any suspend+restore anyway. I am now satisfied that we know why it happened and under what circumstances it should be safe to use it again, but as a precaution we have opted everyone out of it so you can make your own decisions. A direct email has gone out to the main contact for each VM that had previously opted in to this. That email contains far more detail. If you think you had opted in to suspend+restore but don't see that email please check your spam folders etc (and then mark it as "not spam" if necessary!). You can see the current setting (or opt back in) here: https://panel.bitfolk.com/account/config/#prefs You can read more about suspend+restore here: https://tools.bitfolk.com/wiki/Suspend_and_restore Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 7 months

2021-09-06 ~0302Z: Unscheduled power cycle of host "macallan"

by Andy Smith

Hi, We began receiving alerts at approximately 03:02Z today that host "macallan" was unresponsive. There was nothing interesting on its serial console. Its console also did not respond. The out of band access to the BMC worked but didn't show anything unusual. There were no hardware events logged. In the face of a hard lock up all I could do was power cycle it. All customer VMs were booted again by about 03:30Z. I'll be keeping a close eye on this server. If this repeats then we may have to move customers off of it at speed and with little notice. Apologies for the disruption this has caused. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 8 months

Reboots will be necessary to address security issues, probably early hours 24+25 August 2021

by Andy Smith

Hello, Unfortunately some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 25 August 2021. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on Tuesday 24 and Wednesday 25 August. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 45 minutes per bare metal host. We are going to take the opportunity to complete upgrading the kernel and hypervisor on some of the hosts that haven't had that done yet, which is why the work may take a few minutes more for some hosts. There are two hosts left that we are trying to migrate customers off of ("hen" and "paradox"). That was supposed to be done by now but that effort has been hampered by the other issues we've been having and is dragging on. We don't intend to patch or reboot those two hosts, instead mitigating issues with configuration and renewing efforts to clear customers off of them. If you are concerned about that we will be happy to move your service as a priority. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We will be able to migrate² you to already-patched hardware before the regular maintenance starts, at a time of your choosing. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. Thanks, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 8 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BitFolk Announcements