Meltdown/Spectre-related security reboots coming on 21/22/23 February

Hi, Notifications have just been sent out letting you know the hour long maintenance window during which the host that your VPS is on will be rebooted for security patches. If you have not received the notification please check your spam folders etc. and if still having no luck please contact support(a)bitfolk.com. Apologies for the short notice of this, but now that I feel I have a reasonable plan covering a large part of the problem space, it is best to get this done as soon as possible. I've deliberately left most of the technical detail out of the reboot notification. The technical details are overwhelming. If you aren't a particularly technical person then my advice would be: - Make sure you are running a security-supported release of your chosen Linux distribution and that you keep it up to date. Between them and us you should eventually get to safety, just bear in mind that this is an evolving situation and not many Linux vendors are willing to push out the latest fixes without extensive testing. For those who really want them, here are some more technical details. The newly-deployed hypervisor will: - support Page Table Isolation, similar to the Linux kernel's KPTI, to protect against Meltdown. This feature will protect BitFolk's hypervisor from Meltdown attacks from all customers. At the moment all BitFolk VPSes are paravirtual (PV) guests. For 64-bit VPSes, this Xen-level PTI also protects against Meltdown attacks from within their own kernel or user space. Thus, although your kernel will report that KPTI is disabled, you will be protected by Xen's PTI. It is thought that 32-bit Xen PV guests could still use Meltdown on themselves; protecting against this requires use of the KPTI feature inside the Linux kernel. As far as I am aware 32-bit KPTI is lagging behind the 64-bit version so those with 32-bit VPSes may wish to consider switching to a 64-bit kernel or upgrading to a new VPS, if they can't wait. - be compiled with gcc's new retpoline feature, which prevents CPUs from doing insecure branch prediction, therefore protecting you against variant 2 of Spectre. This is a complete protection for BitFolk's hypervisor against Spectre variant 2 attacks from guest kernels. It will not protect guests from attacks from inside their own VPS. For this you will need to make sure that your own kernel is compiled with retpoline support, with a compiler that understands the feature. I am not aware of the situation in other distributions but as I type this, Debian has already pushed out a version of gcc that has the retpoline feature to its stable ("stretch") and oldstable ("jessie") releases. A binary kernel package built with it is not yet available except in unstable, though. - have working PVH mode support. This is another way to run Xen virtual machines. It's faster/simpler than the PV mode that we currently use, it's also more secure, and it doesn't require use of qemu like HVM does. IIRC qemu is about 1.2 million lines of code, many times larger than Xen itself, and I've always been uncomfortable about it. Converting you all to PVH mode would provide the best protection against Meltdown and it would actually be more performant than PV mode, but sadly it requires some fixes in the guest kernel and in the bootloader that have only just gone in (like, late 4.14 kernel, early 4.15). We can't convert people to that mode until Linux distributions are shipping with new enough kernels, but it will be useful to have it available for early testing. - A couple of other unrelated security patches which will come out of embargo later. What's yet to come: - Any sort of mitigation for variant 1 of Spectre. People are still working on it, both in the Linux kernel, in Xen, and in other software. It's possible that fixes may only come in the Linux kernel rather than in Xen. - Updated Intel microcode. Intel released some updated microcode which features new CPU instructions to help avoid these problems, and/or reduce the performance impact of the techniques used. Shortly after release amid many reports of system instability they withdrew the update again, and are not currently recommending its use except for development purposes. So, we're still waiting for a stable release of that, and at the moment it's looking like decent fixes can be done in software so the urgency of a reboot just for this microcode update is low and I am inclined to roll it in with the next maintenance. That could change though. After the maintenance, what you need to think about: - If you're 32-bit you need to make a decision about Meltdown, whether you will wait for a 32-bit kernel fix or look at going to 64-bit by some means. - The retpoline-compiled hypervisor only protects BitFolk from you. To protect your own VPS against Spectre variant 2 attacks coming from within itself (like if it was tricked into running something malicious) you need a kernel that is compiled with retpoline support. These are pretty new, but they are out there. Debian pushed out a version of gcc with retpoline support to its stable ("stretch") and oldstable ("jessie") releases recently, but as I write this the only way to get a binary kernel package that was compiled with it is to use linux-image-amd64 from the unstable repository. Presumably that is going to filter through to stable etc in due course. Until then, you could use the package from unstable, or use the new gcc package to rebuild a kernel package… - If you are compiling C/C++ software, do you need to be doing it with a retpoline-aware compiler? - Look out for Spectre variant 1 fixes - they may be in your applications and/or kernel too. Although we can expect more hypervisor changes, after this (and the microcode) I expect the bulk of it to be in the kernels and applications. Solutions that are not being pursued: - Xen have some other mitigation options. They involve running PV guests inside either a PVH or HVM container. I've investigated these and they're just too complicated, they remove some useful functionality, and they still have performance implications about the same as XPTI. Longer term I'd like to be moving guests to PVH mode, and perhaps optionally HVM. That can't happen in a production capacity until a person can install a stable release of their favourite Linux and not have to know what PVH mode is, though. Regarding <https://github.com/speed47/spectre-meltdown-checker>: - It will always report that you are running under Xen PV and are vulnerable to Meltdown. It doesn't do any actual proof of concept exploit, it just detects PV mode and gives up. Once the new hypervisor has been deployed 64-bit guests will be protected by its PTI feature. As mentioned, 32-bit guests will still need to get PTI from their kernel. - Its reporting of Spectre variant 2 is accurate, so once you're running a retpoline-compiled kernel it will detect that. - Its reporting of microcode and new CPU instructions is, as far as I am aware, accurate. It is my understanding that once there is new microcode, guests will see it and be able to use these instructions. This could change though. That's all I can think of right now. I appreciate this is a lot to take in. If you have any questions please ask on or off the list. Once I get a sense of what is unclear I can perhaps make a wiki page that helps make things clearer. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting