Hi,
If you are running a memcached server please make sure that it
either doesn't listen on UDP or else that it is properly firewalled.
Publicly available memcached servers can provide a 50,000x traffic
amplification:
<https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-por…>
As there is no authentication in the memcached protocol, having it
publicly available is generally a misconfiguration anyway.
We will start scanning for and nagging about this soon.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
On Sat, Jan 06, 2018 at 08:34:48AM +0000, Mike Zanker wrote:
> On 4 Jan 2018, at 17:57, Andy Smith <andy(a)bitfolk.com> wrote:
> > In the mean time you can use the kernel package from the CentOSPlus
> > repository which does have this fix and the KPTI one.
> >
> > https://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus
> >
> > All of this was researched by a customer having the problem today
> > and it resolved it for them.
>
> This was fine until CentOS updated the CentOSPlus kernel
> overnight. Now the updated one fails to boot in exactly the same
> way as the standard CentOS kernel.
Just a quick note that someone has built a CentOS 7 kernel with a
fix for this problem, as described here:
<https://kevandrews.uk/centos-7-xen-pv-guests-failing-boot-kernel-3100-69317…>
Following the link to the bug report at
<https://bugs.centos.org/view.php?id=14347>, it says that the next
CentOS Plus kernel update will include this fix.
So if you're using CentOS 7 and are affected by this then it sounds
like the way to go would be to install that kernel and get future
kernels from Plus also.
Hopefully at the next point release of CentOS 7 they would include a
fixed kernel in the main repository.
As far as I am aware the kernel package in CentOS 6 has already had
this problem fixed.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Notifications have just been sent out letting you know the hour long
maintenance window during which the host that your VPS is on will be
rebooted for security patches.
If you have not received the notification please check your spam
folders etc. and if still having no luck please contact
support(a)bitfolk.com.
Apologies for the short notice of this, but now that I feel I have a
reasonable plan covering a large part of the problem space, it is
best to get this done as soon as possible.
I've deliberately left most of the technical detail out of the
reboot notification. The technical details are overwhelming. If you
aren't a particularly technical person then my advice would be:
- Make sure you are running a security-supported release of your
chosen Linux distribution and that you keep it up to date. Between
them and us you should eventually get to safety, just bear in mind
that this is an evolving situation and not many Linux vendors are
willing to push out the latest fixes without extensive testing.
For those who really want them, here are some more technical details.
The newly-deployed hypervisor will:
- support Page Table Isolation, similar to the Linux kernel's KPTI,
to protect against Meltdown.
This feature will protect BitFolk's hypervisor from Meltdown
attacks from all customers.
At the moment all BitFolk VPSes are paravirtual (PV) guests. For
64-bit VPSes, this Xen-level PTI also protects against Meltdown
attacks from within their own kernel or user space. Thus, although
your kernel will report that KPTI is disabled, you will be
protected by Xen's PTI.
It is thought that 32-bit Xen PV guests could still use Meltdown
on themselves; protecting against this requires use of the KPTI
feature inside the Linux kernel. As far as I am aware 32-bit KPTI
is lagging behind the 64-bit version so those with 32-bit VPSes
may wish to consider switching to a 64-bit kernel or upgrading to
a new VPS, if they can't wait.
- be compiled with gcc's new retpoline feature, which prevents CPUs
from doing insecure branch prediction, therefore protecting you
against variant 2 of Spectre.
This is a complete protection for BitFolk's hypervisor against
Spectre variant 2 attacks from guest kernels. It will not protect
guests from attacks from inside their own VPS. For this you will
need to make sure that your own kernel is compiled with retpoline
support, with a compiler that understands the feature.
I am not aware of the situation in other distributions but as I
type this, Debian has already pushed out a version of gcc that has
the retpoline feature to its stable ("stretch") and oldstable
("jessie") releases. A binary kernel package built with it is not
yet available except in unstable, though.
- have working PVH mode support.
This is another way to run Xen virtual machines. It's
faster/simpler than the PV mode that we currently use, it's also
more secure, and it doesn't require use of qemu like HVM does.
IIRC qemu is about 1.2 million lines of code, many times larger
than Xen itself, and I've always been uncomfortable about it.
Converting you all to PVH mode would provide the best protection
against Meltdown and it would actually be more performant than PV
mode, but sadly it requires some fixes in the guest kernel and in the
bootloader that have only just gone in (like, late 4.14 kernel,
early 4.15). We can't convert people to that mode until Linux
distributions are shipping with new enough kernels, but it will be
useful to have it available for early testing.
- A couple of other unrelated security patches which will come out
of embargo later.
What's yet to come:
- Any sort of mitigation for variant 1 of Spectre.
People are still working on it, both in the Linux kernel, in Xen,
and in other software. It's possible that fixes may only come in
the Linux kernel rather than in Xen.
- Updated Intel microcode.
Intel released some updated microcode which features new CPU
instructions to help avoid these problems, and/or reduce the
performance impact of the techniques used. Shortly after release
amid many reports of system instability they withdrew the update
again, and are not currently recommending its use except for
development purposes.
So, we're still waiting for a stable release of that, and at the
moment it's looking like decent fixes can be done in software so
the urgency of a reboot just for this microcode update is low and
I am inclined to roll it in with the next maintenance. That could
change though.
After the maintenance, what you need to think about:
- If you're 32-bit you need to make a decision about Meltdown,
whether you will wait for a 32-bit kernel fix or look at going to
64-bit by some means.
- The retpoline-compiled hypervisor only protects BitFolk from you. To
protect your own VPS against Spectre variant 2 attacks coming from
within itself (like if it was tricked into running something
malicious) you need a kernel that is compiled with retpoline
support.
These are pretty new, but they are out there. Debian pushed out a
version of gcc with retpoline support to its stable ("stretch")
and oldstable ("jessie") releases recently, but as I write this
the only way to get a binary kernel package that was compiled with
it is to use linux-image-amd64 from the unstable repository.
Presumably that is going to filter through to stable etc in due
course. Until then, you could use the package from unstable, or
use the new gcc package to rebuild a kernel package…
- If you are compiling C/C++ software, do you need to be doing it
with a retpoline-aware compiler?
- Look out for Spectre variant 1 fixes - they may be in your
applications and/or kernel too. Although we can expect more
hypervisor changes, after this (and the microcode) I expect the
bulk of it to be in the kernels and applications.
Solutions that are not being pursued:
- Xen have some other mitigation options. They involve running PV
guests inside either a PVH or HVM container. I've investigated
these and they're just too complicated, they remove some useful
functionality, and they still have performance implications about
the same as XPTI.
Longer term I'd like to be moving guests to PVH mode, and perhaps
optionally HVM. That can't happen in a production capacity until a
person can install a stable release of their favourite Linux and
not have to know what PVH mode is, though.
Regarding <https://github.com/speed47/spectre-meltdown-checker>:
- It will always report that you are running under Xen PV and are
vulnerable to Meltdown. It doesn't do any actual proof of concept
exploit, it just detects PV mode and gives up. Once the new
hypervisor has been deployed 64-bit guests will be protected by
its PTI feature. As mentioned, 32-bit guests will still need to
get PTI from their kernel.
- Its reporting of Spectre variant 2 is accurate, so once you're
running a retpoline-compiled kernel it will detect that.
- Its reporting of microcode and new CPU instructions is, as far as I am
aware, accurate. It is my understanding that once there is new
microcode, guests will see it and be able to use these
instructions. This could change though.
That's all I can think of right now. I appreciate this is a lot to
take in. If you have any questions please ask on or off the list.
Once I get a sense of what is unclear I can perhaps make a wiki
page that helps make things clearer.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
