10 Sep
2017
10 Sep
'17
1:01 a.m.
Hi,
On Tue, Aug 29, 2017 at 05:27:13PM +0000, Andy Smith wrote:
So, we're going to have to patch everything and
reboot before then.
This will very likely be taking place over three nights starting in
the early hours (BST) of Sunday 10 September, but we will be sending
out an individual email to every customer confirming when they will
be affected.
For the last hour I've been working on "elephant", the first (live)
server to be patched and rebooted.
Unfortunately I made an error, which initially I thought would only
cause a few extra minutes of downtime, but turned out later to
require a complete reboot with not even the ability to cleanly shut
down VPSes.
That has now happened, and all customer VPSes on "elephant" should
now have booted. I am now going through our Nagios to check for any
that might be having problems.
I am really sorry that this happened. I could have tested my
proposed action on test hardware but I was sure I had done it before
without incident, and I was wrong.
I can't at this stage go into exact details regarding what my error
was because this would reveal details about where the security
vulnerabilities lie, so that will have to wait until after the
embargo has ended on Tuesday 12th. I will give more details then.
Apologies again,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting