12 Apr
2018
12 Apr
'18
5:51 a.m.
Hi,
Unfortunately a flaw has been discovered in the Xen hypervisor that
has security implications, so we're going to have to apply a fix and
reboot everything. This is likely to take place between 21 and 23
April. In the next couple of days we will send out direct mails to
all customers to let you know exactly when your one hour maintenance
window will be.
We should be able to use suspend/restore on this one so those who
have indicated that they want that should see that happen. You can
set that from:
<https://panel.bitfolk.com/account/config/>
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
16 Apr
16 Apr
6:23 p.m.
New subject: Security reboot needed, likely to be weekend of 21/22/23 Apr
On Thu, Apr 12, 2018 at 05:51:49AM +0000, Andy Smith wrote:
In the next couple of days we will send out direct
mails to all
customers to let you know exactly when your one hour maintenance
window will be.
Notifications for this have been sent out now. If you haven't
received one please look in your spam folder or similar and if still
not found then please let support(a)bitfolk.com know.
Note that the times given are in UTC; if you live in UK then you are
currently in BST time zone (UTC+1) and will have to adjust
accordingly.
As part of this maintenance we'll also be upgrading the CPU
microcode to help with future Spectre and Meltdown mitigations.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
22 Apr
22 Apr
4:04 a.m.
New subject: Why "hobgoblin"'s maintenance took so long today (Was: Re: Security reboot needed, likely to be weekend of 21/22/23 Apr)
Hi,
I made a mistake and as a result today's scheduled maintenance for
host "hobgoblin" took much longer than expected. The first customer
VPSes on this host were shut down at around 01:30Z and VPSes began
to be started up again at 01:57Z, which is about twice as long as it
normally takes.
I first noticed there was a problem when the host reached its grub
bootloader and the desired hypervisor version wasn't present.
At this point I didn't want to allow a normal boot into the previous
hypervisor because all of the suspended VPSes would be started, and
then all of the others would be booted. This would take significant
time and would of course result in two suspend/shutdown events for
most customers on that server.
Instead I had to boot the server into single user mode, disable
starting of VPSes, then tell it to boot normally. I then installed
the new hypervisor that should have been there already and updated
the bootloader. It was then possible to reboot and boot as intended.
The majority of the excess time was spent waiting for BIOS etc to go
through the motions.
After the rest of the work was completed I looked into what happened
here. It seems that due to a typo, the new version of the hypervisor
was not copied to this one particular host, and this was not noticed
until I came to try to boot into it.
I will see if I can revise my procedures to detect problems of this
nature in future. I'm not yet sure what I can do, but after the last
incident of human error¹ I started writing up a plan for each
maintenance and have found this useful even for work which I once
considered routine. At the very least I can include an explicit step
in such plans to check that the hypervisor and kernel version to be
booted in to are actually present before a reboot takes place.
Apologies again for the extended outage period.
Cheers,
Andy
¹ https://lists.bitfolk.com/lurker/message/20170914.003408.5d4ddfaf.en.html
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
23 Apr
23 Apr
3:21 a.m.
New subject: Security reboot needed, likely to be weekend of 21/22/23 Apr
Hi,
This maintenance work has now been completed.
The security issues fixed are listed at:
<http://xenbits.xen.org/xsa/>
but details will not be available until after the embargo ends at
12:00Z on Wednesday 25 April.
Updated CPU firmware was also applied. For one host there is a later
CPU firmware version available, but Intel have not yet released it
as stable so I will not deploy it yet.
Aside from the firmware update that Intel have for some reason held
back, it's possible that there will be further CPU firmware releases
but unless they are recommended as critical I will not deploy them
on their own; I will wait until there is reason to apply some other
form of update that requires a reboot.
Thanks for your patience with the disruption these updates have
caused!
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
2193
days inactive
2204
days old
3 comments
1 participants
participants (1)
-
Andy Smith