21 Jul
2016
21 Jul
'16
11:02 a.m.
Hi,
By now all customers should have received notification of scheduled
maintenance that will be required due to a serious security flaw in
the hypervisor software that we use (Xen).
If you have not seen an email regarding this then please check your
spam folders etc.
The full details¹ are in the email you've already received and I'm
only sending this so as to have a public notification I can link to
when people raise support tickets to ask what is going on. :)
Anyway, the hosts have all been patched and the maintenance consists
of merely rebooting them to boot into the new hypervisor. This will
happen across three nights.
In previous non-SSD days this used to take around 30 minutes to shut
down all VPSes, reboot and boot them all again. These days I expect
it to be much shorter, maybe 5 minutes. So, you should see a clean
shut down followed by a boot a few minutes later.
It is important that you ensure that your VPS boots cleanly with all
services you expect running to be running. We offer free Nagios
monitoring which can be useful for assuring yourself that everything
you expect to be running really is running. Also if I see Nagios
looks more broken afterwards than it was to start with then I will
have a quick investigate. If interested in having that set up then
please contact support(a)bitfolk.com.
Cheers,
Andy
¹ Well, not any details about the bug itself. These are under
embargo until mid day Tuesday 26 July.
--
http://bitfolk.com/ -- No-nonsense VPS hosting
24 Jul
24 Jul
2:28 a.m.
New subject: [bitfolk] Scheduled maintenance on 24–26 July
Hello,
The work scheduled for 2016-07-24 has now been completed. Further
work on the remaining hosts is scheduled for the next two days.
Unless your VPS is on host "hen" there were no unexpected issues and
you can stop reading now.
A mistake was made when working on host "hen": instead of shutting
down customer VPSes prior to rebooting the host, I accidentally
saved them to disk.
Once I'd realised what I'd done I decided it would be best to let
them restore from disk on boot as otherwise they would all
experience something similar to a power fail and have to
replay/repair their filesystems on boot.
In theory save/restore is a great idea (it's similar to hibernation)
as it results in less disruption, but unfortunately not all Linux
kernels cope well with it, which is why we don't do it. I normally
prefer to give you the greater certainty of a clean shut down and
boot.
So, if your VPS is on "hen" and…
- You're wondering why it still shows the same uptime as before…
…it's because it was saved/restored rather than rebooted.
- It's broken in some way…
…it's unfortunately very likely because it didn't handle
save/restore well. A clean reboot will probably be necessary.
Sorry about this. :(
I went through every VPS on "hen" that was showing problems in
Nagios and I found only two that didn't handle the save/restore. One
was running Debian 6.0 and another was running Ubuntu 16.04. I
rebooted these and Nagios is now happy about them.
This does not mean that every Debian 6.0 or Ubuntu 16.04 VPS will
fail to save/restore. Also very few customers have Nagios alerting
so there may be other VPSes which didn't like it but I am unaware of
them.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
25 Jul
25 Jul
2:03 a.m.
New subject: [bitfolk] Scheduled maintenance on 24–26 July
Hello,
On Thu, Jul 21, 2016 at 11:02:41AM +0000, Andy Smith wrote:
By now all customers should have received notification
of scheduled
maintenance that will be required due to a serious security flaw in
the hypervisor software that we use (Xen).
The work scheduled for 2016-07-25 has now been completed. Further
work on the remaining hosts is scheduled for tomorrow.
The work went without incident (form our side). Two customer IPs no
longer respond, but this appears to be a local configuration /
networking issue on those VPSes.
If you are still seeing problems then please connect to your console
to check things out.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
26 Jul
26 Jul
5:06 a.m.
New subject: [bitfolk] Scheduled maintenance on 24–26 July
Hello,
On Thu, Jul 21, 2016 at 11:02:41AM +0000, Andy Smith wrote:
By now all customers should have received notification
of scheduled
maintenance that will be required due to a serious security flaw in
the hypervisor software that we use (Xen).
The work scheduled for the early hours of this morning was completed
without incident from our side. This was the last of the required
work.
According to our monitoring, two customer IPs remain unreachable but
these appear to be local configuration / networking issues.
The details of the vulnerability concerned (XSA-182) should come out
of embargo today at 12:00Z, so if you're interested please see:
http://xenbits.xen.org/xsa/
after that time for more info.
Thanks again for your patience.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
3036
days inactive
3041
days old
3 comments
1 participants
participants (1)
-
Andy Smith