21 Mar
2017
21 Mar
'17
6:03 p.m.
Hi,
A bug¹ has been discovered in Xen which has serious security
implications, therefore we must patch and reboot every host before
the security advisor comes out of embargo on 4 April.
We'll be sending out notifications shortly to let you know the exact
window in which the reboot will take place for your VPS(es), but
this is just a short note to let you know that the work will most
likely be taking place in the early hours (UK) of 30 March, 31 March
and 1 April.
A reminder that if you wish you can have your VPS suspended to and
then restored from storage, instead of shut down and booted:
https://panel.bitfolk.com/account/config/#prefs
https://tools.bitfolk.com/wiki/Suspend_and_restore
Cheers,
Andy
¹ XSA-212 - http://xenbits.xen.org/xsa/
--
https://bitfolk.com/ -- No-nonsense VPS hosting
23 Mar
23 Mar
12:34 a.m.
New subject: Maintenance on 30th, 31st and 1st (Was Re: [bitfolk] Upcoming reboot for security patch, XSA-212)
Hello,
On Tue, Mar 21, 2017 at 06:03:51PM +0000, Andy Smith wrote:
We'll be sending out notifications shortly to let
you know the exact
window in which the reboot will take place for your VPS(es),
Emails for this have now gone out, confirming the times of the
maintenance windows on Thursday 30 March, Friday 31 March and
Saturday 1 April.
Do note that the times are given in UTC; if you're in UK at those
times then local time is BST, one hour ahead.
If you haven't received the email notification, please check your
spam folder or equivalent and mark as "not spam" if you find it
there.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
29 Mar
29 Mar
3:34 p.m.
New subject: [bitfolk] Upcoming reboot for security patch, XSA-212
Reminder that the below work will be happening tonight, starting in
around 8h24m at 2017-03-30 00:00Z, with the rest of the work taking
place the following two nights.
Cheers,
Andy
On Tue, Mar 21, 2017 at 06:03:51PM +0000, Andy Smith wrote:
Hi,
A bug¹ has been discovered in Xen which has serious security
implications, therefore we must patch and reboot every host before
the security advisor comes out of embargo on 4 April.
We'll be sending out notifications shortly to let you know the exact
window in which the reboot will take place for your VPS(es), but
this is just a short note to let you know that the work will most
likely be taking place in the early hours (UK) of 30 March, 31 March
and 1 April.
A reminder that if you wish you can have your VPS suspended to and
then restored from storage, instead of shut down and booted:
https://panel.bitfolk.com/account/config/#prefs
https://tools.bitfolk.com/wiki/Suspend_and_restore
Cheers,
Andy
¹ XSA-212 - http://xenbits.xen.org/xsa/
--
https://bitfolk.com/ -- No-nonsense VPS hosting
30 Mar
30 Mar
3:03 a.m.
New subject: [bitfolk] Upcoming reboot for security patch, XSA-212
Hello,
On Wed, Mar 29, 2017 at 03:34:43PM +0000, Andy Smith wrote:
Reminder that the below work will be happening
tonight, starting in
around 8h24m at 2017-03-30 00:00Z, with the rest of the work taking
place the following two nights.
The first set of maintenance work was completed successfully, with a
few minor issues.
All of the customer VPSes that tried suspend-and-restore worked
fine, as far as I can determine.
pvgrub failed to parse the config files of four customer VPSes. Two
of these were Gentoo, one was CentOS 6 and one was Ubuntu 14.04. In
all cases they were using a legacy GRUB config file, and pvgrub
refused to load the kernel. These were reverted back to pygrub and
will have to be investigated later.
pvgrub failed to start at all for one customer. It may be because
they have more than 4GiB RAM and a 32-bit pvgrub. I'm not sure if
that supports PAE as this was a configuration I did not test.
For many customers this maintenance will be the first time they have
rebooted since we switched to pvgrub. pvgrub was actually tested
with all of the mentioned Linux distributions so I'm not sure why it
is failing for these four customers.
167 VPSes were rebooted tonight so 4 is quite a small number having
problems. As it is pretty quick and easy for me to revert a customer
back to pygrub, I am not going to revert everyone back. I will deal
with any failures as I see them, then investigate the problems after
this round of maintenance is complete.
I will now open support tickets with each of the affected customers,
and same for any problems that are encountered with the next two
days of maintenance work.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
31 Mar
31 Mar
2:49 a.m.
New subject: [bitfolk] Upcoming reboot for security patch, XSA-212
Hi,
On Wed, Mar 29, 2017 at 03:34:43PM +0000, Andy Smith wrote:
Reminder that the below work will be happening
tonight, starting in
around 8h24m at 2017-03-30 00:00Z, with the rest of the work taking
place the following two nights.
The second set of maintenance work has now been completed with one
slight problem.
One of the hosts that was being rebooted is also one of a pair of
routers for our IPv6 traffic, and it happened to be the one that was
active at the time. If/when it goes down it should automatically
fail over to the other one, but I decided to manually fail it over
to make the work simpler.
Unfortunately it seems that when that happens, it also disables IPv6
for all of the VPSes on the host itself (hen). This was unexpected
and took me a few minutes to work out what had happened. After
approximately 9 minutes I realised the problems were restricted to
v6 for VPSes on that single host, and I was able to restore their v6
connectivity. Of course, by that point it was time to reboot the
host anyway.
So, customers on host hen unexpectedly experienced approximately 9
minutes of no IPv6 just before they expectedly experienced it all
being rebooted anyway. Sorry about that! It is likely I never
noticed that issue before because it would have only happened when
the host was down.
Other than that I noticed no suspend-and-restore failures, and there
were only two pvgrub booting problem that I noticed.
The first one was similar to the others seen, which involved Gentoo
and pvgrub complaining that it can't load the kernel. That customer,
like the others, was reverted back to pygrub and I will need to do
further testing.
The other one appeared to be down to the customer having a 64-bit
kernel without telling us, so it wouldn't boot with 32-bit pvgrub.
Switching them to 64-bit pvgrub allowed it to boot.
I will discuss that with the individual customer, but as a reminder:
if you have a 64-bit kernel, we need to know. For a couple of years
now all new installs have been 64-bit, but if you have been a
customer for longer than that you probably started out as 32-bit. If
you've changed it to be 64-bit we need to know otherwise your VPS
won't boot.
You can tell what we think it is by looking at the summary page of
the Panel:
https://panel.bitfolk.com/account/
If it says "/ i686" then we think it's 32-bit.
You can also tell by logging in to your Xen Shell and typing the
"arch" command.
You can tell us it is something different by typing either "arch
x86_64" or "arch i686" in the Xen Shell.
More info here:
https://tools.bitfolk.com/wiki/Booting#Setting_your_architecture
The last remaining work for XSA-212 will take place tomorrow at
00:00Z and 01:00Z.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
1 Apr
1 Apr
2:09 a.m.
New subject: [bitfolk] Upcoming reboot for security patch, XSA-212
On Wed, Mar 29, 2017 at 03:34:43PM +0000, Andy Smith wrote:
Reminder that the below work will be happening
tonight, starting in
around 8h24m at 2017-03-30 00:00Z, with the rest of the work taking
place the following two nights.
The third and final set of maintenance work was completed
successfully.
There was one VPS which didn't boot under pvgrub and had to be
reverted back to pygrub. This time it was Debian jessie, which is
the first of those I had seen, but again the common patter was a
legacy-GRUB config in /boot/grub/grub.conf.
I have opened support tickets with every one of the customers (that
I know of) that had to be reverted back to pygrub, so we can work
out how to get them onto pvgrub. It will probably be as simple as
installing grub2.
All of the reboots for XSA-212 are done with now and the advisory
comes out of embargo at 2017-04-04 12:00Z, so if you are interested
in the details you will find them at http://xenbits.xen.org/xsa/
after that time.
Thanks for your patience through this work, especially to the small
number of customers who unexpectedly had issues with the
pygrub→pvgrub migration.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2822
days inactive
2833
days old
5 comments
1 participants
participants (1)
-
Andy Smith