Reboots will be necessary to address security issues, probably early hours 17/18/19 October
7 Oct
2020
7 Oct
'20
9:20 a.m.
Hello,
Unfortunately - and annoyingly only a month since the last lot -
some serious security bugs have been discovered in the Xen
hypervisor and fixes for these have now been pre-disclosed, with an
embargo that ends at 1200Z on 20 October 2020.
As a result we will need to apply these fixes and reboot everything
before that time. We are likely to do this in the early hours of the
morning UK time, on 17, 18 and 19 October.
In the next few days individual emails will be sent out confirming
to you which hour long maintenance window your services are in. The
times will be in UTC; please note that UK is currently observing
daylight savings and as such is currently at UTC+1. We expect the
work to take between 15 and 30 minutes per bare metal host.
If you have opted in to suspend and restore¹ then your VM will be
suspended to storage and restored again after the host it is on is
rebooted. Otherwise your VM will be cleanly shut down and booted
again later.
If you cannot tolerate the downtime then please contact
support(a)bitfolk.com. We may be able to migrate² you to
already-patched hardware before the regular maintenance starts. You
can expect a few tens of seconds of pausing in that case. This
process uses suspend&restore so has the same caveats.
It is disappointing to have another round of security reboots 28
days after the last lot, though before that there was a gap of about
330 days. Still, as there are security implications we have no
choice in the matter.
Cheers,
Andy
¹ https://tools.bitfolk.com/wiki/Suspend_and_restore
² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
7 Oct
7 Oct
4:05 p.m.
Hello,
On Wed, Oct 07, 2020 at 09:20:29AM +0000, Andy Smith wrote:
In the next few days individual emails will be sent
out confirming
to you which hour long maintenance window your services are in.
These individual emails have now gone out. If you haven't received
yours please take a look in your spam folder or similar — and then
if you do find it there, please whitelist us or mark as "safe
sender" or whatever.
If you still can't find it, please contact us at support(a)bitfok.com.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
4:10 p.m.
New subject: Feedback sought regarding the scheduled maintenance notification (Was: Re: Reboots will be necessary to address security issues, probably early hours 17/18/19 October)
Hi,
On Wed, Oct 07, 2020 at 04:05:49PM +0000, Andy Smith wrote:
These individual emails have now gone out.
I'd be interested in any feedback you have about these emails.
Last time we still did get some people replying just to say "that's
fine" etc desite it saying both at the start and end of the email
that there's no need to do that (it creates a support ticket).
There was some feedback that the email is too long, with too much
information. This time I added some section headings, but I didn't
feel that any of it could be removed. If you think otherwise, do let
us know what you think doesn't need to be in there - though please
do appreciate that not every customer has the same requirements or
lack thereof that you do!
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
4:32 p.m.
New subject: Feedback sought regarding the scheduled maintenance notification (Was: Re: Reboots will be necessary to address security issues, probably early hours 17/18/19 October)
Hi Andy,
You could perhaps be slightly more explicit in asking people not to
reply. Rather than "You do not need to respond unless...", you could
phrase it as "Please do not respond unless..."
The former doesn't make it clear that it's actually a hassle for you if
people respond to say "Fine". Sending a positive response may just feel
like a polite thing to do.
Cheers,
John
--
Xronos Scheduler - https://xronos.uk/
All your school's schedule information in one place.
Timetable, activities, homework, public events - the lot
Live demo at https://schedulerdemo.xronos.uk/
4:56 p.m.
New subject: Feedback sought regarding the scheduled maintenance notification (Was: Re: Reboots will be necessary to address security issues, probably early hours 17/18/19 October)
For me it would be useful to have the reboot time window info at the very beginning of the
mail.
And as a bonus not assume that we all use mono spaced fonts when doing ASCII art tables.
;-)
Something like
"
Your VSPes will be rebooted at this time(s):
xxx (85.119.82.000) 2020-10-19 01:00
yyy (85.119.83.000) 2020-10-17 00:00
If you are OK with that, no need to reply to this email, since it creates more work.
For more info read on.
"
Since I have read your post on the list, I know enough already before I get the mail with
the exact timings
Cheers,.
__
/ony
-------
Wednesday, October 7, 2020, 5:10:44 PM, Andy wrote:
Hi,
On Wed, Oct 07, 2020 at 04:05:49PM +0000, Andy Smith
wrote:
> These individual emails have now gone out.
I'd be interested in any feedback you have about
these emails.
Last time we still did get some people replying just
to say "that's
fine" etc desite it saying both at the start and end of the email
that there's no need to do that (it creates a support ticket).
There was some feedback that the email is too long,
with too much
information. This time I added some section headings, but I didn't
feel that any of it could be removed. If you think otherwise, do let
us know what you think doesn't need to be in there - though please
do appreciate that not every customer has the same requirements or
lack thereof that you do!
Cheers,
Andy
5:28 p.m.
New subject: Feedback sought regarding the scheduled maintenance notification (Was: Re: Reboots will be necessary to address security issues, probably early hours 17/18/19 October)
Hi Tony,
On Wed, Oct 07, 2020 at 05:56:27PM +0100, Tony Andersson wrote:
For me it would be useful to have the reboot time
window info at the very beginning of the mail.
Okay, you're not the only person to have given this feedback, so in
future I will try to put that information nearer the top.
And as a bonus not assume that we all use mono spaced
fonts when doing ASCII art tables. ;-)
Perhaps it is time to consider HTML emails… (with a proper plain
text part, obviously)
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
7:13 p.m.
New subject: Feedback sought regarding the scheduled maintenance notification (Was: Re: Reboots will be necessary to address security issues, probably early hours 17/18/19 October)
On 2020-10-07 17:28+0000, Andy Smith wrote:
Passionate mutt user here, please keep plain text first-class! The HTML
could just be the text/plain within a <pre> :)
Ed
Hi Tony,
On Wed, Oct 07, 2020 at 05:56:27PM +0100, Tony Andersson wrote:
> For me it would be useful to have the reboot time window info at the
> very beginning of the mail.
Okay, you're not the only person to have given
this feedback, so in
future I will try to put that information nearer the top.
At the risk of sounding like a busy person, I don't always get time to
read my mail. I did like the TL;DR that were at the top.
However, when I do get time (75%) I do get to read and I thoroughly
enjoy the well ordered detail when there have been problems. A
writing/cognitive quality that I often lack.
And as a bonus
not assume that we all use mono spaced fonts when
doing ASCII art tables. ;-)
Perhaps it is time to consider HTML emails… (with a proper plain
text part, obviously) 
8 Oct
8 Oct
7:05 a.m.
New subject: Feedback sought regarding the scheduled maintenance notification (Was: Re: Reboots will be necessary to address security issues, probably early hours 17/18/19 October)
Hi,
On Wed 07 Oct 2020 16:10:44 GMT, Andy Smith wrote:
I'd be interested in any feedback you have about
these emails.
I think that a reminder some days before the maintenance could be
usefull. There is a high probabilty that I will forget what my
maintenance window is in some days; and when my VM will be rebooted I
will first ask me why, and then think about XSA.
--
Alarig
19 Oct
19 Oct
2:36 a.m.
Hello,
On Wed, Oct 07, 2020 at 09:20:29AM +0000, Andy Smith wrote:
Unfortunately - and annoyingly only a month since the
last lot -
some serious security bugs have been discovered in the Xen
hypervisor and fixes for these have now been pre-disclosed, with an
embargo that ends at 1200Z on 20 October 2020.
As a result we will need to apply these fixes and reboot everything
before that time. We are likely to do this in the early hours of the
morning UK time, on 17, 18 and 19 October.
This maintenance work has now been completed, without incident. The
details of the security issues which were fixed will appear at:
https://xenbits.xen.org/xsa/
after 1200Z on 20 October.
Thanks for your patience during this disruption.
During the course of this work 102 VPSes were suspended and restored
and as far as I'm aware there were no issues with that. This is
based on the fact that they still respond to ping afterwards and no
one has told us that things broke.
There have been problems in the past; suspend and restore is opt-in
and VPSes which have had problems are opted back out of it again.
You can learn more about suspend and restore here:
https://tools.bitfolk.com/wiki/Suspend_and_restore
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
1284
days inactive
1296
days old
8 comments
5 participants
participants (5)
-
Alarig Le Lay -
Andy Smith -
Ed Neville -
John Winters -
Tony Andersson