30 Nov
2016
30 Nov
'16
6 a.m.
Hi,
Rather disappointingly there's been another round of security
advisories for Xen, some of which affect the configuration in use at
BitFolk:
http://xenbits.xen.org/xsa/
So unfortunately that means another round of "patch and reboot
everything" for us.
We will send out direct emails informing you of a two hour window in
which the work affecting your VPS is going to take place, but that
won't be sent for a week or so as testing of the patches will be
necessary first.
I am going to try to get a checkbox added to the panel website so
that those of you who are brave enough to try it can indicate that
you'd prefer for your VPS to be suspended and restored rather than
shutdown and booted.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
30 Nov
30 Nov
9:53 a.m.
Andy,
You kindly suspended my VM last time and it worked fine. - Certainly for
me less hassle then a reboot.
Thanks
Leo
On 30-11-2016 07:00, Andy Smith wrote:
Hi,
Rather disappointingly there's been another round of security
advisories for Xen, some of which affect the configuration in use at
BitFolk:
http://xenbits.xen.org/xsa/
So unfortunately that means another round of "patch and reboot
everything" for us.
We will send out direct emails informing you of a two hour window in
which the work affecting your VPS is going to take place, but that
won't be sent for a week or so as testing of the patches will be
necessary first.
I am going to try to get a checkbox added to the panel website so
that those of you who are brave enough to try it can indicate that
you'd prefer for your VPS to be suspended and restored rather than
shutdown and booted.
Cheers,
Andy
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
4 Dec
4 Dec
4:31 a.m.
New subject: Upcoming reboots for security fixes prior to 13 December
Hi,
On Wed, Nov 30, 2016 at 06:00:35AM +0000, Andy Smith wrote:
We will send out direct emails informing you of a two
hour window in
which the work affecting your VPS is going to take place, but that
won't be sent for a week or so as testing of the patches will be
necessary first.
An email has now gone out informing you of the 90 minute (not 2
hour) window in which the work will take place for each of your
VPSes. If you haven't received it, please do check your spam folders
and whitelist where appropriate.
As mentioned in the email it is now possible for you to indicate
that you would like your VPS to be suspended and restored instead of
shut down and booted again. You can do that from:
https://panel.bitfolk.com/account/config/#prefs
It is not the default because depending on your kernel and
applications, it sometimes doesn't work. You can read more about it
here:
https://tools.bitfolk.com/wiki/Suspend_and_restore
If you indicate that you would like suspend/restore to be attempted
then we will add Nagios ping checks for your VPS, as most of the
time a failure to restore will lead to your VPS being unpingable. A
failing Nagios ping check will cause us to notice while carrying out
the maintenance, and we'll then have a look at your console and
reboot your VPS where necessary.
There is still the possibility that some application you run will
not behave correctly, however, which is something only you can check
for.
If you have any questions please do let us know, either here or at
support(a)bitfolk.com.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
11 Dec
11 Dec
3:56 a.m.
Hello,
On Sun, Dec 04, 2016 at 04:31:24AM +0000, Andy Smith wrote:
As mentioned in the email it is now possible for you
to indicate
that you would like your VPS to be suspended and restored instead of
shut down and booted again. You can do that from:
https://panel.bitfolk.com/account/config/#prefs
It is not the default because depending on your kernel and
applications, it sometimes doesn't work.
The first day's maintenance has now been completed. Across those
three machines, 5 customers opted to try suspend/restore and as far
as I can see they were all successful.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
13 Dec
13 Dec
3:17 a.m.
New subject: Upcoming reboots for security fixes prior to 13 December
Hi,
On Wed, Nov 30, 2016 at 06:00:35AM +0000, Andy Smith wrote:
Rather disappointingly there's been another round
of security
advisories for Xen, some of which affect the configuration in use at
BitFolk:
The maintenance work to patch against this bug (XSA-200) has now
been completed, without incident. During the embargo for XSA-200
more security bugs were found so fixes for these were deployed also.
As per:
http://xenbits.xen.org/xsa/
the embargo for XSA-200 ends in just under 9 hours, and the embargo
for the others ends at 2016-12-21 12:00Z.
Thanks for your patience during this work.
I am going to try to get a checkbox added to the panel
website so
that those of you who are brave enough to try it can indicate that
you'd prefer for your VPS to be suspended and restored rather than
shutdown and booted.
In the end customers chose for 22 of their VPSes to be suspended and
restored and as far as I am aware¹ there was a 100% success rate in
this.
If you would like to enable suspend and restore for your VPS you can
do so from:
https://panel.bitfolk.com/account/config/#prefs
Cheers,
Andy
¹ We added Nagios ping checks for every one of these VPSes, if they
didn't exist already, and all of those checks failed then recovered
as expected. In the failure modes I have seen network connectivity
does not return.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
14 Dec
14 Dec
12:41 p.m.
On Tue, Dec 13, 2016 at 03:17:53AM +0000, Andy Smith wrote:
Hi,
On Wed, Nov 30, 2016 at 06:00:35AM +0000, Andy Smith wrote:
Now that this is out of embargo:
http://xenbits.xen.org/xsa/advisory-200.html
…perhaps I could have a bit of feedback from you as to whether we
did the right thing in enforcing a reboot here.
Discussion around the bug (unfortunately on a private list for
discussion of the security bugs while they're under embargo, so I
can't show you) indicated that it *probably* wasn't very dangerous.
The impact is basically that a guest VM could read either 32 or 96
bits of hypervisor memory. It would probably be a different 96 bits
each time, but probably not anything private.
Of course, the Xen folks are not being paid to speculate on exactly
how exploitable that bug is, so they can't be drawn into making any
statements about how likely it is that any sensitive data could be
read. Their official analysis quite reasonably stops at "guests can
read some hypervisor memory; here is the patch for that".
As the matter was under embargo it was difficult to discuss any
further than that, hence all I've got is "it probably isn't that
serious". That was based on some insights that it seems similar to
an earlier bug that also probably wasn't as serious as its
corresponding advisory implied - bearing in mind that an advisory
should tell you the worst case. Analysis of a similar bug:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-012-2014.txt
That one involved the ability to read a couple of kB of hypervisor
memory, whereas XSA-200 was limited to 96 bits. So XSA-200 is almost
likely less serious than XSA-108 was.
So, did we do the right thing in patching and rebooting this before
the embargo date (leading to less than 2 weeks of notice of a reboot
for customers)?
Should we have patched and rebooted but over a longer schedule?
Should we have not bothered with a reboot at all?
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Rather disappointingly there's been another
round of security
advisories for Xen, some of which affect the configuration in use at
BitFolk:
The maintenance work to patch against this bug (XSA-200) has now
been completed, without incident. 
1:21 p.m.
Andy Smith <andy(a)bitfolk.com> said, in message
20161214124152.GU21587(a)bitfolk.com:
…perhaps I could have a bit of feedback from you as to whether we
did the right thing in enforcing a reboot here.
Discussion around the bug (unfortunately on a private list for
discussion of the security bugs while they're under embargo, so I
can't show you) indicated that it *probably* wasn't very dangerous.
Short (advertised) outages on my VM aren't that much of a problem, so I'm
pretty relaxed about the reboots.
IMHO...
The thing about security holes like this is that creative hackers have a
tendency to leverage them into much bigger things later. Reacting to the bug
while it's under embargo, even if it seems obscure, is probably the best
thing to do, rather than leaving it until someone finds a way of using
it to do something much scarier. At which point the 2 weeks' notice
might turn into 0 days.
Cheers,
Alun.
--
Dr. Alun Jones, auj(a)aber.ac.uk, 01970 622637
Mathemateg, Ffiseg a Chyfrifiadureg, Prifysgol Aberystwyth
Mathematics, Physics & Computer Science, Aberystwyth University
1:25 p.m.
New subject: Upcoming reboots for security fixes prior to 13 December
I agree entirely with Alun.
Cheers,
Roger
On Wed, Dec 14, 2016 at 1:21 PM, Alun <auj(a)ty-penguin.org.uk> wrote:
Andy Smith <andy(a)bitfolk.com> said, in message
20161214124152.GU21587(a)bitfolk.com:
…perhaps I could have a bit of feedback from you as to whether we
did the right thing in enforcing a reboot here.
Discussion around the bug (unfortunately on a private list for
discussion of the security bugs while they're under embargo, so I
can't show you) indicated that it *probably* wasn't very dangerous.
Short (advertised) outages on my VM aren't that much of a problem, so I'm
pretty relaxed about the reboots.
IMHO...
The thing about security holes like this is that creative hackers have a
tendency to leverage them into much bigger things later. Reacting to the bug
while it's under embargo, even if it seems obscure, is probably the best
thing to do, rather than leaving it until someone finds a way of using
it to do something much scarier. At which point the 2 weeks' notice
might turn into 0 days.
Cheers,
Alun.
--
Dr. Alun Jones, auj(a)aber.ac.uk, 01970 622637
Mathemateg, Ffiseg a Chyfrifiadureg, Prifysgol Aberystwyth
Mathematics, Physics & Computer Science, Aberystwyth University
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
1:42 p.m.
ons 2016-12-14 klockan 13:21 +0000 skrev Alun:
Andy Smith <andy(a)bitfolk.com> said, in message
20161214124152.GU21587(a)bitfolk.com:
>
…perhaps I could have a bit of feedback from you as to whether we
> did the right thing in enforcing a reboot here.
I'm all for these short notice reboots[*]. Partly because of the
rather-safe-than-sorry scenario expressed by Alun. Partly because it's
2016, and we really should know how to build our virtual infrastructure
in such a way that it survives the occasional reboots. Especially if
they are advised a head of time.
// Andreas
2:24 p.m.
New subject: Upcoming reboots for security fixes prior to 13 December
D
itto on the previous comments by Alun and Andreas.
Cheers
Brendan
15 Dec
15 Dec
11:08 a.m.
** Andreas Olsson <andreas(a)arrakis.se> [2016-12-14 13:42]:
ons 2016-12-14 klockan 13:21 +0000 skrev Alun:
** end quote [Andreas Olsson]
Seconded, thirded, or however many we are up to now :)
--
Paul Tansom | Aptanet Ltd. | https://www.aptanet.com/ | 023 9238 0001
Vice Chair, FSB Portsmouth & SE Hampshire Branch | http://www.fsb.org.uk/
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
Andy Smith <andy(a)bitfolk.com> said, in
message
20161214124152.GU21587(a)bitfolk.com:
>
…perhaps I could have a bit of feedback from you as to whether we
> did the right thing in enforcing a reboot
here.
I'm all for these short notice reboots[*]. Partly because of the
rather-safe-than-sorry scenario expressed by Alun. Partly because it's
2016, and we really should know how to build our virtual infrastructure
in such a way that it survives the occasional reboots. Especially if
they are advised a head of time.
// Andreas 
11:12 a.m.
New subject: Upcoming reboots for security fixes prior to 13 December
Yeah, that. Will never complain about Andy taking security seriously.
Kind regards
Murray Crane
On 15 December 2016 at 11:08, Paul Tansom <paul(a)aptanet.com> wrote:
** Andreas Olsson <andreas(a)arrakis.se>
[2016-12-14 13:42]:
ons 2016-12-14 klockan 13:21 +0000 skrev Alun:
** end quote [Andreas Olsson]
Seconded, thirded, or however many we are up to now :)
--
Paul Tansom | Aptanet Ltd. | https://www.aptanet.com/ | 023 9238
0001
Vice Chair, FSB Portsmouth & SE Hampshire Branch |
http://www.fsb.org.uk/
============================================================
=================
Registered in England | Company No: 4905028 | Registered Office: Ralls
House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7
6XP
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
Andy Smith <andy(a)bitfolk.com> said, in
message
20161214124152.GU21587(a)bitfolk.com:
>
…perhaps I could have a bit of feedback from you as to whether we
> did the right thing in enforcing a reboot
here.
I'm all for these short notice reboots[*]. Partly because of the
rather-safe-than-sorry scenario expressed by Alun. Partly because it's
2016, and we really should know how to build our virtual infrastructure
in such a way that it survives the occasional reboots. Especially if
they are advised a head of time.
// Andreas 
3:47 p.m.
New subject: Upcoming reboots for security fixes prior to 13 December
I'm with the consensus here - the mild inconvenience that I can warn my
web-app users about in advance is more than an OK given the importance of
security updates. Also given it leaks a different bit of memory each time
it could be repetitively used until something useful is retrieved.
On 15 Dec 2016 11:12 am, "Murray Crane" <murray.crane(a)gmail.com> wrote:
Yeah, that. Will never complain about Andy taking
security seriously.
Kind regards
Murray Crane
On 15 December 2016 at 11:08, Paul Tansom <paul(a)aptanet.com> wrote:
** Andreas Olsson <andreas(a)arrakis.se>
[2016-12-14 13:42]:
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
ons 2016-12-14 klockan 13:21 +0000 skrev Alun:
** end quote [Andreas Olsson]
Seconded, thirded, or however many we are up to now :)
--
Paul Tansom | Aptanet Ltd. | https://www.aptanet.com/ | 023 9238
0001
Vice Chair, FSB Portsmouth & SE Hampshire Branch |
http://www.fsb.org.uk/
============================================================
=================
Registered in England | Company No: 4905028 | Registered Office: Ralls
House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7
6XP
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
Andy Smith <andy(a)bitfolk.com> said, in
message
20161214124152.GU21587(a)bitfolk.com:
>
…perhaps I could have a bit of feedback from you as to whether we
> did the right thing in enforcing a reboot
here.
I'm all for these short notice reboots[*]. Partly because of the
rather-safe-than-sorry scenario expressed by Alun. Partly because it's
2016, and we really should know how to build our virtual infrastructure
in such a way that it survives the occasional reboots. Especially if
they are advised a head of time.
// Andreas 
14 Dec
14 Dec
3:22 p.m.
On 14/12/2016 14:41, Andy Smith wrote:
So, did we do the right thing in patching and
rebooting this before
the embargo date (leading to less than 2 weeks of notice of a reboot
for customers)?
Yes you did.
My reboot notice is 3 days because I live in a cave and it takes me 2
days to walk back to civilization and refill my supplies once a month.
--GM
3:35 p.m.
On Wed, Dec 14, 2016 at 03:34:13PM +0000, Mike Zanker wrote:
On 14 Dec 2016, at 12:41, Andy Smith
<andy(a)bitfolk.com> wrote:
I'm in favour of patching.
Michael
So, did we do the right thing in patching and
rebooting this before
the embargo date (leading to less than 2 weeks of notice of a reboot
for customers)?
As far as I am concerned, yes. 
8:05 p.m.
On 14/12/2016 12:41, Andy Smith wrote:
So, did we do the right thing in patching and
rebooting this before
the embargo date (leading to less than 2 weeks of notice of a reboot
for customers)?
Should we have patched and rebooted but over a longer schedule?
Should we have not bothered with a reboot at all?
I think it was a good move. While the risk may be lower than with the
previous patches, I'd rather a small amount of downtime than my VM
getting compromised by something that I have no control over. You do it
at a time of the night when it is likely to be of least inconvenience,
and you give us as much notice as you can, and I can't ask for more than
that.
Gavin
15 Dec
15 Dec
9:49 a.m.
New subject: Upcoming reboots for security fixes prior to 13 December
My 5p is very much in line with the list consensus so far.
Any issue that concerns memory or disk access outside of bounds is a
very good reason for patching ASAP.
And it is not like it is you fault Andy. To me, the fact you do patch
the servers as soon as possible says to me that Bitfolk takes security
seriously. So it is not an inconvenience as far as I am concerned,
rather a reason to trust and recommend Bitfolk.
If anyone needed better availability than what Bitfolk has to offer,
they should be prepared to pay quite a bit more. Bitfolk is not a
HA-service provider. Even if it almost feels like that some times! ;-)
Have been using Bitfolk for quite some time now and I am very pleased
as a customer with the availability and service. A lot of other
providers offers worse service for a lot of more money.
Thanks,
__
/ony
-------
Wednesday, December 14, 2016, 12:41:52 PM, Andy wrote:
On Tue, Dec 13, 2016 at 03:17:53AM +0000, Andy Smith
wrote:
> Hi,
>
> On Wed, Nov 30, 2016 at 06:00:35AM +0000, Andy Smith wrote:
> > Rather disappointingly there's been another round of security
> > advisories for Xen, some of which affect the configuration in use at
> > BitFolk:
>
> The maintenance work to patch against this bug (XSA-200) has now
> been completed, without incident.
Now that this is out of embargo:
…perhaps I could have a bit of feedback from you as to
whether we
did the right thing in enforcing a reboot here.
Discussion around the bug (unfortunately on a private
list for
discussion of the security bugs while they're under embargo, so I
can't show you) indicated that it *probably* wasn't very dangerous.
The impact is basically that a guest VM could read
either 32 or 96
bits of hypervisor memory. It would probably be a different 96 bits
each time, but probably not anything private.
Of course, the Xen folks are not being paid to
speculate on exactly
how exploitable that bug is, so they can't be drawn into making any
statements about how likely it is that any sensitive data could be
read. Their official analysis quite reasonably stops at "guests can
read some hypervisor memory; here is the patch for that".
As the matter was under embargo it was difficult to
discuss any
further than that, hence all I've got is "it probably isn't that
serious". That was based on some insights that it seems similar to
an earlier bug that also probably wasn't as serious as its
corresponding advisory implied - bearing in mind that an advisory
should tell you the worst case. Analysis of a similar bug:
That one involved the ability to read a couple of kB
of hypervisor
memory, whereas XSA-200 was limited to 96 bits. So XSA-200 is almost
likely less serious than XSA-108 was.
So, did we do the right thing in patching and
rebooting this before
the embargo date (leading to less than 2 weeks of notice of a reboot
for customers)?
Should we have patched and rebooted but over a longer
schedule?
Should we have not bothered with a reboot at all?
Cheers,
Andy
2689
days inactive
2704
days old
17 comments
14 participants
participants (14)
-
Alun -
Andreas Olsson -
Andy Smith -
Brendan -
G. Miliotis -
Gavin Westwood -
Leo Leibovici -
Michael Stevens -
Mike Zanker -
Murray Crane -
Paul Tansom -
Robert Gauld -
Roger Light -
Tony Andersson