25 Aug
2016
25 Aug
'16
9:11 p.m.
Hi,
By now you should have all received notification of the scheduled
maintenance that will be taking place in the early hours of the
morning (UK time) on 2016-09-02, 03 and 05.
This is the result of an embargoed security update that we have been
made aware of today.
If you have not seen the notification which was sent directly to the
email address we have for you at:
https://panel.bitfolk.com/account/contacts/
please first check your spam folders, etc., and failing that please
do let us know.
Also if you have any questions I am happy to answer these either on
the users mailing list or directly in a private ticket at
support(a)bitfolk.com.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
4 Sep
4 Sep
2:41 a.m.
Hi,
On Thu, Aug 25, 2016 at 09:11:51PM +0000, Andy Smith wrote:
By now you should have all received notification of
the scheduled
maintenance that will be taking place in the early hours of the
morning (UK time) on 2016-09-02, 03 and 05.
I realise I did not follow up on how the work went on 2016-09-02 and
today. Particularly important since there was a mistake made on
2016-09-02.
During the reboot of the first host, "jack", on 2016-09-02 I
attempted to be clever and stab the "Enter" key during the grub
prompt to shave off 5 seconds and have it boot the correct kernel
immediately. Unfortunately I missed the correct key and caused the
cursor to move, selecting an older, incorrect kernel.
As the machines boot quite fast with SSDs now, by the time I
realised what had happened it was already at the point where it was
going to start mounting filesystems and I judged that it would be a
bad idea to attempt to reboot or power off at that stage. I decided
to let it boot and then reboot it again.
This regrettably meant an additional boot/shutdown cycle was seen by
some customers with VPSes on "jack" and a couple of minutes were
added to the outage length.
In future I will not interact with grub unless it is necessary. The
correct kernel to boot is already set from userland with
grub-editenv.
The work for the remaining hosts that day ("macallan" and "leffe")
as well as work for "hen" and "hobgoblin" today went without
incident.
Remaining work for "paradox" and "snaps" is taking place tomorrow,
2016-09-05, as scheduled.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
5 Sep
5 Sep
3:15 a.m.
Hi,
On Thu, Aug 25, 2016 at 09:11:51PM +0000, Andy Smith wrote:
By now you should have all received notification of
the scheduled
maintenance that will be taking place in the early hours of the
morning (UK time) on 2016-09-02, 03 and 05.
Today's maintenance is now completed and went without incident.
Thanks for your patience in this matter. It is unfortunate that we
have had two serious security issues come to light within 6 weeks of
each other.
As mentioned in previous similar events, when we have to reboot a
host we prefer to have all customer VPSes shut down and then boot
again.
Xen and Linux do contain support for suspend and restore, so in
theory it would be possible to suspend customer VPSes to storage and
then restore them again when the host has been booted. To the VPS
and processes inside it this would look like time stood still.
Network connections would most likely drop.
Although this is much less disruptive than a shutdown and boot, it
doesn't always work. Some older Linux kernels don't restore
correctly, leaving the VPS in a hung state (requires destroy and
boot again).
Some applications get massively confused by the clock getting stuck.
I have experienced that with pacemaker (clustering software); I
could not get the restored node to rejoin the cluster without
stopping the cluster on every node and starting it again, which was
much more more disruptive than just rebooting the VPS would have
been, given it was in an HA cluster anyway!
So, we have not been suspending any customer VPSes. I have however
been suspending BitFolk's own infrastructure VMs each time, mainly
to see how well it works. I've generally found that for any given
VM it either works every time or it works none of the time. For most
VMs it works. In the cases I know it won't work or is otherwise
undesirable, I don't suspend them.
If you'd be interested in being able to have your VPS suspended, I
would encourage you to vote¹ for this feature:
https://tools.bitfolk.com/redmine/issues/32
If there's enough interest then we will expose an option that
indicates to us that your VPS should be suspended. You would still
need to check that it does actually work for you, i.e. the default
would have to remain as "shutdown, don't suspend" for now.
Cheers,
Andy
¹ In order to vote for a feature you will need to be logged in. The
credentials are you usual BitFolk ones. Once logged in, up and
down arrows should appear next to the number of votes.
--
http://bitfolk.com/ -- No-nonsense VPS hosting
The optimum programming team size is 1.
Has
Jurassic Park taught us nothing? — pfilandr
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
10:10 a.m.
On 5 September 2016 at 04:15, Andy Smith <andy(a)bitfolk.com> wrote:
On Thu, Aug 25, 2016 at 09:11:51PM +0000, Andy Smith
wrote:
This sounds like it *might* be a Pacemaker bug, or (more likely)
potentially a worthwhile enhancement; I suggest that you send a mail
to the mailing list describing the use case:
http://clusterlabs.org/mailman/listinfo/users
They might say "sorry, we can't support that", or who knows, they
might say "nice idea, thanks!" Either way, I know several of the
Pacemaker folks and can promise that they are super friendly and
helpful :-)
By now you should have all received notification
of the scheduled
maintenance that will be taking place in the early hours of the
morning (UK time) on 2016-09-02, 03 and 05.
Today's maintenance is now completed and went without incident.
Thanks for your patience in this matter. It is unfortunate that we
have had two serious security issues come to light within 6 weeks of
each other.
As mentioned in previous similar events, when we have to reboot a
host we prefer to have all customer VPSes shut down and then boot
again.
Xen and Linux do contain support for suspend and restore, so in
theory it would be possible to suspend customer VPSes to storage and
then restore them again when the host has been booted. To the VPS
and processes inside it this would look like time stood still.
Network connections would most likely drop.
Although this is much less disruptive than a shutdown and boot, it
doesn't always work. Some older Linux kernels don't restore
correctly, leaving the VPS in a hung state (requires destroy and
boot again).
Some applications get massively confused by the clock getting stuck.
I have experienced that with pacemaker (clustering software); I
could not get the restored node to rejoin the cluster without
stopping the cluster on every node and starting it again, which was
much more more disruptive than just rebooting the VPS would have
been, given it was in an HA cluster anyway!
9:21 p.m.
Hi Adam,
On Mon, Sep 05, 2016 at 11:10:30AM +0100, Adam Spiers wrote:
On 5 September 2016 at 04:15, Andy Smith
<andy(a)bitfolk.com> wrote:
Admittedly this is an old cluster running pacemaker+heartbeat so
first I must upgrade it to pacemaker+corosync. Quite possibly it
will work then, anyway. :)
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Some applications get massively confused by the
clock getting stuck.
I have experienced that with pacemaker (clustering software); I
could not get the restored node to rejoin the cluster without
stopping the cluster on every node and starting it again,
This sounds like it *might* be a Pacemaker bug, or (more likely)
potentially a worthwhile enhancement;
11:29 p.m.
On 5 September 2016 at 22:21, Andy Smith <andy(a)bitfolk.com> wrote:
On Mon, Sep 05, 2016 at 11:10:30AM +0100, Adam Spiers
wrote:
Oh wow, yeah - that's ancient ;-)
On 5 September 2016 at 04:15, Andy Smith
<andy(a)bitfolk.com> wrote:
Admittedly this is an old cluster running pacemaker+heartbeat so
first I must upgrade it to pacemaker+corosync. Quite possibly it
will work then, anyway. :) Some applications get massively confused by the
clock getting stuck.
I have experienced that with pacemaker (clustering software); I
could not get the restored node to rejoin the cluster without
stopping the cluster on every node and starting it again,
This sounds like it *might* be a Pacemaker bug, or (more likely)
potentially a worthwhile enhancement;
2789
days inactive
2800
days old
5 comments
2 participants
participants (2)
-
Adam Spiers -
Andy Smith