5 Jan
2014
5 Jan
'14
7:52 p.m.
Hi,
Unfortunately due to what looks like a kernel bug, sol.bitfolk.com
has got stuck in a state where VPSes cannot be booted. I therefore
need to reboot it as a matter of urgency, so those with VPSes hosted
on sol are shortly going to see a clean shutdown and boot.
I apologise for the disruption and I hope to be able to give more
information later. I will follow up again when all customr VPSes
are known to have booted.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
5 Jan
5 Jan
9:41 p.m.
Hi,
On Sun, Jan 05, 2014 at 07:52:32PM +0000, Andy Smith wrote:
I apologise for the disruption and I hope to be able
to give more
information later. I will follow up again when all customr VPSes
are known to have booted.
All customers VPSes are believed to have booted as of about 2050Z.
If yours hasn't, please check out its console. Our Nagios thinks
that everything that was up before is back up again now.
What appears to have happened is that a customer VPS earlier this
afternoon was rebooted while under extreme memory pressure (it was
OOM-killing a lot) and the slow shutdown of that appears to have hit
a race condition in the host kernel which lead to the xenwatch
kernel thread being left in an uninterruptible 'D' state.
In that state, the host was unable to create any further virtual
network devices, so this customer could not complete their reboot
nor could they launch the rescue environment. As no network devices
could be created, no VPS could in fact be started which is why a
full reboot of the host was necessary.
This appears to be a known bug and there is probably a fix for it
that we can apply, but I did not want to do that in the middle of
this semi-emergency reboot.
This kernel version is in use on several hosts with combined uptimes
of thousands of days so I do not think this is a commonly-hit bug
and I would rather take a little bit of time to research the fix and
then schedule a reboot for kernel upgrade with plenty of notice.
I will keep you informed about that.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
9:58 p.m.
This sort of thing is why I love bitfolk. You are so well on top of
every little problem that occurs.
Keep on keeping on!
James
On Sun, 5 Jan 2014 21:41:12 +0000
Andy Smith <andy(a)bitfolk.com> wrote:
Hi,
On Sun, Jan 05, 2014 at 07:52:32PM +0000, Andy Smith wrote:
I apologise for the disruption and I hope to be
able to give more
information later. I will follow up again when all customr VPSes
are known to have booted.
All customers VPSes are believed to have booted as of about 2050Z.
If yours hasn't, please check out its console. Our Nagios thinks
that everything that was up before is back up again now.
What appears to have happened is that a customer VPS earlier this
afternoon was rebooted while under extreme memory pressure (it was
OOM-killing a lot) and the slow shutdown of that appears to have hit
a race condition in the host kernel which lead to the xenwatch
kernel thread being left in an uninterruptible 'D' state.
In that state, the host was unable to create any further virtual
network devices, so this customer could not complete their reboot
nor could they launch the rescue environment. As no network devices
could be created, no VPS could in fact be started which is why a
full reboot of the host was necessary.
This appears to be a known bug and there is probably a fix for it
that we can apply, but I did not want to do that in the middle of
this semi-emergency reboot.
This kernel version is in use on several hosts with combined uptimes
of thousands of days so I do not think this is a commonly-hit bug
and I would rather take a little bit of time to research the fix and
then schedule a reboot for kernel upgrade with plenty of notice.
I will keep you informed about that.
Cheers,
Andy
6 Jan
6 Jan
10:31 a.m.
On 05/01/14 21:58, James Stanley wrote:
[re: Andy's full disclosure of problem, solution, prognosis, etc.]
This sort of thing is why I love bitfolk.
+1
So many organisations and individuals try to sweep things under the
carpet and hope that nobody notices. Even the best of 'em would
only go so far as to inform only the affected customers.
This sort of report actually gives me *greater* confidence in the
reliability of the systems.
<tangent>
I always advise clients to steer clear of any hosting firm offering
"100% guaranteed uptime".
</>
11:12 a.m.
+10 to that.
I am a newish customer - less than a year - but from experience and what I have seen so
far agree entirely. I am very pleased with the excellent communications and service.
Thanks to Andy and all concerned.
Richard.
Dom Latter <bitfolk-users(a)latter.org> wrote:
On 05/01/14 21:58, James Stanley wrote:
[re: Andy's full disclosure of problem, solution, prognosis, etc.]
This sort of thing is why I love bitfolk.
+1
So many organisations and individuals try to sweep things under the
carpet and hope that nobody notices. Even the best of 'em would
only go so far as to inform only the affected customers.
This sort of report actually gives me *greater* confidence in the
reliability of the systems.
<tangent>
I always advise clients to steer clear of any hosting firm offering
"100% guaranteed uptime".
</>
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users 
2:02 p.m.
Dear All,
long votes_for_andy_counter = 11;// see below 10 + 1
votes_for_andy_counter++; // Sams vote
Cheers,
Sam
On 2014-01-06 12:12, Richard Glynos wrote:
+10 to that.
snip
Richard.
Dom Latter <bitfolk-users(a)latter.org> wrote:
> On 05/01/14 21:58, James Stanley wrote:
>
> [re: Andy's full disclosure of problem, solution, prognosis, etc.]
>
>> This sort of thing is why I love bitfolk.
>
> +1
>
> snip
2:18 p.m.
Hi,
long votes_for_andy_counter = 11;// see below 10 + 1
Please don't spoil the concurrency semantics of the current system.
Thanks.
x
votes_for_andy_counter++; // Sams vote
Regards,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
7:32 p.m.
I've been working with Andy (as a vendor and as a customer, sometimes both) since
~2005, and this level of excellence is very much the norm with him. :)
-Graham
On 06 Jan 02014, at 03:12 UTC-0800, Richard Glynos <richardmglynos(a)gmail.com>
wrote:
+10 to that.
I am a newish customer - less than a year - but from experience and what I have seen so
far agree entirely. I am very pleased with the excellent communications and service.
Thanks to Andy and all concerned.
Richard.
Dom Latter <bitfolk-users(a)latter.org> wrote:
On 05/01/14 21:58, James Stanley wrote:
[re: Andy's full disclosure of problem, solution, prognosis, etc.]
This sort of thing is why I love bitfolk.
+1
So many organisations and individuals try to sweep things under the
carpet and hope that nobody notices. Even the best of 'em would
only go so far as to inform only the affected customers.
This sort of report actually gives me *greater* confidence in the
reliability of the systems.
<tangent>
I always advise clients to steer clear of any hosting firm offering
"100% guaranteed uptime".
</>
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
3760
days inactive
3761
days old
7 comments
7 participants
participants (7)
-
Andy Bennett -
Andy Smith -
baechler＠boeser.ch -
Dom Latter -
Graham Freeman -
James Stanley -
Richard Glynos