26 Nov
2018
26 Nov
'18
10:41 p.m.
Hi,
At approximately 22:24Z, host "hen" rebooted itself unexpectedly. It
all appears to be back up again now, and I am investigating to try
to determine the cause.
Apologies for the disruption.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
10 Dec
10 Dec
3:25 p.m.
New subject: Host "hen" crashed again (Was: Re: Host "hen" unexpectedly rebooted 2018-11-26 22:24)
Hi,
On Mon, Nov 26, 2018 at 10:41:43PM +0000, Andy Smith wrote:
At approximately 22:24Z, host "hen" rebooted
itself unexpectedly.
This unfortunately has happened again today, at about 14:23Z.
This time I was logging the serial console to a file and so am able
to see that there was the equivalent of a kernel panic in the
hypervisor.
That is, I do not believe that hen's hardware is at fault. I think
it's tripping against a bug in Xen, and it's happened to the same
host twice because it's been triggered by the same guest doing
something (I do not believe malicious at this stage).
I've not got a quick fix to this because moving all customers on hen
to new hardware is likely just going to crash the hypervisor on the
other hardware. I need to discuss the problem with the Xen
developers and see if I get anywhere.
In between last time and this I also built a new version of the
hypervisor and set every host to boot into it, so hen is now
actually running a very slightly newer version than everything else
(and also compared to what it was running before). This possibly
could help, just by chance, though as far as I am aware it is not a
known bug.
So I am very sorry but I am going to have to ask you to bear with me
for a little while, while I investigate this more. Until I can
establish which guest triggered it I can't move any of the customers
on host hen to other hosts because that possibly just triggers it
elsewhere. And it could still elsewhere anyway.
If I don't make headway with this then I can revert to earlier
versions that we've been stable on for a long time, but security
issues have been fixed since then so I'm not going to do that except
as a last resort.
I will provide more information as soon as I can.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
3:30 p.m.
New subject: Host "hen" crashed again (Was: Re: Host "hen" unexpectedly rebooted 2018-11-26 22:24)
This unfortunately has happened again today, at about 14:23Z.
This time I was logging the serial console to a file
and so am able
to see that there was the equivalent of a kernel panic in the
hypervisor.
That is, I do not believe that hen's hardware is at fault. I think
it's tripping against a bug in Xen, and it's happened to the same
host twice because it's been triggered by the same guest doing
something (I do not believe malicious at this stage).
I've not got a quick fix to this because moving all customers on hen
to new hardware is likely just going to crash the hypervisor on the
other hardware. I need to discuss the problem with the Xen
developers and see if I get anywhere.
In between last time and this I also built a new version of the
hypervisor and set every host to boot into it, so hen is now
actually running a very slightly newer version than everything else
(and also compared to what it was running before). This possibly
could help, just by chance, though as far as I am aware it is not a
known bug.
So I am very sorry but I am going to have to ask you to bear with me
for a little while, while I investigate this more. Until I can
establish which guest triggered it I can't move any of the customers
on host hen to other hosts because that possibly just triggers it
elsewhere. And it could still elsewhere anyway.
If I don't make headway with this then I can revert to earlier
versions that we've been stable on for a long time, but security
issues have been fixed since then so I'm not going to do that except
as a last resort.
I will provide more information as soon as I can.
Thanks,
Andy
If you have a spare HV, why not try to identify the guest involved by
moving half the guests off of hen? The HV that crashes has the errant
guest, move another half from that HV, see which HV crashes. Continue
till you've identified, or you've a small enough number to be worth
contacting users and asking what their guests are doing at the panic time?
16 Dec
16 Dec
12:23 p.m.
New subject: Host "hen" crashed again (Was: Re: Host "hen" unexpectedly rebooted 2018-11-26 22:24)
Hi Johnathon,
On Mon, Dec 10, 2018 at 03:30:48PM +0000, Johnathon Tinsley wrote:
If you have a spare HV, why not try to identify the
guest involved by moving
half the guests off of hen? The HV that crashes has the errant guest, move
another half from that HV, see which HV crashes. Continue till you've
identified, or you've a small enough number to be worth contacting users and
asking what their guests are doing at the panic time?
Okay, so, I do know which guest was executing at the time of the
last crash. There could easily be more than one though, and moving
guests about imposes a reboot on them anyway.
I'm afraid I want to treat customers on hen as guinea pigs at least
one more time for the purposes of collecting information.
It will be interesting to see if the same guest is implicated next
time, in which case certainly I will isolate it then.
Also I've been suggested a setting which may avoid the area of
code where the bug is thought to lie, but it will require a reboot
to apply it. If/when there is another crash, it will come back up
with that setting applied and then we will go from there.
So, there is likely to be 1 more crash at some point, after which I
hope things settle done but if not I will take more extreme steps
such as this.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
21 Dec
21 Dec
2:49 p.m.
New subject: Host "hen" crashed again (Was: Re: Host "hen" unexpectedly rebooted 2018-11-26 22:24)
Hi,
This happened again a few minutes ago.
IPv6 failover appears to have happened correctly due to some fixes
since last time - hence no v6 outage until manual intervention.
hen has now booted with some settings which it is hoped will avoid
the problem in future.
I'm not in front of a computer at the moment so later today I'll
review what was logged.
Apologies for the ongoing hassle.
Cheers,
Andy
1954
days inactive
1979
days old
4 comments
2 participants
participants (2)
-
Andy Smith -
Johnathon Tinsley