23 Oct
2020
23 Oct
'20
10:22 a.m.
Hi,
Server "elephant" unexpectedly crashed, then crashed twice more
shortly after rebooting but before completely starting all VPSes. It
is now crashing every time while trying to boot VPSes. I suspected
bug in last round of XSA patches so reverted to previous hypervisor,
but problem persists. We had an issue with "elephant" not so long
ago so it might be hardware fault *though no logs to back this up).
Still investigating, sorry.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
23 Oct
23 Oct
11:19 a.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
On Fri, Oct 23, 2020 at 10:22:32AM +0000, Andy Smith wrote:
It is now crashing every time while trying to boot
VPSes. I
suspected bug in last round of XSA patches so reverted to previous
hypervisor, but problem persists. We had an issue with "elephant"
not so long ago so it might be hardware fault *though no logs to
back this up).
Still investigating, sorry.
At the moment its kernel is crashing some way towards the end of
starting up all VMs. I'm trying to isolate the issue to one
particular VM because if a guest can crash the host then it's a bug
in the hypervisor and just moving guests around won't solve the
problem.
It would be helpful if people did not keep logging in and restarting
their VPSes while I am doing this. It's not going to help you much
as the host is almost certainly going to crash or be rebooted soon.
Seems I need to toggle off the ability for people to log in, or at
least to start their guests, while under maintenance.
Cheers,
Andy
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
11:26 a.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
On Fri, Oct 23, 2020 at 11:19:21AM +0000, Andy Smith wrote:
On Fri, Oct 23, 2020 at 10:22:32AM +0000, Andy Smith
wrote:
I resemble that remark. Sorry.
My mail runs through the machine, though, so I had to boot it to
read this message. :)
Thanks for the info,
Hugo.
--
Hugo Mills | Putting U back in Honor, Valor, and Trth.
hugo@... carfax.org.uk |
http://carfax.org.uk/ |
PGP: E2AB1DE4 |
It is now crashing every time while trying to
boot VPSes. I
suspected bug in last round of XSA patches so reverted to previous
hypervisor, but problem persists. We had an issue with "elephant"
not so long ago so it might be hardware fault *though no logs to
back this up).
Still investigating, sorry.
At the moment its kernel is crashing some way towards the end of
starting up all VMs. I'm trying to isolate the issue to one
particular VM because if a guest can crash the host then it's a bug
in the hypervisor and just moving guests around won't solve the
problem.
It would be helpful if people did not keep logging in and restarting
their VPSes while I am doing this. It's not going to help you much
as the host is almost certainly going to crash or be rebooted soon.
Seems I need to toggle off the ability for people to log in, or at
least to start their guests, while under maintenance.
11:46 a.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
On Fri, Oct 23, 2020 at 11:19:21AM +0000, Andy Smith wrote:
I'm trying to isolate the issue to one particular
VM because if a
guest can crash the host then it's a bug in the hypervisor and
just moving guests around won't solve the problem.
I can't find it. As we have had problems with elephant before I'm
going to assume hardware problem and start moving customer VMs to
other hosts. This may take quite some time and I'm sorry for the
disruption.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
11:51 a.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
I was wondering what it was :)
Thought it was me again!
Thanks for the update
On 23/10/2020 12:46, Andy Smith wrote:
On Fri, Oct 23, 2020 at 11:19:21AM +0000, Andy Smith
wrote:
I'm trying to isolate the issue to one
particular VM because if a
guest can crash the host then it's a bug in the hypervisor and
just moving guests around won't solve the problem.
I can't find it. As we have had problems with elephant before I'm
going to assume hardware problem and start moving customer VMs to
other hosts. This may take quite some time and I'm sorry for the
disruption.
Cheers,
Andy
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
4:37 p.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
Hi,
On Fri, Oct 23, 2020 at 11:46:11AM +0000, Andy Smith wrote:
On Fri, Oct 23, 2020 at 11:19:21AM +0000, Andy Smith
wrote:
While moving customer VMs to other hosts, booting one of them caused
server "macallan" to crash in exactly the same way. So, I am ruling
out hardware issues with "elephant".
By preventing this particular VM from booting I was able to boot all
of the other VMs on "macallan". I have some hope that it is just
this one VM that is tickling a particularly nasty bug.
I am going to try now starting the remainder of VMs on "elephant".
If that is successful I will then take the suspect VM to test
hardware to see if I can further reproduce.
I am confused because I am sure I tried reverting last weekend's
hypervisor upgrade to the previous version while investigating
matters on "elephant", yet it still crashed. Possibly I made a
mistake (e.g. booted with wrong hypervisor).
Also, everything obviously booted up fine last weekend when I did
the maintenance so possibly this customer has found a new and
unrelated bug.
The best case at this point is that I can reproduce the problem with
just that one VM, report it, get it fixed and then have to reboot
everything to deploy the fix.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
I'm trying to isolate the issue to one
particular VM because if a
guest can crash the host then it's a bug in the hypervisor and
just moving guests around won't solve the problem.
I can't find it. As we have had problems with elephant before I'm
going to assume hardware problem and start moving customer VMs to
other hosts. 
24 Oct
24 Oct
5:06 p.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
On Fri, 2020-10-23 at 16:37 +0000, Andy Smith wrote:
Hi,
On Fri, Oct 23, 2020 at 11:46:11AM +0000, Andy Smith wrote:
Interesting - could you share the kernel panic or does it contain too
much sensitive information?
Conrad
On Fri, Oct 23, 2020 at 11:19:21AM +0000, Andy
Smith wrote:
While moving customer VMs to other hosts, booting one of them caused
server "macallan" to crash in exactly the same way. So, I am ruling
out hardware issues with "elephant".
By preventing this particular VM from booting I was able to boot all
of the other VMs on "macallan". I have some hope that it is just
this one VM that is tickling a particularly nasty bug.
I am going to try now starting the remainder of VMs on "elephant".
If that is successful I will then take the suspect VM to test
hardware to see if I can further reproduce.
I am confused because I am sure I tried reverting last weekend's
hypervisor upgrade to the previous version while investigating
matters on "elephant", yet it still crashed. Possibly I made a
mistake (e.g. booted with wrong hypervisor).
Also, everything obviously booted up fine last weekend when I did
the maintenance so possibly this customer has found a new and
unrelated bug.
The best case at this point is that I can reproduce the problem with
just that one VM, report it, get it fixed and then have to reboot
everything to deploy the fix.
Cheers,
Andy I'm trying to isolate the issue to one
particular VM because if a
guest can crash the host then it's a bug in the hypervisor and
just moving guests around won't solve the problem.
I can't find it. As we have had problems with elephant before I'm
going to assume hardware problem and start moving customer VMs to
other hosts.
6:50 p.m.
New subject: Server "elephant" has crashed a few times, ongoing problems
Hi Conrad,
On Sat, Oct 24, 2020 at 06:06:17PM +0100, Conrad Wood wrote:
Interesting - could you share the kernel panic or does
it contain too
much sensitive information?
I don't think it's going to work out to be very interesting after
all. A newer kernel avoids the issue, not even a very much newer
kernel. Time can be spent bisecting this bug but ultimately it's
going to be a fixed bug, not anything new.
It doesn't actually look Xen related and seems it was only a
coincidence that it first happened about 7 days after deploying Xen
patches.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
1277
days inactive
1278
days old
7 comments
4 participants
participants (4)
-
Andy Smith -
Conrad Wood -
Hugo Mills -
Stuart Sears