Re: [bitfolk] Host "hen" crashed again (Was: Re: Host "hen" unexpectedly rebooted 2018-11-26 22:24)

This time I was logging the serial console to a file and so am able to see that there was the equivalent of a kernel panic in the hypervisor. That is, I do not believe that hen's hardware is at fault. I think it's tripping against a bug in Xen, and it's happened to the same host twice because it's been triggered by the same guest doing something (I do not believe malicious at this stage). I've not got a quick fix to this because moving all customers on hen to new hardware is likely just going to crash the hypervisor on the other hardware. I need to discuss the problem with the Xen developers and see if I get anywhere. In between last time and this I also built a new version of the hypervisor and set every host to boot into it, so hen is now actually running a very slightly newer version than everything else (and also compared to what it was running before). This possibly could help, just by chance, though as far as I am aware it is not a known bug. So I am very sorry but I am going to have to ask you to bear with me for a little while, while I investigate this more. Until I can establish which guest triggered it I can't move any of the customers on host hen to other hosts because that possibly just triggers it elsewhere. And it could still elsewhere anyway. If I don't make headway with this then I can revert to earlier versions that we've been stable on for a long time, but security issues have been fixed since then so I'm not going to do that except as a last resort. I will provide more information as soon as I can. Thanks, Andy