Hi,
After receiving a number of alerts for VMs hosted on server "jack",
I investigated and found the server largely unresponsive.
Unfortunately I had no option but to forcibly reboot it, which I did
at about 06:47Z
It's now 07:01Z and monitoring says everything is back up, except
for two customer VMs which are waiting for a LUKS passphrase on
their console.
This problem was the same as what was experienced with some of the
other servers a few months ago. With the months-long gap I had hoped
it was some undiagnosed kernel issue which we had got past, but
apparently not, as "jack" is on the latest available kernel package.
I'm pursuing some ideas about a config change that may help, and I
managed to put that into place before "jack" was rebooted - it does
require a reboot so if it does help it won't be able to take effect
on the others until next reboot. On the other hand it doesn't hurt
either, so I've made the same change elsewhere also.
If that doesn't fix things then the next line of investigation will
be an upgrade of the hypervisor to latest stable release, though
that is a rather major undertaking.
Apologies for the disruption. It is challenging to debug a problem
that can take several months to occur, with no reliable way of
triggering it. :(
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
We don't have 100% firm date for this yet and I don't want to
announce it properly while the other moves are happening but I also
want to try to give a month of notice: The servers that are not
getting relocated today & 27 April will very likely be relocated on
27 May.
That is:
- elephant
- limoncello
- snaps
- talisker
Now because of the movement of individual customers that is still
taking place, it is very likely that everyone who's currently on
"snaps" will be moved off of it before then, in which case we will
relocate it at a time convenient to us as it won't affect anyone.
The other three have already had an OS upgrade so they definitely
will be relocated all together on the same night, which as I say is
very likely to be 27 May.
As before I will send another mail here when we have confirmation
and then we will send a direct email a week before to everyone who
will be affected. And as before it will be possible to have your
service moved about ahead of time at a time of your choosing.
While I'm here a reminder that:
- Server "hen" is being relocated tonight at some point between
21:00 and 23:00 BST (20:00 to 22:00 UTC). Those affected would
have received individual notifications about that a week ago.
- Servers "clockwork", "hobgoblin", "jack", "leffe", "macallan" and
"paradox" will be relocated on 27 April and later today all those
who will be affected will receive a direct email about this.
The full details of those works are at:
https://tools.bitfolk.com/wiki/Maintenance/2021-04-Re-racking
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
