Hi, On Tue, Apr 27, 2021 at 09:39:25PM +0000, Andy Smith wrote: The work last night mostly went well. The timeline was something like: 22:04Z Customers on "jack" start to have their guests shut down/suspended 22:17Z Customers on "jack" start to have their guests booted/restored again 22:20Z Shut down / suspend for "paradox" 22:40Z Boot / restore for "paradox" 22:40Z Shut down / suspend for "hobgoblin" 22:57Z Boot / restore for "hobgoblin" 22:58Z Shut down / suspend for "macallan" 23:38Z Boot / restore for "macallan" 23:55Z Shut down / suspend for "clockwork" 00:12Z Boot / restore for "clockwork" So, on the whole about 15 to 20 minutes per machine. Things went wrong with "macallan" in that grub failed to run at boot, leading to some amount of panic and an extra 20 minutes or so of downtime. The problem was this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984426 It was solved by: 1. PXE boot in to Debian installer 2. Choose "Rescue" mode 3. Go through the setup process then select the "Reinstall Grub Bootloader" option 4. Reboot as normal I don't yet know why it affected only that server. There was a 4 hour window for the maintenance but it was all done after about 2h15m (~00:15Z). Thanks for your patience! Another round of similar work is planned for on or about 27 May. I'm afraid we haven't managed to verify that date yet, so it might end up being slightly different but not before then. That work will involve the remaining four servers: "elephant", "limoncello", "snaps" and "talisker". More formal notifications will go out when the date is confirmed. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting