[bitfolk] Unscheduled reboot / approximately 1 hour outage for customers on dunkel.bitfolk.com

Hi, At approximately 1106Z today, suite TFM1 in Telehouse lost power for around 2 minutes. The only live BitFolk server in TFM1 at present is dunkel. Servers are set to not automatically start every VPS, as if they have been rebooted it is usually for maintenance purposes. What should have happened is that I should have been alerted, and then I would have sorted things out. A significant part of the internal monitoring is currently running on a VPS on dunkel, but there is external monitoring for situations such as this. However, this has not worked and it wasn't until I received a phone call on the urgent support number that I was aware of the problem. As I type this, all VPSes on dunkel have now been restarted -- some may still be fscking. I am waiting on Telehouse for more information on the power outage but obviously the real failing here is one of BitFolk's monitoring. This is the second time there has been a monitoring failure; the system had actually been changed since the first time, although that was a different class of failure. I will be looking into why the external monitoring failed, and changing how it's done if necessary. In the meantime I will be adding a day of free service to all those on dunkel who were affected. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting "I'd be happy to buy all variations of sex to ensure I got what I wanted." -- Gary Coates (talking about cabling)