[bitfolk] Outage / unexpected shutdown+boot for VPSes on host "snaps", 2017-12-28 ~08:06Z – ~08:40Z

Hi, Around 08:06Z today we received an alert regarding host snaps.bitfolk.com. I found it completely unresponsive over network, but was still able to connect to its console. Despite it believing its network interfaces were up and had link, it was passing no traffic to the colo switches. I spent about 30 minutes trying to diagnose this and not getting anywhere, so decided to try rebooting it. As I had console access I was able to cleanly shut down all VPSes on snaps first. The shutdown and boot went without incident and things seemed fine on boot. By about 08:40Z all VPSes that should be running had been started, and by now Nagios is clear of alerts¹. I am aware that snaps had an unexplained outage a few months ago, on 28 September. This time the symptoms are not the same, other than that the problem is unexplained and clears after a reboot. Clearly there is something wrong there though and it's going to happen again, so over the next few days we will be moving customers off of snaps. We will co-ordinate this directly with customers involved. Apologies for the disruption, Andy BitFolk Ltd ¹ Except for one customer web server which is waiting for a TLS passphrase to be supplied before it will start. -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce