[bitfolk] 2010-10-22 2136Z: Switch power failure caused ~10 minute networking outage for barbar, dunkel, kwak

Hi folks, At approximately 2136Z this evening, unrelated provider maintenance work in the colo dislodged a power cord to one of the switches, causing approximately 10 minutes of network outage for customers on the following servers: barbar dunkel kwak Host corona in the same rack was unaffected because it is connected to a different switch. There are two switches in each rack in order to provide redundancy, but unfortunately it has only recently become part of our standard install procedure to use bonded network interfaces. kwak and corona are due for an OS reinstall and when this is done they will be set up with bonded interfaces. The same goes for these older hosts in the other racks: curacao kahlua obstler dunkel and barbar will not require a reinstall for quite some time, but this is not the first time that switch outages have bitten us and I'm not happy with that, so I will be scheduling a short maintenance in the next couple of months to reconfigure these hosts for bonded network interfaces. This also goes for faustino. The newest servers, urquell and cosmo, have both been installed with bonded network interfaces; this is our standard install practice at the moment. Please accept my apologies for the disruption and be assured that we have learned some lessons from this. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting