30 May
2012
30 May
'12
7:02 p.m.
Hi,
We're experiencing widespread network problems, including a DDoS and
also a possible hardware failure on barbar, all at the same time.
I am investigating and will let you know more as soon as I can.
Apologies for the disruption.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
30 May
30 May
10:27 p.m.
New subject: hardware problems on barbar, 1826Z and ongoing
On Wed, May 30, 2012 at 07:02:17PM +0000, Andy Smith wrote:
Hi,
We're experiencing widespread network problems, including a DDoS and
also a possible hardware failure on barbar, all at the same time.
I am investigating and will let you know more as soon as I can.
Here's the situation:
At just before 1826Z I enabled network device bonding on one of our
older servers, curacao. curacao is in the process of being cleared
of customers and only had a couple of people left on it. That's why
I chose to do it first. It was not expected to cause any issues.
It was done and all seemed fine.
A couple of minutes later, alerts started firing across all servers
about things being unreachable. At the same time, customers on
barbar experienced disk IO problems which caused running processes
to crash.
A few things were established fairly quickly:
- A DDoS was in progress against a customer on barbar.
- The resolvers were unreachable
- barbar was suffering a possible hardware failure.
The customer under DDoS was blackholed and the DDoS stopped a short
time later.
The resolver failure was tracked down to some sort of split brain
scenario in the cluster, and was patched up by forcing certain nodes
to run.
For the last few hours I have been working on barbar.
barbar is one of our older servers and represents an experiment in
SAS disks instead of hardware RAID+SATA. It's therefore the only
server we have that uses Linux software RAID (mdadm).
mdadm has for some reason failed 3 of the 4 disks in one of the
RAID10 arrays, which is fatal for a RAID10. I do not believe that
there is an actual failure of any of the drives because only one of
the arrays was kicked, not all of them, and I have been able to
reboot it and access data on all the arrays that do work.
Unfortunately the array that does not work holds all customer data
for that node as well as /usr and /var for barbar itself.
Customers on barbar should come to terms with the possibility of
complete data loss at this point.
Having said that, I am trying to avoid this. As I say I believe the
data is largely there. It may be slightly corrupted due to one disk
running for a while without the others. For some people the
corruption may be minor; for others it may be huge.
Having forcibly re-assembled the array we are now at the stage where
I have access to all of the devices, but they may contain damaged
data. I am taking a backup of the usr and var data as it is now,
before I will fsck them (fsck -n already says we are in for a bumpy
but probably not catastropic ride). If I manage to get the actual
server up then I will report on what we will do next.
I do not yet know the root cause of why barbar threw three of its
disks out, but I suspect it may be related to the DDoS. I suppose it
is possible that extreme load made it decide that three of its disks
were not responding and that it was a good idea to kick them, though
I've never seen that before. I won't speculate on that further until
I have more information (having access to /var should help!).
Thanks again for your patience.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
31 May
31 May
2:48 a.m.
New subject: [bitfolk] hardware problems on barbar, 1826Z and ongoing
On Wed, May 30, 2012 at 10:27:51PM +0000, Andy Smith wrote:
Having forcibly re-assembled the array we are now at
the stage where
I have access to all of the devices, but they may contain damaged
data. I am taking a backup of the usr and var data as it is now,
before I will fsck them (fsck -n already says we are in for a bumpy
but probably not catastropic ride). If I manage to get the actual
server up then I will report on what we will do next.
barbar itself has been up for a little while now. Damage to its /usr
and /var filesystems appear not to have been too severe.
I have also (with permission) managed to fsck two customer's
filesystems and boot their VPSes. One of them appeared to have only
minimal damage (about what you would expect from a hard power
cycle), the other completed an fsck without incident.
At the moment, every other customer on barbar is administratively
locked out of their Xen Shell in order to prevent them from starting
their VPSes and going straight into an fsck, as they may not have
followed all of this and may be unaware of the potential scale of
the problem.
What we're going to do:
For each customer hosted on barbar whose VPS is still down, I will:
- Run an fsck -n on their block devices provided they are ext3.
IFF that fsck -n returns cleanly:
- Start customer's VPS
OTHERWISE:
- Put a warning message in place in the Xen Shell directing
customers to the URL of this archived email.
- Take a backup copy of the customer's block devices
- Open a support ticket with the customer using the email
address we have on file for them
This support ticket will say something along the lines of:
Wah wah sky has fallen, etc.¹ As a result your VPS is not
currently running. When it IS started up, either by you or
by us, it will very very likely need to have an fsck run and
proceed to do this during the boot process.
Many block devices will have corruption and doing an fsck
could possibly make this worse. Therefore we need you to
reply to this support ticket to let us know that you've read
and understood the situation and are ready to either boot
your VPS yourself or happy to have us do it for you.
Realistically, completing an fsck is the only way that your
filesystems are going to get into a state where your VPS
will run, so this will be necessary at some point; we just
don't want to take the decision out of your hands.
If we do not hear from you in at least 24hours then we
will re-enable your login to your Xen Shell console and
leave you to boot your VPS yourself in your own time.
- Customers who haven't been heard from in at least 24 hours
will have their Xen Shell access restored so that they can
boot their VPSes in their own time. VPSes will not be started
for these customers without their prior permission.
There will be more communications on follow-up actions at a later
date.
Cheers,
Andy
¹ Replace this with a fuller description of the evening's events,
with links to the archives etc.
--
http://bitfolk.com/ -- No-nonsense VPS hosting
5:10 a.m.
New subject: [bitfolk] hardware problems on barbar, 1826Z and ongoing
On Thu, May 31, 2012 at 02:48:07AM +0000, Andy Smith wrote:
What we're going to do:
For each customer hosted on barbar whose VPS is still down, I will:
- Run an fsck -n on their block devices provided they are ext3.
IFF that fsck -n returns cleanly:
- Start customer's VPS
16 of 28 affected customers were able to have their VPSes booted as
fsck was clean.
OTHERWISE:
- Put a warning message in place in the Xen Shell directing
customers to the URL of this archived email.
- Take a backup copy of the customer's block devices
- Open a support ticket with the customer using the email
address we have on file for them
The other 12 of you are about to receive a support ticket as
described.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
4553
days inactive
4554
days old
3 comments
1 participants
participants (1)
-
Andy Smith