ee of its disks
were not responding and that it was a good idea to kick them, though
I've never seen that before. I won't speculate on that further until
I have more information (having access to /var should help!).
Thanks again for your patience.
Cheers,
Andy
--=20
http://bitfolk.com/ -- No-nonsense VPS hosting
--H1spWtNR+x+ondvy
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEAREDAAYFAk/GnucACgkQIJm2TL8VSQuoUgCgmjU9a+jnmN/bj6nr74pOdPWk
c14AoKvTnmJOW77HCPFwq1T8fqXkcWGO
=C7Lp
-----END PGP SIGNATURE-----
--H1spWtNR+x+ondvy--
--===============1596240413==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
announce mailing list
announce@???
https://lists.bitfolk.com/mailman/listinfo/announce
--===============1596240413==--
From announce-bounces+users=lists.bitfolk.com@??? Thu May 31 02:48:12 2012
Received: from localhost ([127.0.0.1] helo=bitfolk.com)
by mail.bitfolk.com with esmtp (Exim 4.72) (envelope-from
<announce-bounces+users=lists.bitfolk.com@???>)
id 1SZvQq-00049l-UM
for users@???; Thu, 31 May 2012 02:48:12 +0000
Received: from andy by mail.bitfolk.com with local (Exim 4.72)
(envelope-from <andy@???>) id 1SZvQl-00048g-LT
for announce@???; Thu, 31 May 2012 02:48:08 +0000
Date: Thu, 31 May 2012 02:48:07 +0000
From: Andy Smith <andy@???>
To: announce@???
Message-ID: <20120531024807.GD11695@???>
References: <20120530190217.GB11695@???>
<20120530222751.GC11695@???>
MIME-Version: 1.0
In-Reply-To: <20120530222751.GC11695@???>
OpenPGP: id=BF15490B; url=http://strugglers.net/~andy/pubkey.asc
X-URL: http://strugglers.net/wiki/User:Andy
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Thu,
31 May 2012 02:48:07 +0000
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
spamd0.lon.bitfolk.com
X-Spam-Level:
X-Spam-ASN:
X-Spam-Status: No, score=-0.0 required=5.0 tests=NO_RELAYS shortcircuit=no
autolearn=disabled version=3.3.1
X-Spam-Report: * -0.0 NO_RELAYS Informational: message was not relayed via SMTP
X-BeenThere: announce@???
X-Mailman-Version: 2.1.13
Precedence: list
Content-Type: multipart/mixed; boundary="===============0668465934=="
Sender: announce-bounces+users=lists.bitfolk.com@???
Errors-To: announce-bounces+users=lists.bitfolk.com@???
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Thu,
31 May 2012 02:48:12 +0000
X-SA-Exim-Connect-IP: 127.0.0.1
X-SA-Exim-Mail-From: announce-bounces+users=lists.bitfolk.com@???
X-SA-Exim-Scanned: No (on mail.bitfolk.com); SAEximRunCond expanded to false
Subject: Re: [bitfolk] hardware problems on barbar, 1826Z and ongoing
X-BeenThere: users@???
Reply-To: users@???
List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
<mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
List-Post: <mailto:users@lists.bitfolk.com>
List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
<mailto:users-request@lists.bitfolk.com?subject=subscribe>
X-List-Received-Date: Thu, 31 May 2012 02:48:13 -0000
--===============0668465934==
Content-Type: multipart/signed; micalg=pgp-ripemd160;
protocol="application/pgp-signature"; boundary="/Uq4LBwYP4y1W6pO"
Content-Disposition: inline
--/Uq4LBwYP4y1W6pO
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed, May 30, 2012 at 10:27:51PM +0000, Andy Smith wrote:
> Having forcibly re-assembled the array we are now at the stage where
> I have access to all of the devices, but they may contain damaged
> data. I am taking a backup of the usr and var data as it is now,
> before I will fsck them (fsck -n already says we are in for a bumpy
> but probably not catastropic ride). If I manage to get the actual
> server up then I will report on what we will do next.
barbar itself has been up for a little while now. Damage to its /usr
and /var filesystems appear not to have been too severe.
I have also (with permission) managed to fsck two customer's
filesystems and boot their VPSes. One of them appeared to have only
minimal damage (about what you would expect from a hard power
cycle), the other completed an fsck without incident.
At the moment, every other customer on barbar is administratively
locked out of their Xen Shell in order to prevent them from starting
their VPSes and going straight into an fsck, as they may not have
followed all of this and may be unaware of the potential scale of
the problem.
What we're going to do:
For each customer hosted on barbar whose VPS is still down, I will:
- Run an fsck -n on their block devices provided they are ext3.
IFF that fsck -n returns cleanly:
- Start customer's VPS
OTHERWISE:
- Put a warning message in place in the Xen Shell directing
customers to the URL of this archived email.
- Take a backup copy of the customer's block devices
- Open a support ticket with the customer using the email
address we have on file for them
This support ticket will say something along the lines of:
Wah wah sky has fallen, etc.=B9 As a result your VPS is not
currently running. When it IS started up, either by you or
by us, it will very very likely need to have an fsck run and
proceed to do this during the boot process.
Many block devices will have corruption and doing an fsck
could possibly make this worse. Therefore we need you to
reply to this support ticket to let us know that you've read
and understood the situation and are ready to either boot
your VPS yourself or happy to have us do it for you.
Realistically, completing an fsck is the only way that your
filesystems are going to get into a state where your VPS
will run, so this will be necessary at some point; we just
don't want to take the decision out of your hands.
If we do not hear from you in at least 24hours then we
will re-enable your login to your Xen Shell console and
leave you to boot your VPS yourself in your own time.
- Customers who haven't been heard from in at least 24 hours
will have their Xen Shell access restored so that they can
boot their VPSes in their own time. VPSes will not be started
for these customers without their prior permission.
There will be more communications on follow-up actions at a later
date.
Cheers,
Andy
=B9 Replace this with a fuller description of the evening's events,
with links to the archives etc.
--=20
http://bitfolk.com/ -- No-nonsense VPS hosting
--/Uq4LBwYP4y1W6pO
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEAREDAAYFAk/G2+cACgkQIJm2TL8VSQuyoQCgqsTEywZVmOUtpRzZ3ZI7CqAs
E3oAoI6Nlkc2GFK+lpEMTOw1nVMFEr3F
=HgTu
-----END PGP SIGNATURE-----
--/Uq4LBwYP4y1W6pO--
--===============0668465934==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
announce mailing list
announce@???
https://lists.bitfolk.com/mailman/listinfo/announce
--===============0668465934==--
From announce-bounces+users=lists.bitfolk.com@??? Thu May 31 05:11:24 2012
Received: from localhost ([127.0.0.1] helo=bitfolk.com)
by mail.bitfolk.com with esmtp (Exim 4.72) (envelope-from
<announce-bounces+users=lists.bitfolk.com@???>)
id 1SZxfP-00012v-I3
for users@???; Thu, 31 May 2012 05:11:24 +0000
Received: from andy by mail.bitfolk.com with local (Exim 4.72)
(envelope-from <andy@???>) id 1SZxex-00011P-4K
for announce@???; Thu, 31 May 2012 05:10:55 +0000
Date: Thu, 31 May 2012 05:10:55 +0000
From: Andy Smith <andy@???>
To: announce@???
Message-ID: <20120531051054.GE11695@???>
References: <20120530190217.GB11695@???>
<20120530222751.GC11695@???>
<20120531024807.GD11695@???>
MIME-Version: 1.0
In-Reply-To: <20120531024807.GD11695@???>
OpenPGP: id=BF15490B; url=http://strugglers.net/~andy/pubkey.asc
X-URL: http://strugglers.net/wiki/User:Andy
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Thu,
31 May 2012 05:10:55 +0000
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
spamd0.lon.bitfolk.com
X-Spam-Level:
X-Spam-ASN:
X-Spam-Status: No, score=-0.0 required=5.0 tests=NO_RELAYS shortcircuit=no
autolearn=disabled version=3.3.1
X-Spam-Report: * -0.0 NO_RELAYS Informational: message was not relayed via SMTP
X-BeenThere: announce@???
X-Mailman-Version: 2.1.13
Precedence: list
Content-Type: multipart/mixed; boundary="===============2001397509=="
Sender: announce-bounces+users=lists.bitfolk.com@???
Errors-To: announce-bounces+users=lists.bitfolk.com@???
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Thu,
31 May 2012 05:11:24 +0000
X-SA-Exim-Connect-IP: 127.0.0.1
X-SA-Exim-Mail-From: announce-bounces+users=lists.bitfolk.com@???
X-SA-Exim-Scanned: No (on mail.bitfolk.com); SAEximRunCond expanded to false
Subject: Re: [bitfolk] hardware problems on barbar, 1826Z and ongoing
X-BeenThere: users@???
Reply-To: users@???
List-Id: Users of BitFolk hosting
