Hi chub, On Sat, Sep 26, 2009 at 04:59:41PM +0100, chub wrote: I'm afraid not. I have been unable to track down any root cause for the problem experienced on Wednesday/Thursday. I would have expected a re-occurrence very quickly if hardware related and in some ways would have preferred this as at least I would have something to fix. But as it is it's likely a bug in Xen which won't be seen again for a long time. Rest assured that if it does appear more likely to be a hardware issue there are spare servers available that customers will be moved to as soon as possible. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

Andy Smith wrote: Andy, keeping us up to date with all this, it's appreciated. -n

Evening Andy, Do you have any definite dates and times for when this work will be carried out yet? Chub.

Hi folks, On Thu, Oct 01, 2009 at 06:53:10PM +0000, Andy Smith wrote: The work is finished now and seems to have gone well. faustino was shut down at 0100Z and everything was back up by 0255Z. I spent a few minutes checking over faustino's event logs and such to see if I could find any obvious traces of hardware fault, but unfortunately there was none. I then removed the disks from a spare server and from faustino. This took ages because the supplier of faustino saw fit to screw the disks into the disk caddies to pointless levels of tightness, even damaging the heads as they did so. A couple of them I thought I was going to completely destroy the heads in trying to unscrew them, and if I had then I'd have been going home without completing the work. Note to self: make sure you can unscrew all the disks out of the caddies when the server is delivered. I tested the spare server's disks in faustino to ensure the RAID card would accept a disk swap (no issues) and then moved faustino's disks to new hardware. Apart from the disks, faustino now has entirely new hardware, in a different rack in a different room. If the same problems manifest then it can't be hardware, and we'll have to schedule a rebuild of the OS. I hope that the problems with this server are now behind us, and if so then I will be adding 12 days of free service (23rd -> 5th) for all of the customers on faustino. I'm going to wait a week before doing this though, just in case celebrations are premature. Please continue paying your bills as normal! ;) faustino used to be a dual core Opteron 2212HE 2GHz with 12GiB RAM. It is now a quad core Xeon 5410 2.33GHz with 16GiB RAM. Thanks for your patience in this matter. The 2 power cycles and a ~2 hour outage you've suffered so far is not great, though hardware problems and software bugs are unavoidable. Realistically when these sorts of things happen I don't think there's much scope for a faster response or resolution than this, at this end of the hosting market. Fortunately BitFolk is now big enough to have spare hardware about, making this a lot less painful than it might have been 2 years ago. I can improve this by adding more spare servers, being more willing to whip the disks out and make use of the spares when there is any hint of hardware issues. Shared storage would of course help but this is a scary topic! Happy to continue that discussion off-list.. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting "Xandros's low-level support for the Eee mostly seemed to consist of a pile of shell scripts made of cheese and failure." -- Matthew Garrett

Duplex mismatch on dom0 NIC? Graham Freeman (Sent from my handheld) On 6 Oct 2009, at 15:49, Hugo Hallqvist <hugohallqvist(a)gmail.com> wrote:

And few minutes later: 21:02 <@grifferz> ander: 4x7.2kRPM SATA disks in a RAID-10 Sorry for spam. ~a On Wed, Oct 7, 2009 at 9:03 PM, Ander Punnar &lt;ander.punnar(a)gmail.com&gt; wrote:

Hello, On Mon, Oct 05, 2009 at 03:38:00AM +0000, Andy Smith wrote: Again just under a week later, faustino locked up and had to be power cycled. With absolutely no log traces or entries in the hardware event logging I can't be 100% sure but I have to assume that the previous problem has followed the software, despite there being all new hardware, therefore I have to conclude that an operating system reinstall is required (which would also be an upgrade). This procedure is not yet fully automated for a live server and I suspect would probably take on the order of 4 hours, it's never been attempted before and there is potential for a mistake that could lose the existing data. Another alternative is to move every VPS (but not the host OS itself) to the newest server which was just provisioned yesterday -- barbar. This would eventually leave faustino empty so it could be reinstalled and soak tested at leisure. This would be far more work, though would involve less downtime to individual customers. I'm going to put some more thought into this and will follow up later today with what the plan is going to be. Rushing at this stage could lead to data loss and you all reinstalling from the backups (that I'm sure you keep), which would be worse than having to power cycle once a week. If I decide to do a reinstall, that would likely take place mid week. If it's to be a move, that would start very soon although may take all week to get through everyone. There may yet be other solutions. Apologies again for the problems with faustino. Naturally my promise of service credit equal to the number of days the problem has persisted still applies and is still counting up. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

So, will it happen or things are magically okay again? On Mon, Oct 12, 2009 at 4:14 AM, Andy Smith &lt;andy(a)bitfolk.com&gt; wrote: