27 Jul
2010
27 Jul
'10
11:57 p.m.
Hello,
I was alerted that urquell was unresponsive. I connected to the
serial console and found no useful info, and no response, so had no
choice but to power cycle. There was nothing in the BIOS event log
either.
Obviously I'm concerned that there might be a hardware problem here,
especially since I added 12GiB of RAM only last Saturday, but I
don't want to cause more disruption by moving VPSes to spare
hardware just yet. I will keep a close eye on it for now.
Apologies for the disruption,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
5 Aug
5 Aug
3:44 p.m.
New subject: 2010-07-27 2321Z - Lock up / powercycle of urquell.bitfolk.com
Hi,
On Tue, Jul 27, 2010 at 11:57:50PM +0000, Andy Smith wrote:
I was alerted that urquell was unresponsive. I
connected to the
serial console and found no useful info, and no response, so had no
choice but to power cycle. There was nothing in the BIOS event log
either.
This happened again today at approximately 1512Z, with exactly the
same symptoms: lock up, no useful messages in hardware event log.
I'm going to have to consider this a fault with the RAM that was
added on 24th July, and will have to take a trip to replace it with
new from spares.
I will follow up when that is planned. It will not be tonight, but
very soon.
Apologies for the disruption,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
9 Aug
9 Aug
9:55 a.m.
New subject: 2010-08-09 2000Z: Emergency work scheduled for urquell.bitfolk.com (Was Re: 2010-07-27 2321Z - Lock up / powercycle of urquell.bitfolk.com)
Hello,
On Thu, Aug 05, 2010 at 03:44:53PM +0000, Andy Smith wrote:
I will follow up when that is planned. It will not be
tonight, but
very soon.
Tonight at 9pm UK time (2000Z) I'm going to swap out the
recently-installed RAM in urquell for new RAM.
Apologies for the short notice, but so far it has locked up 4 days
after the original RAM install, then again 9 days after that. It's
now been 4 days since that and I would like to avoid further
unscheduled outages.
The work will take less than 30 minutes and will involve a clean
shut down and then boot of all VPSes on urquell.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
10 Aug
10 Aug
7:51 p.m.
New subject: 2010-08-09 2000Z: Emergency work scheduled for urquell.bitfolk.com (Was Re: 2010-07-27 2321Z - Lock up / powercycle of urquell.bitfolk.com)
Hi,
On Mon, Aug 09, 2010 at 09:55:59AM +0000, Andy Smith wrote:
On Thu, Aug 05, 2010 at 03:44:53PM +0000, Andy Smith
wrote:
Last night's maintenance went to plan, but nevertheless, a few
minutes ago urquell locked up again and had to be power cycled. So,
the new RAM which was added is unlikely to be faulty and there more
be a more serious problem.
More work will be required, either a switch to spare hardware or a
replacement of motherboard at least. I will follow up with the plans
for this within the next 24 hours.
Again I do apologise to customers on urquell for the disruption.
I recognise that this is not an acceptable level of service and
there will be a decent amount of service credit offered once we've
got to the bottom of the recent problems.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
I will follow up when that is planned. It will
not be tonight, but
very soon.
Tonight at 9pm UK time (2000Z) I'm going to swap out the
recently-installed RAM in urquell for new RAM.
13 Aug
13 Aug
2:16 p.m.
New subject: 2010-08-14 1400Z: Emergency work scheduled for urquell.bitfolk.com (Was: 2010-08-09 2000Z: Emergency work scheduled for urquell.bitfolk.com)
Hello,
On Tue, Aug 10, 2010 at 07:51:28PM +0000, Andy Smith wrote:
On Mon, Aug 09, 2010 at 09:55:59AM +0000, Andy Smith
wrote:
Last night's maintenance went to plan, but nevertheless, a few
minutes ago urquell locked up again and had to be power cycled. So,
the new RAM which was added is unlikely to be faulty and there more
be a more serious problem.
The hardware vendor is convinced it is still a RAM mismatch issue. I
am skeptical given that the RAM used so far was matched to the
motherboard and of the same type, but they are insistent and have
loaned me some that they're sure of, to test.
So, I'll be taking the server down again at 3pm UK time (1400Z) on
Saturday 14th August to install this RAM. Again it will be a clean
shut down, up to 30 minutes of downtime, and then boot again.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
16 Aug
16 Aug
5:19 p.m.
New subject: 2010-08-14 1400Z: Emergency work scheduled for urquell.bitfolk.com (Was: 2010-08-09 2000Z: Emergency work scheduled for urquell.bitfolk.com)
Hi,
On Fri, Aug 13, 2010 at 02:16:52PM +0000, Andy Smith wrote:
The hardware vendor is convinced it is still a RAM
mismatch issue. I
am skeptical given that the RAM used so far was matched to the
motherboard and of the same type, but they are insistent and have
loaned me some that they're sure of, to test.
Server locked up again at approximately 1704Z (18:04 UK time),
proving not an issue with the RAM. I need to discuss with the vendor
but there will need to be further work, very soon. I will update you
of when this is to take place.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
21 Aug
21 Aug
4:40 p.m.
New subject: 2010-08-24 0630Z: Emergency work scheduled for urquell.bitfolk.com
Hi,
On Mon, Aug 16, 2010 at 05:19:24PM +0000, Andy Smith wrote:
Hi,
On Fri, Aug 13, 2010 at 02:16:52PM +0000, Andy Smith wrote:
I've run out of patience with the vendor on this (latest hoop: "run
memtest").
On Monday 23rd I'm installing more servers and provided all goes
well I intend to put this issue to rest as far as customers are
concerned by returning on Tuesday 24th to swap the disks from
urquell into one of the new servers.
It will from then on be using entirely new hardware aside from the
disks, leaving me plenty of times to do all the pointless things the
vendor wants me to do without further affecting your services.
Lesson learned; buy more hardware, not hardware support.
Should there be further issues before Tuesday, I will have to power
cycle it; any other action is going to involve more downtime than a
reboot.
I am planning to carry out the work at 0630Z (7.30am UK time)
Tuesday 24th August and I expect it to take less than 30 minutes,
with no action necessary by yourselves. It will involve shutting
down your VPSes, shutting down the server, taking the disks out and
placing them in the other server, booting up the other server and
then your VPSes will also start again.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
The hardware vendor is convinced it is still a
RAM mismatch issue. I
am skeptical given that the RAM used so far was matched to the
motherboard and of the same type, but they are insistent and have
loaned me some that they're sure of, to test.
Server locked up again at approximately 1704Z (18:04 UK time),
proving not an issue with the RAM. I need to discuss with the vendor
but there will need to be further work, very soon. I will update you
of when this is to take place.
4997
days inactive
5022
days old
6 comments
1 participants
participants (1)
-
Andy Smith