15 Oct
2011
15 Oct
'11
10:53 a.m.
Hi folks,
I'm afraid that we've encountered a problem on obstler.bitfolk.com
where VPSes can't be started again once they've been shut down, and
we're going to have to reboot the host to fix this. We need to do it
as soon as possible as otherwise anyone who shuts their VPS down
will not be able to start it again.
I'm going to start an orderly VPS shutdown in a moment, which will
be followed by a host reboot and VPS startup again. This can take a
while because shutdown/startup happens serially. I'll follow up
again with more details.
10:52:08 up 230 days, 13:05, 4 users, load average: 1.17, 1.08, 0.96
Thanks, and apologies for the disruption.
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
15 Oct
15 Oct
11:40 a.m.
New subject: [bitfolk] Emergency shutdown / reboot required for VPSes on obstler.bitfolk.com
Hi,
On Sat, Oct 15, 2011 at 10:53:07AM +0000, Andy Smith wrote:
I'm going to start an orderly VPS shutdown in a
moment, which will
be followed by a host reboot and VPS startup again. This can take a
while because shutdown/startup happens serially. I'll follow up
again with more details.
All VPSes on obstler were started up again as of about 11:27Z. If
you are experiencing any problems with your VPS please take a look
at its console, and then contact support if necessary.
Since so any of the VPSes on obstler have been up for a long time,
many are doing a fsck at the same time now. As a result, IO latency
will be higher than normal for a short while longer (less than half
an hour I expect):
http://tools.bitfolk.com/cacti/graph_1115.html
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
12:22 p.m.
New subject: [bitfolk] Emergency shutdown / reboot required for VPSes on obstler.bitfolk.com
Hi,
On Sat, Oct 15, 2011 at 11:40:34AM +0000, Andy Smith wrote:
On Sat, Oct 15, 2011 at 10:53:07AM +0000, Andy Smith
wrote:
All VPSes on obstler were started up again as of about 11:27Z. If
you are experiencing any problems with your VPS please take a look
at its console, and then contact support if necessary.
I'm sorry, but it's happened again, almost immediately. When this
happened earlier I was in the process of rebooting a particular
(BitFolk infrastructure) VM. After obstler came back up I tried to
reboot it again and this same situation has occurred again.
I've now moved that VM off to another server, and in a moment I'm
going to have to shutdown and boot obstler again, but my concern now
is that something is broken with the software on obstler itself and
*any* VM being shut down will break it.
There's a couple of things that we can do should that be the case,
but right now I'm afraid I'm just going to have to do it to see what
the situation is.
Thank you for your patience.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
12:56 p.m.
New subject: [bitfolk] Emergency shutdown / reboot required for VPSes on obstler.bitfolk.com
Hello,
On Sat, Oct 15, 2011 at 12:22:24PM +0000, Andy Smith wrote:
I've now moved that VM off to another server, and
in a moment I'm
going to have to shutdown and boot obstler again, but my concern now
is that something is broken with the software on obstler itself and
*any* VM being shut down will break it.
Update:
I've shut down and booted obstler again, and then shut down and
booted some of BitFolk's own VMs on there, and that worked out as
expected, so I've now started all the customer VPSes again.
I'm not satisfied to leave it at that, but I'm not going to rush
into anything right now today.
obstler is well in need of an upgrade - it's one of our oldest
installs. I think the best thing to do will be to migrate everyone
off of that server and then reinstall it with the software versions
we're using for new installs.
I will be in touch with customers on obstler individually over the
next week to arrange migration.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
4817
days inactive
4817
days old
3 comments
1 participants
participants (1)
-
Andy Smith