Reboot scheduled for host "bellini", 2012-02-20 2200Z; host "president", 2012-02-21 2200Z
19 Jan
2012
19 Jan
'12
10:09 a.m.
Hi folks,
Short version:
Two of our servers appear to be subject to a now fixed kernel bug
affecting IPv6, and require a reboot for kernel upgrade. Host
bellini.bitfolk.com will be rebooted on Monday 20th February at
2200Z.
Provided that does fix the problem, host president.bitfolk.com will
be similarly rebooted the following day, Tuesday 21st February also
at 2200Z.
Longer version:
While investigating some recent reports of poor IPv6 performance, it
seems that both bellini.bitfolk.com and president.bitfolk.com are
affected by a bug in the igb Intel gigabit Ethernet driver as
described here:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630730
The symptoms are very poor IPv6 performance in the region of a
maximum of 100kilobit/sec. On doing a tcpdump you will see packets
of length greater than 1500 bytes, followed by an ICMP6 "packet too
big" message coming from our server, and then a retransmit.
These hosts have been up for 138 days (bellini) and 223 days
(president), and unfortunately neither we nor any customers noticed
any problems until recently. On casual inspection IPv6 works. It's
only noticeable when trying to do a larger data transfer.
Since the current impact is so low, I am not going to rush to reboot
these hosts. I would rather give plenty of notice to those who need
it.
When it's time for the reboot we will shut down all VPSes on these
servers cleanly, reboot the machine and then begin booting them up
again. Downtime is expected to be in the region of 15 minutes. If
you are in any doubt as to whether your VPS starts cleanly with all
required services running, you should test this ahead of time.
I am fairly confident that the problems observed are caused by that
bug and therefore that a kernel upgrade will fix it, but
unfortunately we do not have any other hardware that uses the igb
driver. If doing the upgrade on bellini does not resolve the issue
then we will have to consider our options.
In the mean time, if your VPS is hosted on bellini or president then
you may wish to set your VPS to prefer IPv4 DNS results ahead of
IPv6 results:
https://tools.bitfolk.com/wiki/IPv6#Preferring_IPv4_over_IPv6
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
19 Jan
19 Jan
10:19 a.m.
New subject: [bitfolk] Reboot scheduled for host "bellini", 2012-02-20 2200Z; host "president", 2012-02-21 2200Z
Hello,
On Thu, Jan 19, 2012 at 10:09:24AM +0000, Andy Smith wrote:
Two of our servers appear to be subject to a now fixed
kernel bug
affecting IPv6, and require a reboot for kernel upgrade. Host
bellini.bitfolk.com will be rebooted on Monday 20th February at
2200Z.
Forgot to mention:
If you aren't sure which host your VPS is on, it tells you on:
https://panel.bitfolk.com/account/
Also you can tell by resolving
<youraccountname>.console.bitfolk.com, e.g.:
$ host ruminant.console.bitfolk.com
ruminant.console.bitfolk.com is an alias for console.president.bitfolk.com.
console.president.bitfolk.com is an alias for president.bitfolk.com.
president.bitfolk.com has address 85.119.80.16
president.bitfolk.com has IPv6 address 2001:ba8:0:1f1::6
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
22 Feb
22 Feb
4:47 a.m.
New subject: Further emergency maintenance needed for node "president"
Hello,
On Thu, Jan 19, 2012 at 10:09:24AM +0000, Andy Smith wrote:
Provided that does fix the problem, host
president.bitfolk.com will
be similarly rebooted the following day, Tuesday 21st February also
at 2200Z.
This is not going well. Ever since president was rebooted last night
it has suffered from poor IO performance:
http://tools.bitfolk.com/cacti/graph_2639.html
I've been working through the night to try to find the root cause,
but am not having much luck.
There's no higher demands being made of the node compared to earlier
on:
http://tools.bitfolk.com/cacti/graph_2640.html
and as far as I can see, the RAID controller is happy with every
disk and with its battery and write cache.
At the moment I am running a reinitialise on the RAID array which is
going to take a couple of hours. Once that completes, if it hasn't
improved matters then I will be rebooting the node again.
I have some ideas of a couple of settings that should be changed,
that require a reboot, although I can't explain why their current
setting would make any difference now compared to the last ~250
days. It is a bit of a shot in the dark admittedly.
If that doesn't work, I shall be on my way to the datacentre to
replace RAID controller from spares.
I'm afraid it looks like this is going to cut well into UK working
hours today. Please accept my apologies for the disruption; it has
been completely unexpected.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
10:26 a.m.
New subject: [bitfolk] Further emergency maintenance needed for node "president"
Hello,
On Wed, Feb 22, 2012 at 04:47:24AM +0000, Andy Smith wrote:
At the moment I am running a reinitialise on the RAID
array which is
going to take a couple of hours. Once that completes, if it hasn't
improved matters then I will be rebooting the node again.
Literally just a couple of minutes after typing the above email, I
decided that changing IO scheduler would squeeze a little more IOPS
out of it and so would be a good thing to do while waiting for the
initialise to complete.
Normally all BitFolk nodes run with 'cfq' as IO scheduler.
3ware recommends 'deadline' instead, but in previous testing I'd
found that 'cfq' was only slightly worse than 'deadline'. Since
'cfq' allows use of the 'ionice' command for crude disk IO quality
of service adjustments, I go with 'cfq' normally.
Well, to my vast confusion, as soon as it was switched to 'deadline'
all problems went away. Instead of it making for a very slight
benefit as expected, it has completely returned performance to
normal levels.
Lovely as this is, I do not like mysteries, so will need to continue
to investigate. However I don't now predict needing to do a reboot
or anything else drastic.
I disabled a few IO-intensive things in the early hours of the
morning to try to make life more bearable, and I'm now going to
re-enable those things.
Thanks again for your patience.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
4687
days inactive
4721
days old
3 comments
1 participants
participants (1)
-
Andy Smith