Hi folks,
Short version:
Two of our servers appear to be subject to a now fixed kernel bug
affecting IPv6, and require a reboot for kernel upgrade. Host
bellini.bitfolk.com will be rebooted on Monday 20th February at
2200Z.
Provided that does fix the problem, host president.bitfolk.com will
be similarly rebooted the following day, Tuesday 21st February also
at 2200Z.
Longer version:
While investigating some recent reports of poor IPv6 performance, it
seems that both bellini.bitfolk.com and president.bitfolk.com are
affected by a bug in the igb Intel gigabit Ethernet driver as
described here:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630730
The symptoms are very poor IPv6 performance in the region of a
maximum of 100kilobit/sec. On doing a tcpdump you will see packets
of length greater than 1500 bytes, followed by an ICMP6 "packet too
big" message coming from our server, and then a retransmit.
These hosts have been up for 138 days (bellini) and 223 days
(president), and unfortunately neither we nor any customers noticed
any problems until recently. On casual inspection IPv6 works. It's
only noticeable when trying to do a larger data transfer.
Since the current impact is so low, I am not going to rush to reboot
these hosts. I would rather give plenty of notice to those who need
it.
When it's time for the reboot we will shut down all VPSes on these
servers cleanly, reboot the machine and then begin booting them up
again. Downtime is expected to be in the region of 15 minutes. If
you are in any doubt as to whether your VPS starts cleanly with all
required services running, you should test this ahead of time.
I am fairly confident that the problems observed are caused by that
bug and therefore that a kernel upgrade will fix it, but
unfortunately we do not have any other hardware that uses the igb
driver. If doing the upgrade on bellini does not resolve the issue
then we will have to consider our options.
In the mean time, if your VPS is hosted on bellini or president then
you may wish to set your VPS to prefer IPv4 DNS results ahead of
IPv6 results:
https://tools.bitfolk.com/wiki/IPv6#Preferring_IPv4_over_IPv6
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi Gavin,
On Mon, Feb 20, 2012 at 11:55:47PM +0000, Gavin Westwood wrote:
> On 19/01/2012 10:09, Andy Smith wrote:
> > Two of our servers appear to be subject to a now fixed kernel bug
> > affecting IPv6, and require a reboot for kernel upgrade. Host
> > bellini.bitfolk.com will be rebooted on Monday 20th February at
> > 2200Z.
>
> Did I blink and miss this, or has bellini not been rebooted?
> (I was logged into my server, meaning to shut it down and had forgotten).
Um. This is embarrassing.
00:52:33 -!- Topic for #BitFolk: [ http://bitfolk.com/ ] VPS capacity: Lots | ☑ Unicode chaser ∞ | That's renumberwang: http://is.gd/VkINBU | bellini,
president to be rebooted on 21st, 22nd Feb: http://is.gd/UJakWd | April meet: http://is.gd/zEqOZ1
00:52:33 -!- Topic set by grifferz [] [Wed Feb 8 15:28:54 2012]
00:53:42 <@grifferz> okay, why did no one notice that the topic says 21st,22nd february whereas the actual email itself says 20th, 21st??
So basically, since February 8th I've been under the impression that
I actually meant 21st and 22nd, and wrote those dates into every
timekeeping device I have. :(
I am actually prepared for the work though, and it's now "only" ~3
hours late, so I think it would be best to go ahead and do it.
So, I'm really sorry for screwing the timing up, but I'm going ahead
and rebooting bellini in a few minutes (instead of ~3 hours ago).
The work on president will happen at the correct time (Tuesday 21st,
2200Z).
Thanks for pointing this out, Gavin!
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
If you don't make use of BitFolk's secondary DNS service then you
can ignore this email as it has no impact on you.
c.authns.bitfolk.com has been renumbered:
209.20.91.73 → 173.255.227.192
2001:4978:f:392::2 → 2600:3c03::31:2053
This actually has nothing to do with BitFolk's renumbering.
c.authns used to be hosted at Slicehost which recently got bought
by Rackspace, and they are also renumbering.
Most people don't need to do anything. The glue record for
c.authns.bitfolk.com has been updated so over the next couple of
days the new IP will start being used.
If you have made your own DNS records that pointed at 209.20.91.73
or 2001:4978:f:392::2 (for example if you made nameserver host
names inside your own domain(s)) then you will need to update them.
It is recommended not to use such names because of the need to
change them in so many places.
You may also have ACLs that allow zone transfer from 209.20.91.73.
These normally don't get used since the BitFolk nameservers try to do
zone transfer from each other first. If you do have such an ACL then
you will need to update it.
209.20.91.73 and 2001:4978:f:392::2 will go away on 19th March.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting