Hi,
As you may be aware, Ubuntu 12.04 (Precise Pangolin) is due to be
released today.
We have not yet had chance to test an upgrade from 10.04 (Lucid
Lynx) to 12.04, so whilst 12.04 is an LTS release I must warn you
that we are not aware of any customer who is successfully running
it. Unless you feel like being a pioneer it may be best to wait a
little while before trying such an upgrade.
In the next few days I hope to be able to add 12.04 to the net
installer:
https://tools.bitfolk.com/wiki/Self-serve_net_install
which would make it available for new installs and new
self-installs. If I run into a problem with it then it may take
longer, but I don't really expect it to be a problem (Debian testing
works fine, for example).
I'll send another announce email if and when there is more news
about 12.04's availability and use at BitFolk. Meanwhile if any
brave customers would like to try upgrades and document it on the
wiki that would be most welcome.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
The rescue environment has been given a bit of an upgrade and is now
documented:
https://tools.bitfolk.com/wiki/Rescue
It's still launched the same way and works pretty much the same way,
the only real difference is that it's up-to-date and now based on
the debian-live project so will be much easier to keep up-to-date.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
A bunch of alerts for a.authns.bitfolk.com were sent out a few
minutes ago. These were sent in error and don't represent any actual
problem.
I'm in the process of reconfiguring some software on that system and
the monitoring momentarily couldn't connect, so sent out alerts to
that effect. No actual service has been disrupted.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi folks,
Short version:
Two of our servers appear to be subject to a now fixed kernel bug
affecting IPv6, and require a reboot for kernel upgrade. Host
bellini.bitfolk.com will be rebooted on Monday 20th February at
2200Z.
Provided that does fix the problem, host president.bitfolk.com will
be similarly rebooted the following day, Tuesday 21st February also
at 2200Z.
Longer version:
While investigating some recent reports of poor IPv6 performance, it
seems that both bellini.bitfolk.com and president.bitfolk.com are
affected by a bug in the igb Intel gigabit Ethernet driver as
described here:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630730
The symptoms are very poor IPv6 performance in the region of a
maximum of 100kilobit/sec. On doing a tcpdump you will see packets
of length greater than 1500 bytes, followed by an ICMP6 "packet too
big" message coming from our server, and then a retransmit.
These hosts have been up for 138 days (bellini) and 223 days
(president), and unfortunately neither we nor any customers noticed
any problems until recently. On casual inspection IPv6 works. It's
only noticeable when trying to do a larger data transfer.
Since the current impact is so low, I am not going to rush to reboot
these hosts. I would rather give plenty of notice to those who need
it.
When it's time for the reboot we will shut down all VPSes on these
servers cleanly, reboot the machine and then begin booting them up
again. Downtime is expected to be in the region of 15 minutes. If
you are in any doubt as to whether your VPS starts cleanly with all
required services running, you should test this ahead of time.
I am fairly confident that the problems observed are caused by that
bug and therefore that a kernel upgrade will fix it, but
unfortunately we do not have any other hardware that uses the igb
driver. If doing the upgrade on bellini does not resolve the issue
then we will have to consider our options.
In the mean time, if your VPS is hosted on bellini or president then
you may wish to set your VPS to prefer IPv4 DNS results ahead of
IPv6 results:
https://tools.bitfolk.com/wiki/IPv6#Preferring_IPv4_over_IPv6
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi Gavin,
On Mon, Feb 20, 2012 at 11:55:47PM +0000, Gavin Westwood wrote:
> On 19/01/2012 10:09, Andy Smith wrote:
> > Two of our servers appear to be subject to a now fixed kernel bug
> > affecting IPv6, and require a reboot for kernel upgrade. Host
> > bellini.bitfolk.com will be rebooted on Monday 20th February at
> > 2200Z.
>
> Did I blink and miss this, or has bellini not been rebooted?
> (I was logged into my server, meaning to shut it down and had forgotten).
Um. This is embarrassing.
00:52:33 -!- Topic for #BitFolk: [ http://bitfolk.com/ ] VPS capacity: Lots | ☑ Unicode chaser ∞ | That's renumberwang: http://is.gd/VkINBU | bellini,
president to be rebooted on 21st, 22nd Feb: http://is.gd/UJakWd | April meet: http://is.gd/zEqOZ1
00:52:33 -!- Topic set by grifferz [] [Wed Feb 8 15:28:54 2012]
00:53:42 <@grifferz> okay, why did no one notice that the topic says 21st,22nd february whereas the actual email itself says 20th, 21st??
So basically, since February 8th I've been under the impression that
I actually meant 21st and 22nd, and wrote those dates into every
timekeeping device I have. :(
I am actually prepared for the work though, and it's now "only" ~3
hours late, so I think it would be best to go ahead and do it.
So, I'm really sorry for screwing the timing up, but I'm going ahead
and rebooting bellini in a few minutes (instead of ~3 hours ago).
The work on president will happen at the correct time (Tuesday 21st,
2200Z).
Thanks for pointing this out, Gavin!
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
If you don't make use of BitFolk's secondary DNS service then you
can ignore this email as it has no impact on you.
c.authns.bitfolk.com has been renumbered:
209.20.91.73 → 173.255.227.192
2001:4978:f:392::2 → 2600:3c03::31:2053
This actually has nothing to do with BitFolk's renumbering.
c.authns used to be hosted at Slicehost which recently got bought
by Rackspace, and they are also renumbering.
Most people don't need to do anything. The glue record for
c.authns.bitfolk.com has been updated so over the next couple of
days the new IP will start being used.
If you have made your own DNS records that pointed at 209.20.91.73
or 2001:4978:f:392::2 (for example if you made nameserver host
names inside your own domain(s)) then you will need to update them.
It is recommended not to use such names because of the need to
change them in so many places.
You may also have ACLs that allow zone transfer from 209.20.91.73.
These normally don't get used since the BitFolk nameservers try to do
zone transfer from each other first. If you do have such an ACL then
you will need to update it.
209.20.91.73 and 2001:4978:f:392::2 will go away on 19th March.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
If you don't currently take advantage of BitFolk's free backups
service then the following doesn't apply to you.
Two extra checks were added last night on the backups that BitFolk
manages for you.
All this info is going to go on a page I will create at
https://tools.bitfolk.com/wiki/Backups but you need to know it now.
- Backup age
This checks that your most recent successful¹ backup is not too
old. The warning threshold is 2.5x the appropriate interval. So if your
backups happen every 4 hours, you'll receive an alert if they are
ever older than 10 hours. If your backups happen daily then
you'll receive an alert if they are ever more than 60 hours old.
And so on.
This alerting replaces the manual process of me being sent
excerpts of log files that say a customer's backups are failing,
then opening a support ticket with the customer to make them
aware.
If you receive this alert then your backups are definitely not
happening.
The alert looks like this:
From: nagios(a)bitfolk.com
Subject: ** PROBLEM alert - backup0.bitfolk.com/Backup age youraccount is CRITICAL **
***** Nagios *****
Notification Type: PROBLEM
Service: Backup age youraccount
Host: backup0.bitfolk.com
Address: 85.119.80.240
State: CRITICAL
Date/Time: Tue Jan 31 12:37:38 UTC 2012
Additional Info:
FILE_AGE CRITICAL: /data/backup/rsnapshot.6-7-4-6/hourly.0/85.119.82.121 is 16842912 seconds old and 4096 bytes
(Those who haven't had a successful backup run in the last couple
of days will have a huge number of seconds listed there because
the backup system was only modified to record last successful
contact recently)
- Backup space usage
This checks that your total backup space usage is not approaching
your current quota. The thresholds are 95% for a warning and 99%
for a critical.
This alerting replaces the manual process of me being warned about
customers who are exceeding their quota and then opening support
tickets with them to discuss what they want to do about it².
If you receive this alert then your backups are still happening,
but you're in danger of (or already are) using more than the
agreed space. If you exceed your quota then we may disable your
backups, so that would eventually cause the above backup age alert
to fire.
Please note that we can't update the measurement of how much space
you're using very often. Backup directories contain hundreds of
millions of files, many of which are copies of each other, but
it's not possible to tell without looking. Adding it all up takes
quite a long time and stresses disk IO quite badly. So we only
update quotas every day at best at the moment.
Also please note that since this is a backup system, deleting
files on your VPS does not immediately result in using less disk
space for backups. It would be a rather pointless backup system if
it threw away deleted files immediately. :) Anything that gets
backed up is going to be kept for as long as your chosen backup
schedule dictates, e.g. 6 months by default. If you have blown
your quota by accidentally allowing large amounts of data to be
backed up, you are still going to have to contact support to get
it deleted.
The alert looks like this:
From: nagios(a)bitfolk.com
Subject: ** PROBLEM alert - backup2.bitfolk.com/Backup space usage youraccount is WARNING **
***** Nagios *****
Notification Type: PROBLEM
Service: Backup space usage youraccount
Host: backup2.bitfolk.com
Address: 85.119.80.230
State: WARNING
Date/Time: Tue Jan 31 15:52:28 UTC 2012
Additional Info:
WARNING 98.50% (394/400MiB) used
It is possible for the usage to go above 100% because we do allow
you to go over your quota for short periods of time.
Cheers,
Andy
¹ "Successful" as in, rsync connected to your host, did some stuff
and then exited with a non-error exit code. It does not
necessarily mean that what you think should be backed up is being
backed up. As with any backup solution you need to assure yourself
on a regular basis that it's doing what you expect.
² Generally one or more of:
- Buy some more disk space for backups
- Backup fewer files
- Backup less often
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
I've just enabled some additional alerting for the DNS secondary
service. It will copy you in on alerts regarding BitFolk nameservers
that are not correctly serving your domain.
They look like this:
From: nagios(a)bitfolk.com
Subject: ** PROBLEM alert - a.authns.bitfolk.com/Auth. DNS example.com is UNKNOWN **
***** Nagios *****
Notification Type: PROBLEM
Service: Auth. DNS example.com
Host: a.authns.bitfolk.com
Address: 85.119.80.222
State: UNKNOWN
Date/Time: Fri Jan 27 19:46:10 UTC 2012
Additional Info:
DNS UNKNOWN - 1.444 seconds response time (No ANSWER SECTION found)
This would indicate that a.authns.bitfolk.com is not serving the
domain example.com when we would expect it to be.
The reason for this change is that there are quite a few customer
domains that aren't being served correctly and rather than keep
opening tickets and chasing it up, I would rather let Nagios do what
it is designed for.
I am also working on a check that AXFRs are happening.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Today we discovered that the ability to specify a custom kickstart
URL when doing a CentOS or Scientific Linux install was broken. It
just used the default kickstart anyway.
It's now fixed, so if you had tried to do that in the past and
failed, it wasn't your fault - apologies.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
Effective immediately our base data transfer quota is increasing
from 100GB/month to 200GB/month. All existing accounts have had
100GB extra data transfer quota added to their current quota.
If you were paying for additional data transfer quota over and above
100GB then you may now be paying for more than you need. If that is
the case then please contact support to ask for what you don't need
to be removed (down to a minimum of 200GB) before your next payment
is due.
Also, the overage charge has been reduced from £0.75 per GB to £0.25
per GB. It is still *much* cheaper to commit to additional data
transfer than to go into overage[1].
If you have any questions, just reply, or open a support ticket.
All prices exclude VAT.
Cheers,
Andy
[1] Committing to an extra 10GB/month is £0.50. Using 10GB of
overage will now be £2.50.
--
"I never wanted to be a comedian. When I was very young I wanted to be a
writer […] But then, at the age of 16, I saw a comedian called Ted
Chippington open for The Fall in Birmingham in October 1984 and
literature's loss became stand-up comedy's loss also." — Stewart Lee