BitFolk Announcements January 2012

announce@mailman.bitfolk.com

1 participants
7 discussions

Reboot scheduled for host "bellini", 2012-02-20 2200Z; host "president", 2012-02-21 2200Z

by Andy Smith

Hi folks, Short version: Two of our servers appear to be subject to a now fixed kernel bug affecting IPv6, and require a reboot for kernel upgrade. Host bellini.bitfolk.com will be rebooted on Monday 20th February at 2200Z. Provided that does fix the problem, host president.bitfolk.com will be similarly rebooted the following day, Tuesday 21st February also at 2200Z. Longer version: While investigating some recent reports of poor IPv6 performance, it seems that both bellini.bitfolk.com and president.bitfolk.com are affected by a bug in the igb Intel gigabit Ethernet driver as described here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630730 The symptoms are very poor IPv6 performance in the region of a maximum of 100kilobit/sec. On doing a tcpdump you will see packets of length greater than 1500 bytes, followed by an ICMP6 "packet too big" message coming from our server, and then a retransmit. These hosts have been up for 138 days (bellini) and 223 days (president), and unfortunately neither we nor any customers noticed any problems until recently. On casual inspection IPv6 works. It's only noticeable when trying to do a larger data transfer. Since the current impact is so low, I am not going to rush to reboot these hosts. I would rather give plenty of notice to those who need it. When it's time for the reboot we will shut down all VPSes on these servers cleanly, reboot the machine and then begin booting them up again. Downtime is expected to be in the region of 15 minutes. If you are in any doubt as to whether your VPS starts cleanly with all required services running, you should test this ahead of time. I am fairly confident that the problems observed are caused by that bug and therefore that a kernel upgrade will fix it, but unfortunately we do not have any other hardware that uses the igb driver. If doing the upgrade on bellini does not resolve the issue then we will have to consider our options. In the mean time, if your VPS is hosted on bellini or president then you may wish to set your VPS to prefer IPv4 DNS results ahead of IPv6 results: https://tools.bitfolk.com/wiki/IPv6#Preferring_IPv4_over_IPv6 Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

12 years, 2 months

Additional monitoring for backups

by Andy Smith

Hi, If you don't currently take advantage of BitFolk's free backups service then the following doesn't apply to you. Two extra checks were added last night on the backups that BitFolk manages for you. All this info is going to go on a page I will create at https://tools.bitfolk.com/wiki/Backups but you need to know it now. - Backup age This checks that your most recent successful¹ backup is not too old. The warning threshold is 2.5x the appropriate interval. So if your backups happen every 4 hours, you'll receive an alert if they are ever older than 10 hours. If your backups happen daily then you'll receive an alert if they are ever more than 60 hours old. And so on. This alerting replaces the manual process of me being sent excerpts of log files that say a customer's backups are failing, then opening a support ticket with the customer to make them aware. If you receive this alert then your backups are definitely not happening. The alert looks like this: From: nagios(a)bitfolk.com Subject: ** PROBLEM alert - backup0.bitfolk.com/Backup age youraccount is CRITICAL ** ***** Nagios ***** Notification Type: PROBLEM Service: Backup age youraccount Host: backup0.bitfolk.com Address: 85.119.80.240 State: CRITICAL Date/Time: Tue Jan 31 12:37:38 UTC 2012 Additional Info: FILE_AGE CRITICAL: /data/backup/rsnapshot.6-7-4-6/hourly.0/85.119.82.121 is 16842912 seconds old and 4096 bytes (Those who haven't had a successful backup run in the last couple of days will have a huge number of seconds listed there because the backup system was only modified to record last successful contact recently) - Backup space usage This checks that your total backup space usage is not approaching your current quota. The thresholds are 95% for a warning and 99% for a critical. This alerting replaces the manual process of me being warned about customers who are exceeding their quota and then opening support tickets with them to discuss what they want to do about it². If you receive this alert then your backups are still happening, but you're in danger of (or already are) using more than the agreed space. If you exceed your quota then we may disable your backups, so that would eventually cause the above backup age alert to fire. Please note that we can't update the measurement of how much space you're using very often. Backup directories contain hundreds of millions of files, many of which are copies of each other, but it's not possible to tell without looking. Adding it all up takes quite a long time and stresses disk IO quite badly. So we only update quotas every day at best at the moment. Also please note that since this is a backup system, deleting files on your VPS does not immediately result in using less disk space for backups. It would be a rather pointless backup system if it threw away deleted files immediately. :) Anything that gets backed up is going to be kept for as long as your chosen backup schedule dictates, e.g. 6 months by default. If you have blown your quota by accidentally allowing large amounts of data to be backed up, you are still going to have to contact support to get it deleted. The alert looks like this: From: nagios(a)bitfolk.com Subject: ** PROBLEM alert - backup2.bitfolk.com/Backup space usage youraccount is WARNING ** ***** Nagios ***** Notification Type: PROBLEM Service: Backup space usage youraccount Host: backup2.bitfolk.com Address: 85.119.80.230 State: WARNING Date/Time: Tue Jan 31 15:52:28 UTC 2012 Additional Info: WARNING 98.50% (394/400MiB) used It is possible for the usage to go above 100% because we do allow you to go over your quota for short periods of time. Cheers, Andy ¹ "Successful" as in, rsync connected to your host, did some stuff and then exited with a non-error exit code. It does not necessarily mean that what you think should be backed up is being backed up. As with any backup solution you need to assure yourself on a regular basis that it's doing what you expect. ² Generally one or more of: - Buy some more disk space for backups - Backup fewer files - Backup less often -- http://bitfolk.com/ -- No-nonsense VPS hosting

12 years, 2 months

Additional alerting for DNS secondary service

by Andy Smith

Hi, I've just enabled some additional alerting for the DNS secondary service. It will copy you in on alerts regarding BitFolk nameservers that are not correctly serving your domain. They look like this: From: nagios(a)bitfolk.com Subject: ** PROBLEM alert - a.authns.bitfolk.com/Auth. DNS example.com is UNKNOWN ** ***** Nagios ***** Notification Type: PROBLEM Service: Auth. DNS example.com Host: a.authns.bitfolk.com Address: 85.119.80.222 State: UNKNOWN Date/Time: Fri Jan 27 19:46:10 UTC 2012 Additional Info: DNS UNKNOWN - 1.444 seconds response time (No ANSWER SECTION found) This would indicate that a.authns.bitfolk.com is not serving the domain example.com when we would expect it to be. The reason for this change is that there are quite a few customer domains that aren't being served correctly and rather than keep opening tickets and chasing it up, I would rather let Nagios do what it is designed for. I am also working on a check that AXFRs are happening. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

12 years, 3 months

Custom kickstart URLs were broken, now fixed

by Andy Smith

Hi, Today we discovered that the ability to specify a custom kickstart URL when doing a CentOS or Scientific Linux install was broken. It just used the default kickstart anyway. It's now fixed, so if you had tried to do that in the past and failed, it wasn't your fault - apologies. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

12 years, 3 months

Increase of base data transfer quota

by Andy Smith

Hello, Effective immediately our base data transfer quota is increasing from 100GB/month to 200GB/month. All existing accounts have had 100GB extra data transfer quota added to their current quota. If you were paying for additional data transfer quota over and above 100GB then you may now be paying for more than you need. If that is the case then please contact support to ask for what you don't need to be removed (down to a minimum of 200GB) before your next payment is due. Also, the overage charge has been reduced from £0.75 per GB to £0.25 per GB. It is still *much* cheaper to commit to additional data transfer than to go into overage[1]. If you have any questions, just reply, or open a support ticket. All prices exclude VAT. Cheers, Andy [1] Committing to an extra 10GB/month is £0.50. Using 10GB of overage will now be £2.50. -- "I never wanted to be a comedian. When I was very young I wanted to be a writer […] But then, at the age of 16, I saw a comedian called Ted Chippington open for The Fall in Birmingham in October 1984 and literature's loss became stand-up comedy's loss also." — Stewart Lee

12 years, 3 months

Disclosure of information to referrers

by Andy Smith

Hello, As you may be aware, we operate a referral scheme: http://bitfolk.com/referrals.html For a long time there has been an open feature request for panel.bitfolk.com to display a list of referrals: https://tools.bitfolk.com/redmine/issues/9 There is a potential privacy concern since any such list will reveal some amount of information about one customer to another, but I find it difficult to quantify as there has been little feedback. I've therefore decided that a reasonable amount of information to reveal to a referrer would be: * New customer's account name * Referral amount * New customer's payment schedule (monthly / quarterly / yearly) * The date the new customer's service started, and whether they are still a customer or not. I have updated our privacy policy to this effect, and intend to put this feature live in not less than 30 days (Saturday 11th February 2012 or later). If you can't remember who (if anyone) referred you to us, go to: https://panel.bitfolk.com/account/ If you were referred by anyone then it will now say: You were referred to us by <account>. in the "Referrals" section of the page, under "Billing". If there's nothing there then this does not affect you. If you have strong feelings about your referrer seeing this information then I would encourage you to leave feedback in redmine (your usual BitFolk credentials) so we can try to come to a consensus over what is appropriate. If we can't come to a consensus that you're happy with then you can of course ask support to remove the referral. So, provided this does not prove controversial, this email is your 30 day notice of a modification to our privacy policy: http://bitfolk.com/policy/dpa.html Disclosure of personal information [...] If you nominate an existing customer as a referrer then you give consent for some details regarding your account with us to be disclosed to your referrer. You can find full information about this in our referral scheme. If you would prefer that these details were not disclosed, do not nominate a referrer. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

12 years, 3 months

Ongoing network instability from ~2339Z

by Andy Smith

Hello, Starting around 2339Z and still ongoing, we experienced network instability. High latency, high packet loss, occasional complete lack of connectivity. I am currently in contact with our transit provider to investigate and will provide more info when I have it. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

12 years, 3 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BitFolk Announcements January 2012