BitFolk Announcements

announce@mailman.bitfolk.com

1 participants
265 discussions

Spectre / Meltdown CPU bugs - short notice reboots will be needed, little other info yet

by Andy Smith

Hello, As you're probably aware, it turns out that pretty much every CPU made in the last 10 years is broken, and while this affects almost all computers, this is going to have a particularly nasty effect on virtualisation providers such as BitFolk. The Xen project last night released the first version of their advisory which is XSA-254: https://xenbits.xen.org/xsa/advisory-254.html This is with no embargo, because the original embargo had to be abandoned by the discoverers of the bugs. As you can see, unfortunately the Xen project have no resolutions for any of this available as yet. There's three different issues here: 1. SP1/Spectre (CVE-2017-5753) 2. SP2/Spectre (CVE-2017-5715) 3. SP3/Meltdown (CVE-2017-5754) There isn't any known resolution for (1) yet. Xen are working on mitigations for (2). It's possible to avoid (3) by going to HVM mode, but that is a huge change that brings other problems with it. It can also be avoided by running in PVH mode, but very few guest kernels will be new enough to support that. Xen are hoping to come up with a way to run PV-inside-PVH but they're not ready with that yet. There will likely be other strategies to fix or mitigate these issues in the coming days. So I'm afraid there currently is no concrete plan because there is very little information available yet. All I can tell you is that there will be a need for short-notice reboots to apply relevant fixes. I will post again when there is any useful information. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

6 years, 10 months

Latest CentOS 7 kernel doesn't boot under Xen PV - workaround available

by Andy Smith

Hi, I haven't had chance to personally check this out but apparently the latest CentOS 7 kernel package doesn't boot under Xen PV: https://bugs.centos.org/view.php?id=13763 This may be highly relevant to you because an update was just pushed out for the KPTI feature (to help mitigate Spectre/Meltdown etc in Linux). As mentioned in that bug report, there are patches to fix this but they haven't yet been applied to the main CentOS kernel package. In the mean time you can use the kernel package from the CentOSPlus repository which does have this fix and the KPTI one. https://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus All of this was researched by a customer having the problem today and it resolved it for them. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

6 years, 10 months

Outage / unexpected shutdown+boot for VPSes on host "snaps", 2017-12-28 ~08:06Z – ~08:40Z

by Andy Smith

Hi, Around 08:06Z today we received an alert regarding host snaps.bitfolk.com. I found it completely unresponsive over network, but was still able to connect to its console. Despite it believing its network interfaces were up and had link, it was passing no traffic to the colo switches. I spent about 30 minutes trying to diagnose this and not getting anywhere, so decided to try rebooting it. As I had console access I was able to cleanly shut down all VPSes on snaps first. The shutdown and boot went without incident and things seemed fine on boot. By about 08:40Z all VPSes that should be running had been started, and by now Nagios is clear of alerts¹. I am aware that snaps had an unexplained outage a few months ago, on 28 September. This time the symptoms are not the same, other than that the problem is unexplained and clears after a reboot. Clearly there is something wrong there though and it's going to happen again, so over the next few days we will be moving customers off of snaps. We will co-ordinate this directly with customers involved. Apologies for the disruption, Andy BitFolk Ltd ¹ Except for one customer web server which is waiting for a TLS passphrase to be supplied before it will start. -- https://bitfolk.com/ -- No-nonsense VPS hosting

6 years, 10 months

Re: [bitfolk] Unreliable IPv6?

by Andy Smith

Hi Andreas, On Tue, Dec 19, 2017 at 09:17:46AM +0100, Andreas Olsson wrote: > Is it just me, or has Bitfolk's IPv6 connectivity been a bit unreliably > these last few months? I'm sorry about this, it is a known issue that I am working on. The trigger of the problem is that Jump (our transit provider) has a router which has some sort of intermittent fault and has been rebooting itself. For the last few months they've been replacing bits of it to try to track down where the fault lies, but haven't got to the bottom of it yet. They're going to replace it entirely in January. As far as IPv4 goes, VRRP is in use and so when this router dies you don't really notice. On the IPv6 side of things it's relying on the usual IPv6 router advertisements and something is wrong with BitFolk's configuration there. All of BitFolk's hosts have two IPv6 default routes learned from router advertisements, and what is supposed to happen is that when of the routers dies the route is removed. This isn't happening, so the useless route stays there until the expiry time (30 minutes) or the reconvergence after the faulty router has rebooted (about 10 minutes). I've been trying to work out why BitFolk's setup doesn't remove the useless route, trying stuff out in test networks etc and so far I can't work out what is different in BitFolk's setup to make it not work. I will continue working on that even after Jump's router is replaced. I will keep you up to date on developments with this. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

6 years, 11 months

Cost of additional data transfer is halved

by Andy Smith

Hi, I've lowered the cost of 10GB of additional data transfer by half, so the changes are: ----------+-------+------ | Old | New ----------+-------------- Monthly | £0.50 | £0.25 Quarterly | £1.40 | £0.70 Yearly | £5.00 | £2.50 ----------+-------+------ If you do not already pay for additional monthly data transfer then the rest of this email probably won't be of interest. Those paying by Direct Debit will just be charged less. Those paying by PayPal have already had their subscription details altered and PayPal should have told you about that. Those paying by standing order will need to take care to adjust their regular payment otherwise you will be paying too much. It will build up as credit on your account. You can see the cost of your current spec at: https://panel.bitfolk.com/account/config/ Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

6 years, 11 months

XSA-246 and XSA-247 - no maintenance required

by Andy Smith

Hi, Two Xen Security Advisories were made public today: http://xenbits.xen.org/xsa/ They do not apply to the configurations that BitFolk runs at present, so there is no disruptive maintenance work required. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

6 years, 11 months

Security reboots coming, likely on 10/11/12 October

by Andy Smith

Hello, Unfortunately some more serious security issues have been uncovered in Xen which affect versions and configurations which we have deployed. These were pre-disclosed yesterday, with full public disclosure coming two weeks later on Thursday 12 October as normal. So, we're going to have to patch everything and reboot before then. This will very likely be taking place over three nights starting in the early hours (BST) of Tuesday 10 October, but we will be sending out an individual email to every customer confirming when they will be affected. For those unaware of what this entails, it means that at some point within an hour-long maintenance window we will shut your VPS down cleanly as the machine it's on is shut down, and then boot it again once the machine has booted up. It typically takes 5-10 minutes. As a reminder, you are able to opt for your VPS to be suspended to and restored from SSD if you don't like losing program state: https://tools.bitfolk.com/wiki/Suspend_and_restore Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

7 years, 1 month

Host "snaps" had to be power cycled, investigating

by Andy Smith

Hi, Around 0920Z today, monitoring notified me that host "snaps" wasn't responding. I could get no response over the network and even the serial console was completely unresponsive. I had no option but to hard power cycle it. It is now booted again and all customer VPSes on it should be started again but I do not yet know the reason for the outage and am still investigating. Apologies for the disruption. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

7 years, 1 month

Slight changes to Direct Debit payment method

by Andy Smith

Hi, As those of you on the "users" list may have seen, I've been making some necessary changes to our Direct Debit integration with GoCardless: https://lists.bitfolk.com/lurker/message/20170920.113352.8bfd6684.en.html On 31 October GoCardless are retiring the original API we integrated with in 2012 so I've had to make the changes necessary to move to their current "Pro" API. Sadly there aren't any really any new features to talk about, hence my delay in taking care of this. One thing I do need to mention is that the capped mandates of the legacy API are no longer supported. All Direct debit mandates set up through the legacy API were capped to the exact amount per time period for your recurring VPS bill, e.g. £10.79 per month, £29.64 per quarter, etc. In hindsight this was a bad idea as it caused problems as soon as anyone upgraded or tried to move their payment date earlier. An awkward consequence of this is that pre-existing Direct Debit mandates will still respect the caps set on them but there is no longer any way for us to query those caps. Whereas before we could tell that a charge might fail today but succeed tomorrow, and thus would be happy enough to wait until tomorrow to submit it, we now can't tell at all. We have to submit the charge and see if it fails. And if it does fail, we can't tell the difference between "too soon" and "too much". This is unfortunate because "too soon" is of course a temporary issue, but "too much" is a permanent one. I decided not to cancel everyone's Direct Debit mandates because this is still a relatively rare problem: it only becomes an issue if you upgrade your VPS or try to pay a one-off invoice, which then causes the cap to be exceeded the next time your scheduled payment tries to be charged. Instead I've decided that we will just catch the "cap exceeded" error and ask you to cancel and re-authorise on a case by case basis. To assist with that I've made it so you can cancel your own Direct Debit from the panel. I whinged some more about this here: http://strugglers.net/~andy/blog/2017/09/21/tricky-issues-when-upgrading-to… and wrote some more about it here: https://tools.bitfolk.com/wiki/Direct_Debit#Legacy_Integration As before you can fiddle with the Direct Debit stuff from the Billing section of the Panel: https://panel.bitfolk.com/account/billing/ It's been live and taking payments for about 5 days now, but if anyone has any issues with it of course do let me know. And I welcome anyone with a UK bank account to switch to it as it is still the best payment method. :) Speaking of which: there was some hope that we could start taking Direct Debits from customers with bank accounts in EUR and SEK, as the Pro API does supports that. It turns out though that this also requires BitFolk to have bank accounts in those currencies and to also bill in those currencies. That's not something that I want to do in the foreseeable future so instead I will work on doing recurring payments through Stripe (credit/debit cards). Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

7 years, 1 month

Security reboots coming, likely 10/11/12 September

by Andy Smith

Hello, Unfortunately some serious security issues have been uncovered in Xen, which affect versions and configurations which we have deployed. These were pre-disclosed today, with full public disclosure coming on Tuesday 12 September as normal. So, we're going to have to patch everything and reboot before then. This will very likely be taking place over three nights starting in the early hours (BST) of Sunday 10 September, but we will be sending out an individual email to every customer confirming when they will be affected. For those unaware of what this entails, it means that at some point within an hour-long maintenance window assigned to your VPS we will shut your VPS down cleanly as the machine it's on is shut down, and then boot it again once the machine has booted up. It typically takes 5-10 minutes. If you wish you can opt for your VPS to be suspended to and restored from SSD if you don't like losing program state: https://tools.bitfolk.com/wiki/Suspend_and_restore Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

7 years, 2 months

← Newer
1
...
9
10
11
12
13
14
15
...
27
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BitFolk Announcements