Hello,
Unfortunately some more serious security issues have been uncovered
in Xen which affect versions and configurations which we have
deployed.
These were pre-disclosed yesterday, with full public disclosure
coming two weeks later on Thursday 12 October as normal.
So, we're going to have to patch everything and reboot before then.
This will very likely be taking place over three nights starting in
the early hours (BST) of Tuesday 10 October, but we will be sending
out an individual email to every customer confirming when they will
be affected.
For those unaware of what this entails, it means that at some point
within an hour-long maintenance window we will shut your VPS down
cleanly as the machine it's on is shut down, and then boot it again
once the machine has booted up. It typically takes 5-10 minutes.
As a reminder, you are able to opt for your VPS to be suspended to
and restored from SSD if you don't like losing program state:
https://tools.bitfolk.com/wiki/Suspend_and_restore
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Around 0920Z today, monitoring notified me that host "snaps" wasn't
responding.
I could get no response over the network and even the serial console
was completely unresponsive. I had no option but to hard power cycle
it.
It is now booted again and all customer VPSes on it should be
started again but I do not yet know the reason for the outage and am
still investigating.
Apologies for the disruption.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
As those of you on the "users" list may have seen, I've been making
some necessary changes to our Direct Debit integration with
GoCardless:
https://lists.bitfolk.com/lurker/message/20170920.113352.8bfd6684.en.html
On 31 October GoCardless are retiring the original API we integrated
with in 2012 so I've had to make the changes necessary to move to
their current "Pro" API.
Sadly there aren't any really any new features to talk about, hence
my delay in taking care of this. One thing I do need to mention is
that the capped mandates of the legacy API are no longer supported.
All Direct debit mandates set up through the legacy API were capped
to the exact amount per time period for your recurring VPS bill,
e.g. £10.79 per month, £29.64 per quarter, etc. In hindsight this
was a bad idea as it caused problems as soon as anyone upgraded or
tried to move their payment date earlier.
An awkward consequence of this is that pre-existing Direct Debit
mandates will still respect the caps set on them but there is no
longer any way for us to query those caps. Whereas before we could
tell that a charge might fail today but succeed tomorrow, and thus
would be happy enough to wait until tomorrow to submit it, we now
can't tell at all. We have to submit the charge and see if it fails.
And if it does fail, we can't tell the difference between "too soon"
and "too much". This is unfortunate because "too soon" is of course
a temporary issue, but "too much" is a permanent one.
I decided not to cancel everyone's Direct Debit mandates because
this is still a relatively rare problem: it only becomes an issue if
you upgrade your VPS or try to pay a one-off invoice, which then
causes the cap to be exceeded the next time your scheduled payment
tries to be charged.
Instead I've decided that we will just catch the "cap exceeded"
error and ask you to cancel and re-authorise on a case by case
basis. To assist with that I've made it so you can cancel your own
Direct Debit from the panel.
I whinged some more about this here:
http://strugglers.net/~andy/blog/2017/09/21/tricky-issues-when-upgrading-to…
and wrote some more about it here:
https://tools.bitfolk.com/wiki/Direct_Debit#Legacy_Integration
As before you can fiddle with the Direct Debit stuff from the
Billing section of the Panel:
https://panel.bitfolk.com/account/billing/
It's been live and taking payments for about 5 days now, but if
anyone has any issues with it of course do let me know. And I
welcome anyone with a UK bank account to switch to it as it is still
the best payment method. :)
Speaking of which: there was some hope that we could start taking
Direct Debits from customers with bank accounts in EUR and SEK, as
the Pro API does supports that. It turns out though that this also
requires BitFolk to have bank accounts in those currencies and to
also bill in those currencies. That's not something that I want to
do in the foreseeable future so instead I will work on doing
recurring payments through Stripe (credit/debit cards).
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
Unfortunately some serious security issues have been uncovered in
Xen, which affect versions and configurations which we have deployed.
These were pre-disclosed today, with full public disclosure coming
on Tuesday 12 September as normal.
So, we're going to have to patch everything and reboot before then.
This will very likely be taking place over three nights starting in
the early hours (BST) of Sunday 10 September, but we will be sending
out an individual email to every customer confirming when they will
be affected.
For those unaware of what this entails, it means that at some point
within an hour-long maintenance window assigned to your VPS we will
shut your VPS down cleanly as the machine it's on is shut down, and
then boot it again once the machine has booted up. It typically
takes 5-10 minutes.
If you wish you can opt for your VPS to be suspended to and restored
from SSD if you don't like losing program state:
https://tools.bitfolk.com/wiki/Suspend_and_restore
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
We were recently made aware that our Cacti¹ bandwidth graphs for a
particular customer were dramatically different from reality.
On investigation I realised that it was a bug in the Linux kernel
where it wasn't using 64-bit counters for Xen backend network
devices. As a result for readings above ~228Mbit/s the counter was
wrapping twice and reporting incorrect values on an SNMP read (as
used by Cacti).
This is a fairly minor issue because we do not use SNMP counters for
billing. It does mean that if your VPS has ever done more than about
228Mbit/s average in a 5 minute period that Cacti won't be showing
it properly.
The kernel bug has since been fixed but deploying a fixed version
would involve using a self-compiled² backports kernel. I am not
going to do this because I haven't tested it enough yet.
Instead I have identified the 30 or so customers that have ever
recorded that much bandwidth use in the last 12 months and am adding
new bandwidth graphs for you, using 1-minute polling. Also new
customers will have the 1-minute resolution graphs. That should be
safe to about 1.1Gbit/s.
So, if you are looking at Cacti and see you now have two bandwidth
graphs with one cutting off where the other began, this is the
reason why.
I wrote a blog article about this at:
http://strugglers.net/~andy/blog/2017/09/03/when-is-a-64-bit-counter-not-a-…
Cheers,
Andy
¹ https://tools.bitfolk.com/cacti/ - Log in with your usual BitFolk credentials
² We already use self-compiled kernels based on Debian kernel
packages, because some security patches have not yet made it into
Debian's packages. Building the kernel isn't the problem, it's
testing it well enough.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
