Hello,
We are in the process of renumbering from our transit provider's IP
space into our own IPv4 allocation (85.119.80.0/21). The first stage
of this is to renumber all of our own infrastructure. You'll be
seeing a few notifications like this over the coming weeks.
Our NTP servers have been renumbered. By default, BitFolk VPSes use
our NTP servers by host name. You should restart your NTP server now
in order to pick up the new IPs.
If you have ACLs that operate on IP addresses, you should change
them to:
85.119.80.232
85.119.80.233
2001:ba8:1f1:f205::53
2001:ba8:1f1:f206::53
On 11th October 2011 the old NTP IP addresses will stop responding.
Please hit reply or direct an email to support(a)bitfolk.com if you
are unsure of what to do.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
This email is a bit of a ramble about block device IO and SSDs and
contains no information immediately relevant to your service, so
feel free to skip it.
In considering what the next iteration of BitFolk infrastructure
will be like, I wonder about the best ways to use SSDs.
As you may be aware, IO load is the biggest deal in virtual hosting.
It's the limit everyone hits first. It's probably what will dismay
you first on Amazon EC2. Read
http://wiki.postgresql.org/images/7/7f/Adam-lowry-postgresopen2011.pdf
or at least pages 8, 29 and 30 of it.
Usually it is IO load that tells us when it's time to stop putting
customers on a server, even if it has a bunch of RAM and disk
space left. If disk latency gets too high everything will suck,
people will complain and cancel their accounts. When the disk
latency approaches 10ms we know it's time to stop adding VMs.
Over the years we've experimented with various solutions. We built
a server with 10kRPM SAS drives, and that works nicely, but the
storage then costs so much that it's just not economical.
After that we started building bigger servers with 8 disks instead of
4, and that's where we are now. This worked out, as we can usually
get around twice as many VMs on one server, and it saves having to
pay for an extra chassis, motherboard, PSUs and RAID controller.
SSD prices have now dropped enough that it's probably worth looking
at how they can be used here. I can think of several ways to go:
- Give you the option of purchasing SSD-backed capacity
=====================================================
Say SSD capacity costs 10 times what SATA capacity does. You get
to choose between 5G of SATA-backed storage or 0.5G of SSD-backed
storage for any additional storage you might like to purchase, the
same price for either.
Advantages:
- The space is yours alone; you get to put what you like on it. If
you've determined where your storage hot spots are, you can put
them on SSD and know they're on SSD.
Disadvantages:
- In my experience most people do not appreciate choice, they just
want it to work.
Most people aren't in a position to analyse their storage use
and find hot spots. They lack either the inclination or the
capability or both - the service is fine until it's not.
- It means buying two expensive SSDs that will spend most of their
time being unused.
Two required because they'll have to be in a RAID-1.
Most of the time unused because the capacity won't be sold
immediately.
Expensive because they will need to be large enough to cater to
as large a demand as I can imagine for each server.
Unfortunately I have a hard time guessing what that demand would
be like so I'll probably guess wrong.
- Find some means of using SSDs as a form of tiered storage
=========================================================
We could continue deploying the majority of your storage from SATA
disks while also employing SSDs to cache these slower disks in
some manner.
The idea is that frequently-accessed data is backed on SSD whereas
data that is accessed less often is left on the larger-capacity
SATA, and *this remains transparent to the end user*, i.e. the VM.
This is not a new idea; plenty of storage hardware already does
it, ZFS can do it and so can BTRFS.
Advantages:
- For whatever benefit there is, everyone gets to feel it. If done
right, any VM that needs more IOPs should get more IOPs.
- Expensive SSDs purchased can be used immediately, in full.
Disadvantages:
- Since we can't use ZFS or expensive storage hardware, any
short-term solution is likely to be rather hacky. Do we want to
be pioneers here? This is your data.
- Customers with VMs that don't have heavy IO requirements (most)
will be subsidising those who *do* have heavy IO requirements.
It's very unlikely we will put prices up, but SSDs are not free
so it has the effect of delaying the usual progression of
more-for-less that this type of service goes through.[1]
- Beyond what might be quite a blunt instrument, customers will
have no way to request faster storage and rely on it being
present. You have "some storage" and if that storage isn't
performing as fast as you would like, all we would be able to do
is try to see why it's not being cached on SSD.
- Both?
=====
Perhaps there is some way to do both? Maybe to start with using the
whole SSD as cache, but as requests to purchase SSD-backed storage
come in the cache could be reduced?
Advantages:
- Again everyone feels the benefit immediately and hardware isn't
wasted.
- If the customer needs to buy SSD-backed storage then they can.
Disadvantages:
- If the caching is good enough then no one would feel the need to
buy SSD anyway, so why add complexity?
Questionable:
- If people buy all of the SSD, does that reduce caching benefit
to zero and suddenly screw everyone else over?
Presumably SSD-backed storage could be priced such that if a lot
of people did buy it, it would be economical to go out and buy a
pair of larger ones and swap them over without downtime[2].
So, if anyone has any thoughts on this I'd be interested in hearing
them.
If you had an IO latency problem, would you know how to diagnose it
to determine that it was something you were doing as opposed to
"BitFolk's storage is overloaded but it's not me"?
If you could do that, would you be likely to spend more money on
SSD-backed storage?
If we came to you and said that your VPS service was IO-bound and
would run faster if you bought some SSD-backed storage, do you think
that you would?[3]
My gut feeling at the moment is that while I would love to be
feeding the geek inside everyone and offering eleventy-billion
choices, demand for SSD-backed storage at an additional cost will be
low.
I also think it's going to be very difficult for an admin of a
virtualised block device to tell the difference between:
"All my processes are really slow at talking to storage; it's
because of my process ID 12345 which is a heavy DB query"
and:
"All my processes are really slow at talking to storage; that's
definitely a problem with BitFolk's storage and not anything I
am doing."
By the way, I think we've done reasonably well at keeping IO latency
down, over the years:
barbar: http://tools.bitfolk.com/cacti/graphs/graph_1634_6.png
bellini: http://tools.bitfolk.com/cacti/graphs/graph_2918_4.png
cosmo: http://tools.bitfolk.com/cacti/graphs/graph_2282_4.png
curacao: http://tools.bitfolk.com/cacti/graphs/graph_1114_6.png
dunkel: http://tools.bitfolk.com/cacti/graphs/graph_1485_6.png
faustino: http://tools.bitfolk.com/cacti/graphs/graph_1314_6.png
kahlua: http://tools.bitfolk.com/cacti/graphs/graph_1192_6.png
kwak: http://tools.bitfolk.com/cacti/graphs/graph_1113_6.png
obstler: http://tools.bitfolk.com/cacti/graphs/graph_1115_6.png
president: http://tools.bitfolk.com/cacti/graphs/graph_2639_4.png
urquell: http://tools.bitfolk.com/cacti/graphs/graph_2013_6.png
(Play at home quiz: which four of the above do you think have eight
disks instead of four? Which one has four 10kRPM SAS disks? Answers
at [4])
In general we've found that keeping the IO latency below 10ms keeps
people happy.
There have been short periods where we've failed to keep it below
10ms and I'm sure that many of you can remember times when you've
found your VPS sluggish. Conversely I suspect that not many
customers can think of times when their VPSes have been the *cause*
of high IO load, yet high IO load is in general only caused by
customer VMs! So for every time you have experienced this, someone
else was causing it![5]
I think that, being in the business of providing virtual
infrastructure at commodity prices, we can't really expect too many
people to want or be able to take the time to profile their storage
use and make a call on what needs to be backed by SATA or SSD.
I think we first need to try to make it as good as possible for
everyone, always. There may be a time in the future where it's
commonplace for customers to evaluate storage in terms of IO
operations per second instead of gigabytes, but I don't think we are
there yet.
As for the "low-end customers subsidise higher-end customers"
argument, that's just how shared infrastructure works and is already
the case in many existing metrics, so what's one more? While we
continue to not have a good way to ration out IO capacity it is
difficult to add it as a line item.
So, at the moment I'm more drawn to the "both" option but with the
main focus being on caching with a view to making it better for
everyone, and hopefully overall reducing our costs. If we can sell
some dedicated SSD storage to those who have determined that they
need it then that would be a bonus.
Thoughts? Don't say, "buy a big SAN!" :-)
Cheers,
Andy
[1] You know, when we double the RAM or whatever but keep the price
to you the same.
[2] Hot swap trays plus Linux md = online array grow. In theory.
[3] "Nice virtual machine you have here. Would be a real shame if
the storage latency were to go through the roof, yanno? We got
some uh… extras… that can help you right out of that mess.
Pauly will drop by tomorrow with an invoice."
— Tony Soprano's Waste Management and Virtual Server Hosting,
Inc.
[4] echo "oryyvav, pbfzb, cerfvqrag naq hedhryy unir rvtug qvfxf.
oneone unf sbhe FNF qvfxf." | rot13
[5] Barring *very* occasional problems like a disk broken in such a
way that it doesn't die but delays every IO request, or a
battery on a RAID controller going defective, which disables the
write cache.
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi folks,
A disk has broken in obstler.bitfolk.com and I've just replaced it
as of ~14:00Z. This means that the array is degraded and has no
redundancy until the rebuild completes, which I would estimate at 6
hours.
I don't anticipate any outage, but now would be a great time to
ensure your backup regime is in order, if you have not already done
so.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
In case you haven't already heard:
----- Forwarded message from Jan Henkins -----
Hello there,
Forwarding this to official support due to it's importance (should have
done this earlier!). Please pass this on to the Bitfolk list!
Since I've sent the below message, I have found a mitigation strategy for
Debian:
1) Create /etc/apache2/conf.d/setenvif with the following content:
---star---
<IfModule mod_setenvif.c>
# Drop the Range header when more than 5 ranges.
# CVE-2011-3192
SetEnvIf Range (,.*?){5,} bad-range=1
RequestHeader unset Range env=bad-range
# optional logging.
CustomLog /var/log/apache2/range-CVE-2011-3192.log common
env=bad-range
</IfModule>
---end---
Be advised that the above should not work out of the box, since "headers"
module was not enabled by default (this could be the actual Debian and
Ubuntu standard).
2) Enable the headers and rewrite modules:
a2enmod headers
a2enmod rewrite
3) Restart apache
---------------------------- Original Message ----------------------------
Subject: Apache 1.* and 2.* vulnerability
From: "Jan Henkins"
Date: Thu, August 25, 2011 11:00
--------------------------------------------------------------------------
Hello Andy,
Sorry for not posting this to the Bitfolk list directly, I'm on my
web-mail (didn't put the mailing list address in my address book), so
please consider passing this on.
Yesterday a nasty Apache DoS vuln was released. So far all versions of
Apache is affected by this. Here are some advisories:
RedHat:
https://bugzilla.redhat.com/show_bug.cgi?id=732928
Debian:
https://mail-archives.apache.org/mod_mbox/httpd-announce/201108.mbox/%3C201…
While I have not managed to work out a mitigation strategy for
Ubuntu/Debian servers, the following works rather nicely on RHEL5 and
RHEL6 (so could be good to go for CentOS too):
Create /etc/httpd/conf.d/setenvif.conf with the following:
<IfModule mod_setenvif.c>
# Drop the Range header when more than 5 ranges.
# CVE-2011-3192
SetEnvIf Range (,.*?){5,} bad-range=1
RequestHeader unset Range env=bad-range
# optional logging.
CustomLog /your/log/dir/range-CVE-2011-3192.log common
env=bad-range
</IfModule>
Restart apache
That should do it nicely! :-)
More reading here:
http://eromang.zataz.com/2011/08/24/cve-2011-3192-apache-httpd-killer-remot…
Please pass on to the Bitfolk community at your discretion.
--
Regards,
Jan Henkins
--
Regards,
Jan Henkins
Hi,
We've had reports that the password reset feature of the panel is
broken at the moment.
Some new code was pushed live a few days ago and this obviously got
past testing. I am working on fixing this as a top priority, but in
the mean time if you do require a password reset please:
1. Check to see if we've fixed it yet
2. If not, contact support@ requesting reset
3. Use phone if urgent and you haven't received confirmation that
it's been done yet
I do expect it to be fixed today.
https://tools.bitfolk.com/redmine/issues/80
Apologies for the inconvenience.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
With immediate effect we are moving to a calendar monthly /
quarterly / yearly billing cycle instead of a rigid 30 / 90 / 360
day cycle.
This was requested in:
https://tools.bitfolk.com/redmine/issues/13
You do not need to take any action. You will effectively get 5 or 6
days of additional service per year compared to before.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
There was a request to provide an additional non-standard port for
SSH which would have relaxed Fail2Ban settings:
https://tools.bitfolk.com/redmine/issues/75
This is now done, and the additional port is 922. You won't be
blocked by Fail2Ban on that port no matter how many times you fail
your password.
I would also like to remind you that SSH key upload is also
supported, which you may prefer to password auth:
https://panel.bitfolk.com/account/security/
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hello,
As you may be aware, CentOS 6 was released recently. We've also had
requests to investigate another RHEL clone, Scientific Linux.
Here's the feature request for CentOS 6:
https://tools.bitfolk.com/redmine/issues/77
If you read that, you'll see there's a bit of a problem with not
being able to specify partitioning details in the text mode
installer any longer.
I'm not sure of the best way to proceed with this, so it would be
really helpful if any customers with a stake (i.e., if you're likely
to want to run CentOS / SL 6 at some point in the future) could log
in at the above URL and update the feature request with your
thoughts.
Replying to this email instead is OK too, though it'd be nicer to
keep it all in the feature tracker.
Thanks!
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
I was hoping to only mention this much closer to the time when it
could actually happen, but a few people have been asking about this
so I thought it was only fair to give an update.
TL;DR version:
We're planning to soon double the amount of RAM offered for the
current price. You'll be contacted individually when this can
happen; hopefully it just happens, but if you're unlucky then you
might need to be migrated or keep the same amount of RAM for a lower
price instead. Do not panic. You can stop reading now. :)
Longer version:
For a while now it has been clear to us that BitFolk has fallen
behind its competitors in how much RAM we offer for a given price
point, and I wanted to reassure you that I have been taking steps to
remedy this. These have included:
- 56GiB of RAM installed across several machines at the recent
maintenance at the end of February.
- Ongoing migration of all customers off of dunkel.bitfolk.com so
that some non-urgent hardware maintenance and a RAM upgrade can be
done.
- New server with 48GiB RAM being prepared for colo now.
We are hoping to double the RAM allocation for the same price, i.e.
the smallest VPS will come with 480MiB RAM and additional RAM may be
purchased 240MiB at a time.
We could of course just offer this right now to new customers, but I
really dislike it when existing customers are not offered the same
deal as new, even if it does make some business sense to chase the
new. So, I'm holding back from doing that until I know it will be
possible to apply the upgrade to the majority of existing customers.
I can't give any firm timescale for this right now; it's going to
happen as soon as possible.
For many customers this is all going to be very simple; you'll
receive an email that says "congrats, shut down and boot again to
enjoy twice as much RAM". That's it. Unfortunately for some of you
it's not going to be possible to "just do it".
Some of you will be on hosts that cannot be economically upgraded.
8GiB DDR2 DIMMs are stupidly expensive; it's cheaper to buy new
machines with 8GiB DDR3. Here's how it's going to work:
Customers with 240 or 360MiB RAM will be upgraded first. That's
because 240 and 360MiB plans won't be available to new customers
afterwards. If we can't upgrade you because the host is out of
RAM then you will be offered migration[1] to a host that does have
enough RAM. If there's no host available with enough RAM then you
will remain with the same amount of RAM and your future renewals
will be cheaper, however we will upgrade you to at least 480MiB RAM
as soon as we can and you will have no choice in this.[2]
Next, customers with 480MiB or more RAM will be upgraded. If we
can't upgrade you then you'll stay with the RAM you have, at the
lower cost.[3]
As soon as we start upgrading people, the new plans will be
available to new customers, although we may run out of capacity
quickly due to upgrading existing customers. Once this happens,
please don't try to cancel your monthly contract and sign up again
to get the extra RAM quicker; I would hope to have upgraded everyone
that I can within a couple of days of starting. :)
As usual if anyone wants to downgrade then they will be free to do
so from their next contract renewal.
However, if:
- you currently have 480MiB or more of RAM *and*;
- you know you don't need twice as much as you have now *and*;
- you're definitely going to ask to be downgraded back towards what
you started with;
then it would be helpful if you could email support(a)bitfolk.com with
your VPS account name and the amount of RAM you'd like to end up
with. We *may* be able to both downgrade you (and give you some
service credit back) and use that RAM to give to someone else. No
promises though; you may have to wait until your next renewal as
normal.
If you have any questions about all this then please just hit reply
to direct them to the users list, or send an email to
support(a)bitfolk.com if you'd rather discuss off-list.
Cheers,
Andy
[1] Migrating your VPS between hosts will require us to shut it down
and then boot it up again, with something like 5-10 minutes of
downtime in between. If a migration is necessary then we'll
contact you individually to co-ordinate the work.
[2] While you may be satisfied that 240 or 360MiB RAM is adequate
for you, I don't want to keep lots of custom plans going and I
don't want to bill below £8.99/mo if possible.
[3] e.g. If you're on 480MiB RAM, 10GiB disk, 100GB/mo data transfer
then that currently costs £13.99/mo. If we couldn't upgrade you
to 960MiB RAM then you will remain on 480MiB but the plan will
only cost you £8.99/mo from your next renewal.
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Earlier this evening a couple of customers on kwak.bitfolk.com
shut down their VPSes but were unable to start them again.
It appears we've hit some sort of bug here and customers aren't able
to start new virtual machines on this host. Therefore I'm going to
have to reboot the host now as a matter of urgency, i.e. in the next
few minutes.
The host has been up for 680 days now and this is the first real
problem encountered but I'll also take the opportunity to apply some
upgrades at the same time.
All customers on kwak are going to experience a clean shut down and
a boot up again a few minutes later.
Please accept my apologies for the disruption.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
