Hi,
It was pointed out to us that the HTTPS checks on our monitoring system
were only checking for a valid TLS certificate, not for a success code
from the URL. e.g. serving a completely secure 503 error page would
result in an "OK" check result.
This morning at around 09:55 we fixed that so that the HTTPS checks are
really checking the status code of the URL supplied. This has caused a
few new alerts to start being sent to people.
By fixing that, TLS certificate validity is now NOT being checked. We
will shortly add an additional check for this. You don't have to do
anything.
HTTPS and many other checks through our monitoring system are available
free upon request.
https://tools.bitfolk.com/wiki/Monitoring
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
At approximately 00:03Z we start receiving alerts of various services
not responding and it was determined that host talisker was having some
problems with its storage.
There were lots of errors being spewed into the kernel log from the SAS
controller's driver mostly of a timeout variety, and none of the drives
attached to it were responding. A number of its MD RAID arrays fell
apart as a result and IO errors would have been seen inside your virtual
machines.
I did try a few things around resetting the controller but nothing
worked so at around 00:35 I had to forcibly kill all running VPSes and
reboot the host, which happened at about 00:29.
The host talisker booted without incident and all its RAID arrays synced
up. By around 00:39 all customer VPSes should have booted, and all those
we have monitoring for did show as up by then.
Due to abruptly losing access to storage, some data in memory will have
been lost, but hopefully apps are aware of that. I do not think any
reads or writes were corrupted so I don't think there should be any
filesystem corruption. If you are seeing any problems and your VPS is
actually on talisker than you should first have a look at your Xen
Shell consoles.
Apologies for the disruption. We will keep an eye on talisker to gain
some assurance that this was a one-off event.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
It seems that since about 0500Z today the Debian mirror CDN at
deb.debian.org has been rejecting requests from our apt-cacher with a
message like:
E: Failed to fetch http://apt-cacher.lon.bitfolk.com/debian/deb.debian.org/debian/dists/bullse… 421 Misdirected Request [IP: 2001:ba8:1f1:f079::2 80]
I am looking into this. I don't know what is wrong yet. If you are
urently needing to uodate or install new packages just disable use of
BitFolk;s apt0cacher by removing "apt-cacher.lon.bitfolk.com/debian/"
from every line of your sources.list file(s).
I suspect some unfortunate interaction between Debian CDN (Fastly?) and
apt-cacher, as other mirrors are still working fine through our
apt-cacher.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
If you do not use or care about IPv6 with regard to your BitFolk VM(s)
you can stop reading this now.
As of this month we have started assigning IPv6 /48 netblocks to new
customers out of BitFolk's own allocation rather than continue giving
out /64s from our colo provider's allocation. Yesterday evening we also
assigned /48s to all existing customer VMs.
New installs (including those done yourself) will get set up with your
/48 from the start but existing VMs do need a few changes to make use of
this new address apace. If you know what you are doing you can just look
at:
https://panel.bitfolk.com/dns/
to find your /48 assignment and start configuring addresses and routes
from within that. They should work.
If that doesn't work or if you need more guidance here is an article
aimed at existing customers:
https://tools.bitfolk.com/wiki/New_/48_assignments,_October_2024
If you still have any questions not covered by the Troubleshooting or
Frequently Asked Questions sections then please do ask, by reply email
or support ticket or on Telegram or IRC.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
I do hope that none of you would be taken in by this sort of thing but
just to warn you there have been a few phishing emails directed at
various BitFolk addresses pretending to be from us, for example this
one today which was directed at the "users" mailing list (and was caught
by spam filters):
https://ibb.co/MZh7p7K
All emails that come from BitFolk should have SPF, DKIM and DMARC. If
you notice any that don't then please let us know.
And of course, we wouldn't send out emails asking for passwords.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Between about 20:25Z and ~20:50Z today host "Jack" lost all
networking. All of the VMs on it became unreachable.
It seems to have been some sort of kernel driver bug in the
Ethernet module as it was "stuck" not passing traffic but the
interface still showed as up.
The hosts have bonded network interfaces to protect against switch
failure, but as the interface stayed up this was not considered
failed. Also they are in active-backup mode and the currently-active
interface was the one that was stuck, so all traffic was trying to
go that way.
Networking was restored by setting the link down and up again.
Traffic started to flow again, BGP sessions re-established and all
was fine again.
We could look into some sort of link keepalive method on the bonded
interfaces as opposed to just relying on link state, but we have
already decided to move away from bonded networking in favour of
separate BGP sessions on each interface, That is how the next new
servers will be deployed; they will not have network bonding. We
have not yet tackled moving existing servers to this setup.
If we had been in the situation without bonding I think we would
have fared better here: there would have been a short blip while one
BGP session went down, but the other would remain and we'd be left
with some alerting and me scratching my head wondering why an
interface that is up doesn't pass traffic.
I will do some more investigation of this failure mode but in light
of doing away with bonding being the direction we are already going,
I don't think I want to alter how bonding is done on what will soon
be a legacy setup.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
An unauthenticated remote root exploit has been discovered in SSH,
including in versions shipped by Debian stable and newer, and most
other up to date Linux distributions.
https://security-tracker.debian.org/tracker/CVE-2024-6387
Please make sure you have applied the necessary upgrades.
If for some reason you are unable to apply an upgrade, the issue can
be mitigated by setting LoginGraceTime to 0 in /etc/ssh/sshd_config.
This will make it easier for people to tie up all connection slots,
denying access to legitimate connections, but does avoid the remote
root exploit.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
As you may be aware, the next LTS release of Ubuntu is supposed to
be ready in a couple of days.
I've tested a do-release-upgrade from a basic 22.04 cloud image
(what you get when you install 22.04 at BitFolk) and it seemed to go
fine. As usual with this sort of thing though, all the complexity is
in the packages you have installed, so that is no promise that it
would be plain sailing for you.
We will try to get a Xen Shell installer option added for 24.04 as
soon after release as we can, but in the mean time just installing
22.04 and then typing "sudo do-release-upgrade -d" should get you
there.
I *think* it is the case that you need the "-d" as
do-release-upgrade normally doesn't like doing it until the first
point release.
Thanks,
Andy
Ubuntu 24.04 LTS debtest1.vps.bitfolk.space hvc0
debtest1 login: ubuntu
Password:
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-31-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Tue Apr 23 13:31:00 UTC 2024
System load: 0.07 Memory usage: 6% Processes: 131
Usage of /: 14.6% of 19.20GB Swap usage: 0% Users logged in: 0
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
ubuntu@debtest1:~$ uname -a
Linux debtest1.vps.bitfolk.space 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat
Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
Since BitFolk moved datacentre from Telehouse to IP House, and since
some customers had previously actually asked about a renewable
energy statement, I've just updated it.
TL;DR: it's 59% now and they aim for it to be 100% by February 2025.
More detail:
https://tools.bitfolk.com/wiki/Renewable_energy_statement
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
