BitFolk Users

users@mailman.bitfolk.com

1117 discussions

Host "elephant" needed to be power cycled just now

by Andy Smith

Hi, At around 00:15Z we started receiving alerts regarding some servers on host "elephant". Looking at the machine's console it was reporting errors with its SAS controller, and was generally unresponsive to anything requiring block IO, so I had no choice but to power cycle it. On boot I couldn't find any issue with its SAS controller, and it was able to find all its storage devices and seemingly boot normally. The last few customer VPSes have finished booting as I type this. I will keep an eye on things for the next few hours and let you know about further actions. Please accept my apologies for the disruption. This is unrelated to the problems with "elephant" last month which were tracked down to a kernel bug. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 1 month

Developer discount (5%)

by Andy Smith

Hi, A few days ago someone asked if we would match a 5% discount that another hosting company offered to developers of significant open source projects. After thinking about it for a bit I decided we would. So, if you are a registered Debian/Ubuntu Developer, Fedora maintainer, BSD committer etc etc please feel free to email support(a)bitfolk.com and ask if you qualify. When you do, please provide some sort of link to your work so we can verify it. More information here: https://tools.bitfolk.com/wiki/Developer_discount Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 1 month

Some notes on BitFolk's plans to switch to PVH mode

by Andy Smith

Hello, This email doesn't discuss anything that you need to change (or in fact CAN change, at the moment), it's mostly a status update and fishing for feedback so can be safely ignored. As you may know, all BitFolk VPSes currently run under Xen PV (paravirtual) mode. We would like to switch to PVH mode instead. Here's some notes about our investigations of PVH mode: https://tools.bitfolk.com/wiki/PVH On the one hand we can get started right away: BitFolk's hypervisors support it, the newer stable releases of Debian and Ubuntu support it, the mainline kernel supports it, the rescue VM will work, etc etc. All new customers can be run in PVH mode as long as they don't choose an older release. On the other hand, it's not quite that simple as there's still a huge amount of existing customers whose kernels won't support it. I think it probably makes sense to immediately switch the Rescue VM to PVH mode, and make all installers also use it where the chosen install is known to support it. i.e. make it so that if you go and install Debian buster or testing or Ubuntu 20.04 right now, it silently switches you to PVH mode, because it's known to work. At the same time, we can add a Xen Shell command to toggle you between PV and PVH so you can try it out. If it doesn't work you could switch back, try again later at your own leisure etc. We don't know what Linux distribution you are running and can't tell without snooping on your console, so we can't make a good enough guess on your behalf. I don't really want to make a big thing of this because it's too complicated, so I'm thinking of hiding it away unless you need it. New customers / installs shouldn't have to think about it. It's only a concern for those of you with older, possibly 32-bit installs. For that reason I don't think I want to expose any of these details on the web panel or allow switching of the guest type there. On the subject of 32-bit PV support the deadlines are fairly laid back as — assuming you have no need of running a 5.9+ kernel — 32-bit PV booting could possibly¹ continue to work until 2023, which is the security support EOL for the last version of Xen that will support it. We're currently using Xen 4.10 and 4.12, and with the release of Debian 11 (bullseye) I will probably be looking to rolling upgrade to that with Xen 4.14. Any questions or thoughts on this? Cheers, Andy ¹ It is conceivable that some new CPU side channel attack lands and it's found that there is no way to stop a PV guest from snooping on the memory of the hypervisor or other PV guests. In that case there will be a sudden scramble to switch everyone. -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 1 month

Debian now has a "crossgrader" package

by Andy Smith

Hello, Perhaps of interest to those of you still on 32-bit installs¹, Debian now has a "crossgrader" package that has just had its first version uploaded to the "unstable" suite: https://salsa.debian.org/crossgrading-team/debian-crossgrading/ I have used this manual process: https://wiki.debian.org/CrossGrading many times to go from i386 to amd64, but it is unsupported and scary and depending on the exact packages you have installed can be very tricky. It will ask you to approve actions that may destroy your system. I haven't yet had chance to try "crossgrader" myself yet, but I will try it next time I have to do this. I would be interested in reading your experiences of using it. I gather that it would be wise to fully upgrade to the latest stable release (buster) on i386 before trying to crossgrade to amd64, either by this method or manually. For those keen to switch to 64-bit, much of the benefits can be obtained without most of the risk by only changing your kernel to the amd64 architecture. After following the wiki article above to the end of the "Install a kernel that supports both architectures in userland" section, you would: 1. Connect to Xen Shell and use the "console" command if not already in it. 2. Halt your VPS. 3. Use the Xen Shell "arch" command to switch to x86-64 bootloader. 4. Use the Xen Shell "boot" command to boot it. 5. When your own grub menu appears, select the amd64 kernel that is listed, not the i686 one. Provided everything seems to work okay you can then remove the i686 kernel packages, and the system will keep the -amd64 kernel packages up to date. Your VPS is then a 64-bit one running a 64-bit kernel but with almost entirely 32-bit userland. Changing over the 32-bit userland by either the crossgrader package or manually is where most of the complexity lies. For context as to why you might want to switch away from 32-bit: - The next major Xen release will remove 32-bit PV support. We'll switch to PVH mode before then to allow remaining 32-bit guests to still run. - At the recent Xen Developer summit it was stated that "anyone still running 32-bit PV is setting fire to 30% of their CPU". - All manner of 32-bit-specific fixes, including security, are being delayed and overlooked in the upstream Linux kernel, so switching away from running an i686 kernel would be a good idea. Cheers, Andy ¹ 41.1% of customer VMs, according to our database. -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 1 month

Server "elephant" has crashed a few times, ongoing problems

by Andy Smith

Hi, Server "elephant" unexpectedly crashed, then crashed twice more shortly after rebooting but before completely starting all VPSes. It is now crashing every time while trying to boot VPSes. I suspected bug in last round of XSA patches so reverted to previous hypervisor, but problem persists. We had an issue with "elephant" not so long ago so it might be hardware fault *though no logs to back this up). Still investigating, sorry. -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 2 months

Problem with ntpd after suspend/restore for hypervisor reboots

by Paul Lewis

Hi all, I’ve noticed that over the past few reboot cycles for security patches, my VM suspends and restores fine, and all services restore fine except ntp, which never recovers. When checking the status of the service I get: root@jaguar:~# service ntp status ● ntp.service - Network Time Service Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled) Active: inactive (dead) Docs: man:ntpd(8) Restarting the service restores everything back to working order. But I can’t always guarantee that I can get to the console quickly, so the icinga email alerts keep rolling in… If it makes a difference, I’m running Ubuntu 18.04.4, and ntp 1:4.2.8p10+dfsg-5ubuntu7.3. I presume the issue is the huge jump in time the kernel/ntp service sees when the VM is restored, is there a good way of getting ntpd to handle this? Do other people see this issue, and if so, what solutions/workarounds do you use to prevent it happening? Thanks, Paul

4 years, 2 months

Reboots will be necessary to address security issues, probably early hours 17/18/19 October

by Andy Smith

Hello, Unfortunately - and annoyingly only a month since the last lot - some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 20 October 2020. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 17, 18 and 19 October. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to migrate² you to already-patched hardware before the regular maintenance starts. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. It is disappointing to have another round of security reboots 28 days after the last lot, though before that there was a gap of about 330 days. Still, as there are security implications we have no choice in the matter. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 2 months

Issues receiving e-mails from BT Internet customers

by John Winters

Hi all, I run an e-mail server on one of my BitFolk VPSs. It seems fine except for one oddity which has arisen lately. Some e-mails from BT Internet customers take of the order of 3 weeks to arrive. Looking at the headers, they move around between a couple of BT server to begin with, then sit on the final one for up to 3 weeks before they are delivered to my server. I can see no failed earlier attempts to deliver in my logs, and my server receives e-mails fine from everyone else - just BT seems to be the problem, and not even all the e-mails coming from BT. Anyone else seen anything like this or can suggest a likely explanation? In case it's of relevance, BT's delivery still seems to be IPv4 only. My server advertises both IPv6 and IPv4 addresses. Cheers, John -- Xronos Scheduler - https://xronos.uk/ All your school's schedule information in one place. Timetable, activities, homework, public events - the lot Live demo at https://schedulerdemo.xronos.uk/

4 years, 2 months

Missing .well-known

by Hugh Frostick

My brain hurts. A domain failed to renew Let’s Encrypt today on Centos 7 running Virtualmin. No sign of .well-known directory under public_html. Had a look on all 4 VPS on Centos 7 and Centos 8 (two with Bitfolk, two overseas) and none of the sites I checked have a .well-known directory any more! Anyone seen this, or can offer a clue? Kind regards, Hugh

4 years, 2 months

Reminder: Reboots will be necessary to address security issues, early hours 17/18/19 October

by Andy Smith

Hi, A reminder that maintenance is scheduled for the early hours (UK time) of 17, 18 and 19 October. Irritatingly, this may end up having to be postponed. One of the patches has problems and the vendor is still working on that. If they come up with something in the next few hours I will still have time to test it appropriately, but if they don't then I won't and we'll have to postpone this maintenance for one week. Please assume it is going ahead unless you are notified otherwise. You should have all received a direct email telling you the hour long maintenance window that each of your VMs is in. If you can't find it please check your spam folders etc; it was sent on 7 October. If you still can't find it, work out which host machine you're on¹, and then the maintenance windows are: elephant 2020-10-17 00:00 hen 2020-10-18 02:00 hobgoblin 2020-10-18 01:00 jack 2020-10-19 00:00 leffe 2020-10-19 01;00 macallan 2020-10-17 02:00 paradox 2020-10-18 00:00 snaps 2020-10-19 02:00 talisker 2020-10-17 03:00 These times are all in UTC so add 1 hour for UK time (BST). Cheers, Andy ¹ This is listed on https://panel.bitfolk.com/ and is also evident from resolving <accountname>.console.bitfolk.com in DNS, e.g.: $ host ruminant.console.bitfolk.com ruminant.console.bitfolk.com is an alias for console.leffe.bitfolk.com. console.leffe.bitfolk.com is an alias for leffe.bitfolk.com. leffe.bitfolk.com has address 85.119.80.22 leffe.bitfolk.com has IPv6 address 2001:ba8:0:1f1::d ----- Forwarded message from Andy Smith <andy(a)bitfolk.com> ----- Date: Wed, 7 Oct 2020 09:20:29 +0000 From: Andy Smith <andy(a)bitfolk.com> To: announce(a)lists.bitfolk.com Subject: [bitfolk] Reboots will be necessary to address security issues, probably early hours 17/18/19 October User-Agent: Mutt/1.5.23 (2014-03-12) Reply-To: users(a)lists.bitfolk.com Hello, Unfortunately - and annoyingly only a month since the last lot - some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 20 October 2020. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 17, 18 and 19 October. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to migrate² you to already-patched hardware before the regular maintenance starts. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. It is disappointing to have another round of security reboots 28 days after the last lot, though before that there was a gap of about 330 days. Still, as there are security implications we have no choice in the matter. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting ----- End forwarded message ----- _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 2 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

BitFolk Users