BitFolk Users

users@mailman.bitfolk.com

1123 discussions

Server "elephant" has crashed a few times, ongoing problems

by Andy Smith

Hi, Server "elephant" unexpectedly crashed, then crashed twice more shortly after rebooting but before completely starting all VPSes. It is now crashing every time while trying to boot VPSes. I suspected bug in last round of XSA patches so reverted to previous hypervisor, but problem persists. We had an issue with "elephant" not so long ago so it might be hardware fault *though no logs to back this up). Still investigating, sorry. -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 5 months

Problem with ntpd after suspend/restore for hypervisor reboots

by Paul Lewis

Hi all, I’ve noticed that over the past few reboot cycles for security patches, my VM suspends and restores fine, and all services restore fine except ntp, which never recovers. When checking the status of the service I get: root@jaguar:~# service ntp status ● ntp.service - Network Time Service Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled) Active: inactive (dead) Docs: man:ntpd(8) Restarting the service restores everything back to working order. But I can’t always guarantee that I can get to the console quickly, so the icinga email alerts keep rolling in… If it makes a difference, I’m running Ubuntu 18.04.4, and ntp 1:4.2.8p10+dfsg-5ubuntu7.3. I presume the issue is the huge jump in time the kernel/ntp service sees when the VM is restored, is there a good way of getting ntpd to handle this? Do other people see this issue, and if so, what solutions/workarounds do you use to prevent it happening? Thanks, Paul

4 years, 5 months

Reboots will be necessary to address security issues, probably early hours 17/18/19 October

by Andy Smith

Hello, Unfortunately - and annoyingly only a month since the last lot - some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 20 October 2020. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 17, 18 and 19 October. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to migrate² you to already-patched hardware before the regular maintenance starts. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. It is disappointing to have another round of security reboots 28 days after the last lot, though before that there was a gap of about 330 days. Still, as there are security implications we have no choice in the matter. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 5 months

Issues receiving e-mails from BT Internet customers

by John Winters

Hi all, I run an e-mail server on one of my BitFolk VPSs. It seems fine except for one oddity which has arisen lately. Some e-mails from BT Internet customers take of the order of 3 weeks to arrive. Looking at the headers, they move around between a couple of BT server to begin with, then sit on the final one for up to 3 weeks before they are delivered to my server. I can see no failed earlier attempts to deliver in my logs, and my server receives e-mails fine from everyone else - just BT seems to be the problem, and not even all the e-mails coming from BT. Anyone else seen anything like this or can suggest a likely explanation? In case it's of relevance, BT's delivery still seems to be IPv4 only. My server advertises both IPv6 and IPv4 addresses. Cheers, John -- Xronos Scheduler - https://xronos.uk/ All your school's schedule information in one place. Timetable, activities, homework, public events - the lot Live demo at https://schedulerdemo.xronos.uk/

4 years, 5 months

Missing .well-known

by Hugh Frostick

My brain hurts. A domain failed to renew Let’s Encrypt today on Centos 7 running Virtualmin. No sign of .well-known directory under public_html. Had a look on all 4 VPS on Centos 7 and Centos 8 (two with Bitfolk, two overseas) and none of the sites I checked have a .well-known directory any more! Anyone seen this, or can offer a clue? Kind regards, Hugh

4 years, 5 months

Reminder: Reboots will be necessary to address security issues, early hours 17/18/19 October

by Andy Smith

Hi, A reminder that maintenance is scheduled for the early hours (UK time) of 17, 18 and 19 October. Irritatingly, this may end up having to be postponed. One of the patches has problems and the vendor is still working on that. If they come up with something in the next few hours I will still have time to test it appropriately, but if they don't then I won't and we'll have to postpone this maintenance for one week. Please assume it is going ahead unless you are notified otherwise. You should have all received a direct email telling you the hour long maintenance window that each of your VMs is in. If you can't find it please check your spam folders etc; it was sent on 7 October. If you still can't find it, work out which host machine you're on¹, and then the maintenance windows are: elephant 2020-10-17 00:00 hen 2020-10-18 02:00 hobgoblin 2020-10-18 01:00 jack 2020-10-19 00:00 leffe 2020-10-19 01;00 macallan 2020-10-17 02:00 paradox 2020-10-18 00:00 snaps 2020-10-19 02:00 talisker 2020-10-17 03:00 These times are all in UTC so add 1 hour for UK time (BST). Cheers, Andy ¹ This is listed on https://panel.bitfolk.com/ and is also evident from resolving <accountname>.console.bitfolk.com in DNS, e.g.: $ host ruminant.console.bitfolk.com ruminant.console.bitfolk.com is an alias for console.leffe.bitfolk.com. console.leffe.bitfolk.com is an alias for leffe.bitfolk.com. leffe.bitfolk.com has address 85.119.80.22 leffe.bitfolk.com has IPv6 address 2001:ba8:0:1f1::d ----- Forwarded message from Andy Smith <andy(a)bitfolk.com> ----- Date: Wed, 7 Oct 2020 09:20:29 +0000 From: Andy Smith <andy(a)bitfolk.com> To: announce(a)lists.bitfolk.com Subject: [bitfolk] Reboots will be necessary to address security issues, probably early hours 17/18/19 October User-Agent: Mutt/1.5.23 (2014-03-12) Reply-To: users(a)lists.bitfolk.com Hello, Unfortunately - and annoyingly only a month since the last lot - some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 20 October 2020. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 17, 18 and 19 October. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to migrate² you to already-patched hardware before the regular maintenance starts. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. It is disappointing to have another round of security reboots 28 days after the last lot, though before that there was a gap of about 330 days. Still, as there are security implications we have no choice in the matter. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting ----- End forwarded message ----- _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 5 months

Virtualmin on CentOS 8

by Hugh Frostick

Hi All, Not sure if this is a valid thing to post here. Bit long sorry. I have a VPS running Virtualmin on Centos 8 here no probs: Linux version 5.7.10-1.el8.elrepo.x86_64 (mockbuild@072389ed160c4f05a56ad0894a89bf2f) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC), GNU ld version 2.30-73.el8) I have a new VPS in New Zealand with a good supplier (call it NZS) - only a few levels behind Andy :-) When I set up Virtualmin on a new VPS there on CentOS 8, Virtualmin does not work. Linux version 4.19.144-rh174-20200910010734.xenU.x86_64 (tomcat(a)ci.build.***hosting.com) (gcc version 8.3.0 (Debian 8.3.0-6)) NZS has option to change kernel in their excellent customer control panel: Kernel Architecture Comments default-4.14.xenU.x86_64 x86_64 After a VM reboot, use the latest kernel in the 4.14.197-rh229-20200910022519.xenU.x86_64 series. default-4.19.xenU.x86_64 x86_64 After a VM reboot, use the latest kernel in the 4.19.144-rh174-20200910010734.xenU.x86_64 series. Booted on 4.19.144-rh174-20200910010734.xenU.x86_64 default-5.4.xenU.x86_64 x86_64 After a VM reboot, use the latest kernel in the 5.4.64-rh70-20200910022002.xenU.x86_64 series. 4.14.197-rh229-20200910022519.xenU.x86_64 x86_64 lts kernel 4.19.144-rh174-20200910010734.xenU.x86_64 x86_64 lts kernel, for both 64bit and 32bit installs 5.4.64-rh70-20200910022002.xenU.x86_64 x86_64 lts kernel, for both 64bit and 32bit installs VM installed kernel via PV Grub 4.7.2.pv-grub.x86_64 x86_64 64bit only. Lets you boot your own kernel. I switched to the 5.4 and no difference. I got onto NZS support with this message: *** I want to run Virtualmin under Centos 8 on my new VPS. (Currently I have a VPS running Virtualmin under Centos 7.) I have reinstalled a few times, full vps and also virtualmin manually and from your install package, and tried various things but get these issues: • SuExec cannot be used to run PHP scripts in CGI or FCGId modes : The Suexec command on your system is configured to only run scripts under /var/www, but the Virtualmin virtual server home directory is /home. CGI scripts run as domain owners will not be executed. • The following PHP-FPM versions cannot be used : 7.2.24 (Apache module mod_proxy is missing or not enabled) My searches finally turned up this: https://www.virtualmin.com/documentation/installation/faq Q. "I installed manually or using packages from a third party source, and I have the following error after install: "The Suexec command on your system is configured to only run scripts under /var/www, but the Virtualmin virtual server home directory is /home. CGI and PHP scripts run as domain owners will not be executed” A. The Apache suexec command on your system is misconfigured for use in a virtual hosting environment, and needs to be recompiled or configured (on systems that provide a configurable suexec command) with the docroot set to /home. On Debian/Ubuntu systems, you can install the apache2-suexec-custom package, and modify /etc/apache2/www-data to include /home. On other systems, you will need to recompile the Apache package or the suexec binary. Or, you can use our automated install script, which insures a correctly configured suexec binary is installed. Am I on the right track here, and can you please help with suexec problem? *** Reply from NZS (wihtin 4 hours) " After a quick look through, this is more than just a couple bugs. There are some seriously broken things by default and it does not work, and is not easily fixable. Things that do not work out of the box are 1. Dav - easily disabled and not used often (HF: don’t care abotu this myself) 2. PHPFCGID - suexec. you need to use this or PHP-FPM, this is broken because the chroot is /var/www 3. php-FPM - this is the deal breaker, it needs mod_proxy, but even when enabled it fails to detect it , so that does not work At this stage I'm going to suggest we do not allow installers of the Centos 8 to have virtualmin install as an option since its far more broken than a small thing. I would suggest you move to a debian based distro (ubuntu or similar) and virtualmin, since we know these work well without issue." *** So now, looking for recommendations. Server has to be in NZ and with that supplier who I like. I know Virtualmin CAN work with CentOS8 because it does here. The options are: 1. Wait and investigate - but was hoping to move everything over from an old server this weeek before end of month renewal where I really don’t want the supplier to have any more money, they have gone really bad. 2. Choose Centos7 that works with their build - ony 4 years life though? Seems like I am making an unnecessary extra problem for myself? 3. Can I install my own distro same as Andy has, somehow? 4. Ideas? Cheers Hugh

4 years, 6 months

Reboots will be necessary to address security issues, probably early hours 19/20/21 September

by Andy Smith

Hello, Unfortunately some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 22 September 2020. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 19, 20 and 21 September. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to almost-live migrate you to already-patched hardware before the regular maintenance starts. You can expect a few tens of seconds of pausing in that case. This is still a somewhat experimental process and also requires you to opt in to suspend and restore. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 6 months

Pushover notifications now available (beta)

by Andy Smith

Hi, I make use of Pushover myself: https://pushover.net/ A customer asked for non-email notifications from BitFolk's monitoring and I suggested Pushover for them too. I've now added the necessary bits to generalise it for any customer. If you would like Pushover notifications then please mail support(a)bitfolk.com to ask for them. When you ask please: - supply your Pushover User Key (visible in your Pushover dashboard) - state whether you want it for just host notifications (is the host down or unreachable?) or for service notifications as well (is service X on host Y in a non-OK state?). Services might generate quite a few notifications. If you only want SOME services to generate these notifications, you can let us know what those are too. Limitations: - Will only send a notification on a non-OK state, so you won't receive RECOVERY/OK notifications. This is in the interest of not deluging you with notifications. - Priority is always set to 1 (high), which means it will try to make a noise and vibrate your mobile device. You can override that in your mobile device. The full API does allow an emergency priority which would require you to acknowledge it, and three quieter options, but I'm not ready to support any of those yet. - You can't currently have Pushover notifications without email notifications too. - You can't tell monitoring to stop sending you Pushover notifications but carry on sending email ones. Anything you'd usually do to stop notifications will stop both kinds. If more than a few people request this then I will integrate it with the address book in the panel so you can get Pushover notifications by adding a contact with a Pushover key there. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

4 years, 6 months

What do you expect to happen when you authorise a Direct Debit mandate?

by Andy Smith

Hello, There is a frequent cause of confusion with Direct Debit. The way Direct Debit works is, first you authorise an organisation to make Direct Debit requests. This is called a Direct Debit mandate. That gets sent to your bank. Once your bank accepts it the organisation can then request funds from your bank. At present BitFolk only requests funds by Direct Debit when there is a mandate in place and: a) A new invoice is created, or; b) You go into the Panel and select one or more invoices to pay by Direct Debit. Notably what does NOT happen ever is that invoices which already exist are suddenly submitted by Direct Debit. So what happens quite often is: 1. Invoice for ongoing service is created and emailed to customer. 2. Customer ignores this for some period of time. 3. A nagging automated email is sent saying this is going to be due soon. Quite often ignored for some time. 4. A more strident yet still automated nagging email is sent saying that it would be a really good idea to consider paying this now as otherwise there might be a loss of service. 5. Customer decides that Direct Debit would be convenient and does authorise a Direct Debit mandate now, but doesn't actually pay the outstanding invoice(s) by any means because they think they have now told us to take payments by Direct Debit and that it will just happen. 6. Invoice is now weeks overdue and an automated email is sent out saying that the service is now going to be suspended for non-payment. 7. Since non-payment suspensions are manual, we think to check if the customer recently authorised a Direct Debit mandate. If they did then we consider it likely that they thought that would do the job, so we have to contact them and explain and ask if they did actually want this existing invoice paid by DD. This is annoyingly manual, takes time, and is sometimes hard to explain. There have been a non-zero number of occasions where we have forgotten to check for a mandate in step #7 and have suspended the customer's service for non-payment. There have been many occasions where customers have received the "you're being suspended for non-payment" email of step #6 and contacted us in a panic. Every time there is one of these misunderstandings I explain why this happened and ask how they would like it to be changed so that it doesn't happen any more, but sadly I have never really received any concrete suggestions even from the people it has happened to. I'm pretty sick of this happening so I want to do something about it. So, I shall ask all of you, how would you expect it to work? a) As soon as a mandate is authorised, just charge all existing invoices immediately Very tempting. Very simple. I fear there will be at least one person that will claim they never expected that to happen, and a returned Direct Debit has caused them to incur an eleventy billion pound penalty charge from their bank, their mortgage payment got rejected, and now there are men outside in shiny leather jackets. b) As soon as the mandate is authorised, if the customer has existing invoices that are unpaid, there is a very noticeable message on the screen like: You seem to have unpaid invoices: #41234 £107.88 #41239 £1.92 Pre-existing invoices won't be automatically submitted for payment by Direct Debit. You can <a href="…">pay them now</a> by a one-off Direct Debit or any of our other supported payment methods. I like (b). I am open to other ideas if you have any. I can't really think of any. I understand that many people will be happy with (a), but I feel it's one of those things that when there is someone that is unhappy, they are very unhappy, and that wipes out the good feelings from the many more people that never had a problem. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

4 years, 6 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

BitFolk Users