BitFolk Announcements

announce@mailman.bitfolk.com

259 discussions

Debian 12 bookworm - a reminder about eth0 → enX0

by Andy Smith

Hi, As you may be aware, the latest stable release Debian 12 bookworm was released on 10 June. It is available for new orders and you can of course upgrade your existing Debian VMs to this release, but we haven't yet updated the web site to reflect this nor the Xen Shell to allow a clean self-install. That will happen in the next couple of days. A reminder though, that in this new release udev has learned about Xen network interfaces and therefore will rename your eth0 to enX0. This was previously discussed here: https://mailman.bitfolk.com/mailman/hyperkitty/list/users@mailman.bitfolk.c… Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

11 months, 1 week

The SPF record for bitfolk.com was tightened from ~all to -all on 3 June 2023

by Andy Smith

Hi, After a minor incident of some phishing emails being sent purporting to be from bitfolk.com, on 3 June I tightened the SPF record of bitfolk.com from ~all to -all. This basically says that ONLY the hosts listed in the SPF record are permitted to use a bitfolk.com envelope sender on email, and that any other host trying to do so should be rejected. I have since noticed that some customers are using traditional server-side forwarding, e.g. on role addresses, to send BitFolk emails to a group of people, and some of those recipients are doing as asked and rejecting the email. More are probably silently discarding or filing the mails away in spam/junk folders. This happens because when your mail server forwards an email from e.g. billing(a)bitfolk.com through role(a)yourdomain.co.uk and out to joe.bloggs(a)example.com, your mail server is pretending to be billing(a)bitfolk.com. Since you do not match bitfolk.com's SPF record, the mail server for example.com rejects the email (unless configured otherwise). Unfortunately we can't go back on this configuration. It's just the way that email works in the 21st century. Server-side forwarding of email in this way is not something that can be expected to work any more, unless you control both the recipient address and every address it expands out to. So what can you do if you currently do forwarding of addresses like this? - The generic answer is Snder Rewriting Scheme: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme - A more limited answer is to run a real mailing list of some sort, so that, for example, billing(a)yourdomain.co.uk goes through a real mailing list manager like majordomo or Mailman, is rewritten and sent out to the people who should receive it. - If you control or have strong influence on all recipients, you can configure them to allowlist particular senders. - You can set up real mailboxes for your role accounts and have interested parties download the email by IMAP or POP. I'm sorry that this change has broken some previously-working forwarding setups. It isn't something we can revert though (and indeed, it will have to get stricter, with additional DKIM and DMARC to come). Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

11 months, 1 week

Secondary nameservers changed from PowerDNS to BIND - beware your expire timers

by Andy Smith

Hello, If you do not make use of BitFolk's secondary DNS service then you can safely skip this email. Over the last few days we've upgraded the servers used for our secondary DNS service and also switched software from PowerDNS to BIND. On the whole nothing changes, but there is one thing I would like to draw your attention to. BIND actually pays attention to the expire timers that you set in your SOA records whereas PowerDNS does not. An SOA record looks like this: $ dig +multi +noall +answer -t soa bitfolk.com bitfolk.com. 86383 IN SOA a.authns.bitfolk.co.uk. hostmaster.bitfolk.com. ( 2023042101 ; serial 14400 ; refresh (4 hours) 7200 ; retry (2 hours) 1209600 ; expire (2 weeks) 43200 ; minimum (12 hours) ) The "expire" timer tells authoritative DNS servers how long the records they hold are valid for, if they have not been able to contact the primary nameservers. In the above example, should the primary nameserver be unreachable, any secondary nameservers that are still responding will serve the zone content for a further two weeks. After that time they will respond with SERVFAIL. Compliant DNS client behaviour is to retry any other servers when that happens. PowerDNS does not implement these "expire" semantics and always answers queries. In watching logs carefully over the last few days I have seen that some of you have extremely short expire timers. I'm not sure whether you intend for that to be the case. For example, there are many zones currently on BitFolk's servers with an expire time of 300 seconds. That means that you are indicating that the entire zone should not be served 5 minutes¹ after your primary server stops responding. It doesn't seem likely to me that you really want all authoritative servers for your domain to stop working 5 minutes into any sort of outage. Since the secondary servers will now really believe you on this, I urge you to review your expire timers. If in doubt please put your domain name into this: https://zonemaster.net/en/run-test and it'll advise you if any of those timers seem wrong. Cheers, Andy ¹ Different DNS server implementations actually behave slightly differently here. As mentioned, PowerDNS doesn't handle expire timers at all. BIND has a minimum of 300 seconds, or the refresh+retry timers, whichever is larger. So in fact the shortest expire behaviour I can see at the moment is 600 seconds (10 minutes). Which still seems unusually short. See: https://jpmens.net/2022/01/14/fun-with-the-dns-soa-expire-field/ for more info about how different implementations treat the expire timer. -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year

2023-04-19: b.authns maintenance

by Andy Smith

Hi, Today and tonight (UK time) we're going to be doing some work on b.authns.bitfolk.com (secondary DNS service). I'm going to disable alerts for it and stop it from responding (so it can't give any incorrect answers). The other servers will remain up and working; I'm just letting you know in case you notice it is intermittently down. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year

There are currently problems with host "talisker"

by Andy Smith

Hi, We are experiencing some problems with host "talisker" and I'm currently looking into it. We are likely going to have to do an emergency reboot in a monent. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year, 5 months

If running CentOS, check your grub.cfg as your VM may not boot otherwise

by Andy Smith

Hi, I held off posting this to the "announce" list because I don't understand the problem nor fully how to avoid/fix it, but another CentOS-using customer has today ended up unbootable through not getting the warning so I think I better had. Have a read of this: https://mailman.bitfolk.com/mailman/hyperkitty/list/users@mailman.bitfolk.c… Basically it seems at some point some CentOS update without notice converts your grub.cfg to use "BLS" which is a thing that's Red Hat specific and not available in upstream Grub. Since BitFolk uses upstream Grub to boot your VMs, this leaves your VM unbootable. There's info in that post on how to discover if this has been enabled, and how to disable it, but it's unclear to me: - what does it - if it can actually be reverted once it's happened So if you know more, please do let us know. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year, 10 months

BitFolk's mailing lists have migrated to Mailman 3

by Andy Smith

Hi, == TL;DR: == The BitFolk mailing lists changed address, but important stuff got copied over. You don't need to do anything. The new list archives are at: https://mailman.bitfolk.com/mailman/hyperkitty/ and if you DO want to change any of your settings the main interface is at: https://mailman.bitfolk.com/mailman/postorius/lists/ But you'll have to do a Mailman password reset before you can log in to it and change settings. You can stop reading now unless you are particularly interested in the details. == What == BitFolk's announce@ and users@ mailing lists which used to be at lists.bitfolk.com have now been migrated to <listname>@mailman.bitfolk.com, which is Mailman 3. == Why == The VM which runs lists.bitfolk.com needs to be upgraded. It runs Mailman 2, which is a strictly Python 2 application. Python 2 was EOL in 2020 sometime and is not available in newer releases of Debian, and so neither is Mailman 2. An upgrade will leave Mailman 2 non-functional. Running Mailman 3, which is a complete rewrite, is a major undertaking that I have little experience of and I'm not entirely happy about it. However, unless we either want to outsource the mailing lists to another provider or switch to something like Discourse¹, it seems to be still the best option. While it may have been possible to install Python 2 and Mailman 2 from source on an otherwise upgraded host, that is a lot of work for what seems like a dead end. == Your subscriptions == All subscribed addresses have been copied over, which is why you are seeing this email. You don't need to do anything, but if you do want to adjust your settings you'll need to do a password reset in Mailman as your passwords have not been copied over. The web interface for the lists is at: https://mailman.bitfolk.com/mailman/postorius/lists/ In the footer of messages to the "users" list it will have a link and tell you which address you are subscribed with. The "announce" list does not have a footer so if you're unsure what your subscribed address is there you'll need to look at the email headers. == Functional changes == === Headers === I'm tired of list postings being rejected or going to spam folders. We can't keep sending emails on your behalf, i.e. with a From: address that is yours, because that breaks DKIM. So over on the "users" list all posts will appear to come from: From: Your Name via BitFolk Users <users(a)mailman.bitfolk.com> Cc: Your Name <you(a)example.com> The Cc header is so that people can still contact you off-list if they want. Obviously the List-Id: headers have changed so if you're filtering email based upon those you will need to update your filters. Sorry about that. === Reply-To: added, no more "users-replyto" === We used to have a separate mailing list "users-replyto" which got all the same emails as the "users" list, but added: Reply-To: users(a)lists.bitfolk.com to them for people who preferred to explicitly have replies directed back to the list. The main "users" list is going to have an explicit Reply-To: added now, making "users-replyto" redundant. Anyone who was subscribed to "users-replyto" has been added to "users". The reason for this is that I got feedback that despite it being a long held belief that forcing Reply-To: is bad², these days most people do prefer it. I'm happy to revisit that decision if people don't like it, but the "users-replyto" hack won't be coming back. === List addresses === Obviously the list addresses have both changed from @lists.bitfolk.com to @mailman.bitfolk.com. This was necessary so that the older Mailman could continue working while the new one was set up. No doubt through muscle memory and address books the occasional email will still end up being sent to whatever(a)lists.bitfolk.com. These emails are being redirected to the correct place and I anticipate keeping that redirection in place forever. === List archives === The public archives of the "announce" and "users" lists have been imported into Mailman3's archiver, HyperKitty, at: https://mailman.bitfolk.com/mailman/hyperkitty/ I tried to port over the existing Lurker archives but I just couldn't get it to work. Lurker is mostly undocumented and hasn't seen any development since 2007. This unfortunately means that all the old archive links that are out there in previous emails and in the wiki, etc. will break if and when I turn off Lurker on the old host. I hope to be able to transplant a static copy of https://lists.bitfolk.com/lurker/ over to https://mailman.bitfolk.com/lurker/ so that old links will still work, with automatic redirection. There doesn't appear to be any way to map an individual Lurker message link to a HyperKitty link, sadly. The new HyperKitty archives are fully searchable, so if you do happen to see a Lurker link in a wiki article please do feel free to find the same message in HyperKitty and update the article. If you find it in some other place that you can't edit please do just let me know and I'll update it. The same goes for any other references to lists.bitfolk.com. == Let us know about any issues or suggestions for the lists == As I say I'm not very experienced in running Mailman 3 so it's possibly it might break or be misconfigured in some way that I haven't yet found. So if you spot anything that you think looks broken, please do let us know. Also if you have suggestions for how it might work differently or look differently, I am interested to hear and try to implement, though I cannot promise that it is possible. Certainly I am not in love with the default appearance of the HyperKitty archives or the colour scheme of Mailman 3's web interface in general, and it may be possible to improve those. Thanks, Andy ¹ Discourse: https://www.discourse.org/ ² https://marc.merlins.org/netrants/reply-to-harmful.html and its counter-argument https://marc.merlins.org/netrants/reply-to-useful.html Both of these are over 20 years old so I'm not sure how relevant they are (or that mailing lists are) for the modern Internet. It should also be noted that one of the objections to Reply-To: munging is that it can remove all information about where the author wants replies to go. In Mailman3's case, if you supply Reply-To: headers of your own then it adds them, so that objection doesn't hold. -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year, 10 months

Another reminder that Debian 11 (bullseye) and beyond WILL NOT WORK in 32-bit PV mode

by Andy Smith

Hi, TL;DR: If you're running in 32-bit PV mode you should not try to update your kernel to version 5.9 or beyond, which means not to Debian 11. Take some action first, like switching to PVH mode. Otherwise it will not boot and you'll have to boot into the older kernel again. During this week's maintenance a worrying number of customer VMs didn't boot because they were 32-bit PV and had been updated to a kernel version of 5.9 or beyond. Debian 11 (bullseye) has kernel version 5.10. It's been mentioned a number of times here over the years that support for 32-bit PV in the Linux kernel was going away at version 5.9, but it looks like we haven't been able to communicate that as well as we would have liked. The most recent attempt was here: https://lists.bitfolk.com/lurker/message/20210930.104643.2ab5f9c0.en.html If 32-bit PV is you then probably what you want to do is just switch to PVH mode as mentioned at the top of that email. Possibly the remaining people who need to see this just aren't on the announce@ list. We could email every customer who is running 32-bit PV mode, but it would probably just serve to annoy those of you who are never planning to update your kernels. Finally, although switching to PVH should allow you to continue for many years to come, nobody should be running 32-bit. We strongly advise you to be planning your switch to amd64 as soon as possible [advice since 2015]. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year, 11 months

Reboots will be necessary to address security issues, probably early hours 7/8/9 June

by Andy Smith

Hello, Unfortunately some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 9 June 2022. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 7, 8 and 9 June. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to migrate² you to already-patched hardware before the maintenance is scheduled for your host. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year, 11 months

DNS secondary zone serial consistency check is now enabled

by Andy Smith

Hi, As requested this check is now enabled for all customer DNS zones in the secondary DNS service. The alert will look like this: https://tools.bitfolk.com/wiki/Secondary_DNS_service#Zone_serials_match If you receive it then the first thing to do is to check your nameserver's logs as it will normally be due to an issue that (only) you can fix yourself. Cheers, Andy ----- Forwarded message from Mike Zanker via users <users(a)lists.bitfolk.com> ----- […] >> Maybe we should add some serial number monitoring, so if your zone >> serial number changes but ours doesn't (because AXFR failed) then >> that difference would be an alert. I do like the sound of that, Andy (Smith). ----- End forwarded message ----- -- https://bitfolk.com/ -- No-nonsense VPS hosting

1 year, 12 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BitFolk Announcements