BitFolk Users

users@mailman.bitfolk.com

1117 discussions

by Ian Hobson

Hi all, I was working on my VM about an hour ago, and suddenly lost all connection to my VM on macallan.bitfolk.com the recovery machine .console.bitfolk.com the bitfolk website. All back now. It was not local to here (Bangkok) because I could reach every other site I tried. I would like to know what happened? I'm guessing something got to hot on this hot day in London, and let out its magic smoke. Regards Ian -- Ian Hobson Tel (+66) 626 544 695

2 years, 5 months

Help with Apache serving incorrect VirtualHost DocumentRoot

by Paul Lewis

Hi all, I’ve been banging my head against this issue for a couple of weeks and I’m hoping someone here can provide some guidance or sugestions. Some Background I run several WordPress sites on my Bitfolk VPS, and I have AWS CloudFront in front of them. Because CloudFront serves the www.domain.name<http://www.domain.name> URLs, I have origin-www.domain.name DNS entries pointing to my VPS, and Apache VirtualHosts configured that respond to multiple ServerNames and ServerAliases: * domain.name * origin-www.domain.name * www.domain.name * www-test.domain.name On the VPS I am serving the websites over HTTP and HTTPS (using Let’s Encrypt for certificates), but CloudFront is configured to redirect HTTP requests to HTTPS (using AWS Certificate Manager certificates), and to only communicate with the origin over HTTPS. The Apache VirtualHosts are configured to redirect any domain name to https://www.domain.name, which is the URL that each WordPress site is configured to use (for the WordPress Address and the Site Address). Now to the problem I see different behaviours with the redirects on different websites. For example, if I test all the URI combinations for one of my sites using a site like https://httpstatus.io/, I get: 1. http://wofeckcsclub.co.uk --> https://wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads 2. http://origin-www.wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads 3. http://www.wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads 4. http://www-test.wofeckcsclub.co.uk --> https://www-test.wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads 5. https://wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads 6. https://origin-www.wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads 7. https://www.wofeckcsclub.co.uk --> website loads 8. https://www-test.wofeckcsclub.co.uk --> https://www.wofeckcsclub.co.uk --> website loads On the whole, this is fine. I’d rather 1) and 4) had a single redirect to https://www.wofeckcsclub.co.uk but I think this is a result of CloudFront first redirecting to HTTPS and then the HTTPS site redirecting to the correct URI, so not sure I can do much about this (except maybe enable CloudFront to communicate with the origin via HTTP, and allow the HTTP origin site to redirect to the correct URI). However, a few of my websites do this instead: 1. http://savouredescapes.com --> https://savouredescapes.com --> https://www.savouredescapes.com --> website loads 2. http://origin-www.savouredescapes.com --> https://www.savouredescapes.com --> website loads 3. http://www.savouredescapes.com --> https://www.savouredescapes.com --> website loads 4. http://www-test.savouredescapes.com --> https://www.savouredescapes.com --> website loads 5. https://savouredescapes.com --> https://www.savouredescapes.com --> website loads 6. https://origin-www.savouredescapes.com --> website loads [not correct] 7. https://www.savouredescapes.com --> website loads 8. https://www-test.savouredescapes.com --> website loads [not correct] For these websites, URIs 6) and 8) are not being redirected, and display the website with the original URI in the browser. All the VirtualHosts are configured the same, and look like this: <VirtualHost *:80> DocumentRoot /var/www/savouredescapes ServerName www.savouredescapes.com ServerAlias savouredescapes.com origin-www.savouredescapes.com www-test.savouredescapes.com <Directory /var/www/savouredescapes/> AllowOverride All </Directory> ErrorDocument 403 /error/noindex.html ErrorLog ${APACHE_LOG_DIR}/savouredescapes-error.log CustomLog ${APACHE_LOG_DIR}/savouredescapes-access.log combined RewriteEngine on RewriteCond %{SERVER_NAME} =savouredescapes.com [OR] RewriteCond %{SERVER_NAME} =origin-www.savouredescapes.com [OR] RewriteCond %{SERVER_NAME} =www.savouredescapes.com [OR] RewriteCond %{SERVER_NAME} =www-test.savouredescapes.com RewriteRule ^ https://www.savouredescapes.com%{REQUEST_URI} [END,NE,R=permanent] </VirtualHost> As far as I can tell, a URI like https://origin-www.savouredescapes.com should be hitting my VPS direct (no CloudFront, DNS points to my VPS) and getting redirected to https://www.savouredescapes.com, which should then be served via CloudFront. However, Apache doesn’t seem to be performing this redirect. In the Apache logs, I see entries like this in the ‘access’ log: 188.166.109.207 - - [09/Jul/2022:11:19:04 +0100] "GET / HTTP/1.1" 200 58957 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0" As opposed to something like this when the redirect does work: 188.166.109.207 - - [09/Jul/2022:11:11:18 +0100] "GET / HTTP/1.1" 301 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0" And nothing in the error logs. So I’m not sure how to troubleshoot further why the redirects aren’t working as expected for this particular website. In addition, because this does work on some of my other sites (like the https://www.wofeckcsclub.co.uk site I first showed the results for), I presume WordPress (or something else?!?) must be doing an internal redirect, rather than Apache? Again, as far as I can tell the WordPress configurations are the same; there are no obvious plugins that would configure 301/302 redirects on the WordPress instances that do work, although that’s not to say such a plugin hasn’t been installed and configured in the past and since removed, leaving the configuration running on the WordPress site…? I’m wondering if anyone on this list has any relevant WordPress experience, and any suggestions for files/configurations to check in WordPress to see where these redirects might be happening? Also more than happy to explore the fact that my Apache configuration could be wrong and need tuning, and open to any other suggestions to try and ‘fix’ these redirects one way or another (the main thing I want is consistency across all of them!). Many thanks in advance, Paul

2 years, 5 months

If running CentOS, check your grub.cfg as your VM may not boot otherwise

by Andy Smith

Hi, I held off posting this to the "announce" list because I don't understand the problem nor fully how to avoid/fix it, but another CentOS-using customer has today ended up unbootable through not getting the warning so I think I better had. Have a read of this: https://mailman.bitfolk.com/mailman/hyperkitty/list/users@mailman.bitfolk.c… Basically it seems at some point some CentOS update without notice converts your grub.cfg to use "BLS" which is a thing that's Red Hat specific and not available in upstream Grub. Since BitFolk uses upstream Grub to boot your VMs, this leaves your VM unbootable. There's info in that post on how to discover if this has been enabled, and how to disable it, but it's unclear to me: - what does it - if it can actually be reverted once it's happened So if you know more, please do let us know. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 5 months

LXC containers on a Bitfolk VM

by Ian Hobson

Hi All, I would like to simplify my VM, and make it more secure by using LXC containers. E.g. nginx for vanilla websites Special nginx compile for webrtc stun and turn. Email forwarder MySQL server Has anyone tried this? Does it work? What about IP addresses? Any hints tips or advice welcome. I will naturally be testing on a VM here at home, but that will be under Vbox, which might not work like XEN. I hesitate to start playing with the live config again, as last time I brought down all https sites for a week! :( Regards Ian -- Ian Hobson

2 years, 5 months

TCP 5900 no longer okay for Tor exit nodes

by Andy Smith

Hi, There's been a huge increase in the number of abuse reports we are receiving for dictionary attacks against VNC servers through Tor exit nodes at BitFolk. I can't see any good reason why someone would want to use VNC through Tor, so we've added port 5900 to the required exit policies: https://tools.bitfolk.com/wiki/Running_a_Tor_node#Required_exit_policies All those known to be running Tor exit nodes at BitFolk are being contacted directly to ensure this change is made, but we don't always know if a customer is running one (until the abuse reports come in). Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 5 months

BitFolk's mailing lists have migrated to Mailman 3

by Andy Smith

Hi, == TL;DR: == The BitFolk mailing lists changed address, but important stuff got copied over. You don't need to do anything. The new list archives are at: https://mailman.bitfolk.com/mailman/hyperkitty/ and if you DO want to change any of your settings the main interface is at: https://mailman.bitfolk.com/mailman/postorius/lists/ But you'll have to do a Mailman password reset before you can log in to it and change settings. You can stop reading now unless you are particularly interested in the details. == What == BitFolk's announce@ and users@ mailing lists which used to be at lists.bitfolk.com have now been migrated to <listname>@mailman.bitfolk.com, which is Mailman 3. == Why == The VM which runs lists.bitfolk.com needs to be upgraded. It runs Mailman 2, which is a strictly Python 2 application. Python 2 was EOL in 2020 sometime and is not available in newer releases of Debian, and so neither is Mailman 2. An upgrade will leave Mailman 2 non-functional. Running Mailman 3, which is a complete rewrite, is a major undertaking that I have little experience of and I'm not entirely happy about it. However, unless we either want to outsource the mailing lists to another provider or switch to something like Discourse¹, it seems to be still the best option. While it may have been possible to install Python 2 and Mailman 2 from source on an otherwise upgraded host, that is a lot of work for what seems like a dead end. == Your subscriptions == All subscribed addresses have been copied over, which is why you are seeing this email. You don't need to do anything, but if you do want to adjust your settings you'll need to do a password reset in Mailman as your passwords have not been copied over. The web interface for the lists is at: https://mailman.bitfolk.com/mailman/postorius/lists/ In the footer of messages to the "users" list it will have a link and tell you which address you are subscribed with. The "announce" list does not have a footer so if you're unsure what your subscribed address is there you'll need to look at the email headers. == Functional changes == === Headers === I'm tired of list postings being rejected or going to spam folders. We can't keep sending emails on your behalf, i.e. with a From: address that is yours, because that breaks DKIM. So over on the "users" list all posts will appear to come from: From: Your Name via BitFolk Users <users(a)mailman.bitfolk.com> Cc: Your Name <you(a)example.com> The Cc header is so that people can still contact you off-list if they want. Obviously the List-Id: headers have changed so if you're filtering email based upon those you will need to update your filters. Sorry about that. === Reply-To: added, no more "users-replyto" === We used to have a separate mailing list "users-replyto" which got all the same emails as the "users" list, but added: Reply-To: users(a)lists.bitfolk.com to them for people who preferred to explicitly have replies directed back to the list. The main "users" list is going to have an explicit Reply-To: added now, making "users-replyto" redundant. Anyone who was subscribed to "users-replyto" has been added to "users". The reason for this is that I got feedback that despite it being a long held belief that forcing Reply-To: is bad², these days most people do prefer it. I'm happy to revisit that decision if people don't like it, but the "users-replyto" hack won't be coming back. === List addresses === Obviously the list addresses have both changed from @lists.bitfolk.com to @mailman.bitfolk.com. This was necessary so that the older Mailman could continue working while the new one was set up. No doubt through muscle memory and address books the occasional email will still end up being sent to whatever(a)lists.bitfolk.com. These emails are being redirected to the correct place and I anticipate keeping that redirection in place forever. === List archives === The public archives of the "announce" and "users" lists have been imported into Mailman3's archiver, HyperKitty, at: https://mailman.bitfolk.com/mailman/hyperkitty/ I tried to port over the existing Lurker archives but I just couldn't get it to work. Lurker is mostly undocumented and hasn't seen any development since 2007. This unfortunately means that all the old archive links that are out there in previous emails and in the wiki, etc. will break if and when I turn off Lurker on the old host. I hope to be able to transplant a static copy of https://lists.bitfolk.com/lurker/ over to https://mailman.bitfolk.com/lurker/ so that old links will still work, with automatic redirection. There doesn't appear to be any way to map an individual Lurker message link to a HyperKitty link, sadly. The new HyperKitty archives are fully searchable, so if you do happen to see a Lurker link in a wiki article please do feel free to find the same message in HyperKitty and update the article. If you find it in some other place that you can't edit please do just let me know and I'll update it. The same goes for any other references to lists.bitfolk.com. == Let us know about any issues or suggestions for the lists == As I say I'm not very experienced in running Mailman 3 so it's possibly it might break or be misconfigured in some way that I haven't yet found. So if you spot anything that you think looks broken, please do let us know. Also if you have suggestions for how it might work differently or look differently, I am interested to hear and try to implement, though I cannot promise that it is possible. Certainly I am not in love with the default appearance of the HyperKitty archives or the colour scheme of Mailman 3's web interface in general, and it may be possible to improve those. Thanks, Andy ¹ Discourse: https://www.discourse.org/ ² https://marc.merlins.org/netrants/reply-to-harmful.html and its counter-argument https://marc.merlins.org/netrants/reply-to-useful.html Both of these are over 20 years old so I'm not sure how relevant they are (or that mailing lists are) for the modern Internet. It should also be noted that one of the objections to Reply-To: munging is that it can remove all information about where the author wants replies to go. In Mailman3's case, if you supply Reply-To: headers of your own then it adds them, so that objection doesn't hold. -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ BitFolk Announcements mailing list -- announce(a)mailman.bitfolk.com To unsubscribe send an email to announce-leave(a)mailman.bitfolk.com

2 years, 5 months

Keep an eye on your CentOS Stream 8 grub.cfg for "blscfg" being introduced

by Andy Smith

Hi, In this week's maintenance one customer VM running CentOS Stream 8 failed to boot because its /boot/grub2/grub.cfg file contained bls commands which BitFolk's (upstream) Grub does not understand. BLS is a Red Hat invention supported only by systemd-boot, zipl and their own fork of Grub as far as I know. It certainly hasn't been pushed to upstream Grub as yet. More info: https://systemd.io/BOOT_LOADER_SPECIFICATION/ https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault I was aware of its use in Fedora and in CentOS Stream 9 but somehow it got enabled on the customer's CentOS Stream 8 when it previously wasn't. Unfortunately the VM in question has since been reinstalled so there is no way to know how it happened. If I knew how it happened then this would be an email to the announce@ list instead. But I don't, so I am just warning CentOS users to keep an eye out for this, because if your Grub configuration starts using this then your VM won't be bootable. To see if it is enabled, just: $ grep bls /boot/grub2/grub.cfg It shouldn't return anything. I believe to turn it off you can do: # echo "GRUB_ENABLE_BLSCFG=false" >> /etc/default/grub # grub2-mkconfig -o /boot/grub2/grub.cfg # grep bls /boot/grub2/grub.cfg but I am unable to verify this as I don't know how it got enabled or if it might get re-enabled again afterwards. I see that the binary /usr/sbin/grub2-switch-to-blscfg does exist on CentOS Stream 8 so that might have got called somehow. There is some talk that installing a new kernel might do this, but we use the kernel-ml package which comes from EPEL, and it doesn't appear to. I do see mentions all over that "CentOS Stream 8 uses blscfg by default", but I am not seeing that either on an existing install or a new one IO just did. So I don't know what went on with that VM. It definitely did get blscfg enabled somehow. If anyone has any more info, please do let us know. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 6 months

Another reminder that Debian 11 (bullseye) and beyond WILL NOT WORK in 32-bit PV mode

by Andy Smith

Hi, TL;DR: If you're running in 32-bit PV mode you should not try to update your kernel to version 5.9 or beyond, which means not to Debian 11. Take some action first, like switching to PVH mode. Otherwise it will not boot and you'll have to boot into the older kernel again. During this week's maintenance a worrying number of customer VMs didn't boot because they were 32-bit PV and had been updated to a kernel version of 5.9 or beyond. Debian 11 (bullseye) has kernel version 5.10. It's been mentioned a number of times here over the years that support for 32-bit PV in the Linux kernel was going away at version 5.9, but it looks like we haven't been able to communicate that as well as we would have liked. The most recent attempt was here: https://lists.bitfolk.com/lurker/message/20210930.104643.2ab5f9c0.en.html If 32-bit PV is you then probably what you want to do is just switch to PVH mode as mentioned at the top of that email. Possibly the remaining people who need to see this just aren't on the announce@ list. We could email every customer who is running 32-bit PV mode, but it would probably just serve to annoy those of you who are never planning to update your kernels. Finally, although switching to PVH should allow you to continue for many years to come, nobody should be running 32-bit. We strongly advise you to be planning your switch to amd64 as soon as possible [advice since 2015]. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

2 years, 6 months

Automatically unlocking a LUKS container for unattended reboots

by Andy Smith

Hello, Going through a total reboot as we are just now, it naturally leads to quite a few VMs sitting waiting for the customer to unlock their LUKS (encrypted block devices) before they will boot. That's a major downside of encrypting your disk. Various methods to automatically unlock LUKS have existed for many years, but can be somewhat fiddly to configure. That's obviously less secure as well, unless you really put some effort and some infrastructure into it. Mandos is an example of something you might use if you had another server somewhere: https://www.recompile.se/mandos/man/intro.8mandos Basically it stores an OpenPGP private key inside your initrd, and uses that to decrypt the LUKS passphrase that is stored on the mandos-server. The threat model for that is that an attacker with BitFolk-level access can read your (unencrypted) initrd to get the OpenPGP key out and also read the encrypted passphrase out of the mandos-server, put the two together to decrypt the passphrase and unlock your LUKS. If your mandos-server was not at BitFolk then that would be slightly more robust, but there is still the issue that an attacker at BitFolk could use a copy of your initrd to impersonate your VM during a maintenance window. You can of course instead do something a lot simpler like literally just have a script in your initrd contact a remote server to get the passphrase. The threat model is still about how your VM identifies itself to the secrets server in a way that can't be faked. For example if your script does an SSH to your secrets server, it means storing the private SSH key unencrypted inside your initrd, so an attacker with BitFolk-level access can read those SSH keys at any time without your knowledge and impersonate your VM to your secrets server to get a plain text copy of your passphrase. I can think of various ways you might harden that (or Mandos). Like you might require that the request comes from your IP address and within a certain time window, like within a few hours either side of a known maintenance window. I'm sure you can see how it would still be possible for an attacker with BitFolk-level access to impersonate your VM during a maintenance window in order to get a copy of your secrets to use at a later date. I'm guessing that of those customers who use disk encryption: - You mostly do it to try to hamper access by a fairly unskilled attacker who gains access to BitFolk's systems, or who physically steals the storage, or to guard against improper disposal of storage. - You haven't set up automatic unlocking because it's too fiddly, or you lack other servers elsewhere to serve the secrets, or you can see no way to make it as secure as you want. I was wondering if there was anything BitFolk could do to make it easier for those of you using LUKS, to have some form of automated unlock, perhaps only during maintenance events? If any of you have any thoughts I'd be interested to hear. It all really hinges on whether you care about someone impersonating your VM to get the LUKS passphrase, and if you do, if it is even possible to come up with a way for your initrd to identify itself that can't be subverted. Note that while it would be possible for BitFolk to run a secrets service and provide you with a minimally invasive package that adds a script to your initrd to unlock it, there isn't really any way to secure that against an attacker with BitFolk-level access. They could for example just take a snapshot of your block device and boot it during the maintenance window, unlock your LUKS, copy out your data, shut it down and then boot your real VM. The only record would be in BitFolk's secrets server that it received two requests for unlock. Which could be made to lie to you if you asked it how many requests it had seen. So really, BitFolk probably should not provide a secrets server. I'd be happy to investigate a few options for interacting with remote servers that you provide yourself, though. Mostly a documentation exercise. == On threat models == I think it is important to be realistic about what using LUKS on a virtual server actually can protect you against. When I say "an attacker with BitFolk-level access" I mean either BitFolk staff acting inappropriately, or outside attackers who have gained root access, or agents of the state who can compel BitFolk to do almost anything with your data. Against one of those sorts of attackers who is really motivated and specifically interested in you, you have no chance. It's trivial to dump your VM's memory image to disk, and probably quite feasible to find the LUKS keys within that - I've seen it done with KVM a few years ago. Then snapshot the block devices and take away a copy to decrypt at leisure. Even if you *were* running on bare metal, there are techniques to take dumps of the contents of DIMMs that are still in a running server but we don't have to speculate about that because you're not on bare metal and dumping VM memory is easy with root access to the hypervisor. Most attackers aren't like that though, not even the state. Criminals pick easier targets; the state prefers to either sniff traffic for long periods of time or just seize all the hardware. If you're doing something where you think the state might go as far as reading your LUKS keys out of a memory dump, you probably should not be doing it on a virtual machine in the UK! Another thing to consider is your Xen console. The kind of attacker we're talking about can read and write to it, so if you leave yourself logged in they can just be root straight away, and if you ever log in they can read the root password that you type, then reuse it later. When it comes to unlocking LUKS you are actually safer SSHing in to an sshd that's running in your initrd, not using the Xen console. It is really hard to secure VMs against hypervisor-level attackers with a particular interest, and so most uses of LUKS on VMs should only be for protecting against simpler attacks. It's still a LOT more secure than not using encrypted storage though… Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting

2 years, 6 months

Reboots will be necessary to address security issues, probably early hours 7/8/9 June

by Andy Smith

Hello, Unfortunately some serious security bugs have been discovered in the Xen hypervisor and fixes for these have now been pre-disclosed, with an embargo that ends at 1200Z on 9 June 2022. As a result we will need to apply these fixes and reboot everything before that time. We are likely to do this in the early hours of the morning UK time, on 7, 8 and 9 June. In the next few days individual emails will be sent out confirming to you which hour long maintenance window your services are in. The times will be in UTC; please note that UK is currently observing daylight savings and as such is currently at UTC+1. We expect the work to take between 15 and 30 minutes per bare metal host. If you have opted in to suspend and restore¹ then your VM will be suspended to storage and restored again after the host it is on is rebooted. Otherwise your VM will be cleanly shut down and booted again later. If you cannot tolerate the downtime then please contact support(a)bitfolk.com. We may be able to migrate² you to already-patched hardware before the maintenance is scheduled for your host. You can expect a few tens of seconds of pausing in that case. This process uses suspend&restore so has the same caveats. Cheers, Andy ¹ https://tools.bitfolk.com/wiki/Suspend_and_restore ² https://tools.bitfolk.com/wiki/Suspend_and_restore#Migration -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

2 years, 6 months

← Newer
1
...
9
10
11
12
13
14
15
...
112
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

BitFolk Users