Hi All,
I would like to simplify my VM, and make it more secure by using LXC
containers. E.g.
nginx for vanilla websites
Special nginx compile for webrtc stun and turn.
Email forwarder
MySQL server
Has anyone tried this? Does it work? What about IP addresses? Any hints
tips or advice welcome.
I will naturally be testing on a VM here at home, but that will be under
Vbox, which might not work like XEN.
I hesitate to start playing with the live config again, as last time I
brought down all https sites for a week! :(
Regards
Ian
--
Ian Hobson
Hi,
== TL;DR: ==
The BitFolk mailing lists changed address, but important stuff got
copied over. You don't need to do anything.
The new list archives are at:
https://mailman.bitfolk.com/mailman/hyperkitty/
and if you DO want to change any of your settings the main interface
is at:
https://mailman.bitfolk.com/mailman/postorius/lists/
But you'll have to do a Mailman password reset before you can log in
to it and change settings.
You can stop reading now unless you are particularly interested in
the details.
== What ==
BitFolk's announce@ and users@ mailing lists which used to be at
lists.bitfolk.com have now been migrated to
<listname>@mailman.bitfolk.com, which is Mailman 3.
== Why ==
The VM which runs lists.bitfolk.com needs to be upgraded. It runs
Mailman 2, which is a strictly Python 2 application. Python 2 was
EOL in 2020 sometime and is not available in newer releases of
Debian, and so neither is Mailman 2. An upgrade will leave Mailman 2
non-functional.
Running Mailman 3, which is a complete rewrite, is a major
undertaking that I have little experience of and I'm not entirely
happy about it. However, unless we either want to outsource the
mailing lists to another provider or switch to something like
Discourse¹, it seems to be still the best option.
While it may have been possible to install Python 2 and Mailman 2
from source on an otherwise upgraded host, that is a lot of work for
what seems like a dead end.
== Your subscriptions ==
All subscribed addresses have been copied over, which is why you are
seeing this email. You don't need to do anything, but if you do want
to adjust your settings you'll need to do a password reset in
Mailman as your passwords have not been copied over.
The web interface for the lists is at:
https://mailman.bitfolk.com/mailman/postorius/lists/
In the footer of messages to the "users" list it will have a link
and tell you which address you are subscribed with. The "announce"
list does not have a footer so if you're unsure what your subscribed
address is there you'll need to look at the email headers.
== Functional changes ==
=== Headers ===
I'm tired of list postings being rejected or going to spam folders.
We can't keep sending emails on your behalf, i.e. with a From:
address that is yours, because that breaks DKIM. So over on the
"users" list all posts will appear to come from:
From: Your Name via BitFolk Users <users(a)mailman.bitfolk.com>
Cc: Your Name <you(a)example.com>
The Cc header is so that people can still contact you off-list if
they want.
Obviously the List-Id: headers have changed so if you're filtering
email based upon those you will need to update your filters. Sorry
about that.
=== Reply-To: added, no more "users-replyto" ===
We used to have a separate mailing list "users-replyto" which got
all the same emails as the "users" list, but added:
Reply-To: users(a)lists.bitfolk.com
to them for people who preferred to explicitly have replies directed
back to the list.
The main "users" list is going to have an explicit Reply-To: added
now, making "users-replyto" redundant. Anyone who was subscribed to
"users-replyto" has been added to "users".
The reason for this is that I got feedback that despite it being a
long held belief that forcing Reply-To: is bad², these days most
people do prefer it. I'm happy to revisit that decision if people
don't like it, but the "users-replyto" hack won't be coming back.
=== List addresses ===
Obviously the list addresses have both changed from
@lists.bitfolk.com to @mailman.bitfolk.com. This was necessary so
that the older Mailman could continue working while the new one was
set up.
No doubt through muscle memory and address books the occasional
email will still end up being sent to whatever(a)lists.bitfolk.com.
These emails are being redirected to the correct place and I
anticipate keeping that redirection in place forever.
=== List archives ===
The public archives of the "announce" and "users" lists have been
imported into Mailman3's archiver, HyperKitty, at:
https://mailman.bitfolk.com/mailman/hyperkitty/
I tried to port over the existing Lurker archives but I just
couldn't get it to work. Lurker is mostly undocumented and hasn't
seen any development since 2007. This unfortunately means that all
the old archive links that are out there in previous emails and in
the wiki, etc. will break if and when I turn off Lurker on the old
host.
I hope to be able to transplant a static copy of
https://lists.bitfolk.com/lurker/ over to
https://mailman.bitfolk.com/lurker/ so that old links will still
work, with automatic redirection. There doesn't appear to be any way
to map an individual Lurker message link to a HyperKitty link, sadly.
The new HyperKitty archives are fully searchable, so if you do
happen to see a Lurker link in a wiki article please do feel free to
find the same message in HyperKitty and update the article. If you
find it in some other place that you can't edit please do just let
me know and I'll update it. The same goes for any other references
to lists.bitfolk.com.
== Let us know about any issues or suggestions for the lists ==
As I say I'm not very experienced in running Mailman 3 so it's
possibly it might break or be misconfigured in some way that I
haven't yet found. So if you spot anything that you think looks
broken, please do let us know.
Also if you have suggestions for how it might work differently or
look differently, I am interested to hear and try to implement,
though I cannot promise that it is possible.
Certainly I am not in love with the default appearance of the
HyperKitty archives or the colour scheme of Mailman 3's web
interface in general, and it may be possible to improve those.
Thanks,
Andy
¹ Discourse: https://www.discourse.org/
² https://marc.merlins.org/netrants/reply-to-harmful.html and its
counter-argument
https://marc.merlins.org/netrants/reply-to-useful.html
Both of these are over 20 years old so I'm not sure how relevant
they are (or that mailing lists are) for the modern Internet.
It should also be noted that one of the objections to Reply-To:
munging is that it can remove all information about where the
author wants replies to go. In Mailman3's case, if you supply
Reply-To: headers of your own then it adds them, so that objection
doesn't hold.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
BitFolk Announcements mailing list -- announce(a)mailman.bitfolk.com
To unsubscribe send an email to announce-leave(a)mailman.bitfolk.com
Hi,
In this week's maintenance one customer VM running CentOS Stream 8
failed to boot because its /boot/grub2/grub.cfg file contained bls
commands which BitFolk's (upstream) Grub does not understand.
BLS is a Red Hat invention supported only by systemd-boot, zipl and
their own fork of Grub as far as I know. It certainly hasn't been
pushed to upstream Grub as yet. More info:
https://systemd.io/BOOT_LOADER_SPECIFICATION/https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
I was aware of its use in Fedora and in CentOS Stream 9 but somehow
it got enabled on the customer's CentOS Stream 8 when it previously
wasn't.
Unfortunately the VM in question has since been reinstalled so there
is no way to know how it happened. If I knew how it happened then
this would be an email to the announce@ list instead. But I don't,
so I am just warning CentOS users to keep an eye out for this,
because if your Grub configuration starts using this then your VM
won't be bootable.
To see if it is enabled, just:
$ grep bls /boot/grub2/grub.cfg
It shouldn't return anything.
I believe to turn it off you can do:
# echo "GRUB_ENABLE_BLSCFG=false" >> /etc/default/grub
# grub2-mkconfig -o /boot/grub2/grub.cfg
# grep bls /boot/grub2/grub.cfg
but I am unable to verify this as I don't know how it got enabled
or if it might get re-enabled again afterwards.
I see that the binary /usr/sbin/grub2-switch-to-blscfg does exist on
CentOS Stream 8 so that might have got called somehow.
There is some talk that installing a new kernel might do this, but
we use the kernel-ml package which comes from EPEL, and it doesn't
appear to.
I do see mentions all over that "CentOS Stream 8 uses blscfg by
default", but I am not seeing that either on an existing install or
a new one IO just did. So I don't know what went on with that VM. It
definitely did get blscfg enabled somehow.
If anyone has any more info, please do let us know.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Hi,
TL;DR: If you're running in 32-bit PV mode you should not try to
update your kernel to version 5.9 or beyond, which means not to
Debian 11. Take some action first, like switching to PVH mode.
Otherwise it will not boot and you'll have to boot into the older
kernel again.
During this week's maintenance a worrying number of customer VMs
didn't boot because they were 32-bit PV and had been updated to a
kernel version of 5.9 or beyond. Debian 11 (bullseye) has kernel
version 5.10.
It's been mentioned a number of times here over the years that support
for 32-bit PV in the Linux kernel was going away at version 5.9, but
it looks like we haven't been able to communicate that as well as
we would have liked. The most recent attempt was here:
https://lists.bitfolk.com/lurker/message/20210930.104643.2ab5f9c0.en.html
If 32-bit PV is you then probably what you want to do is just switch
to PVH mode as mentioned at the top of that email.
Possibly the remaining people who need to see this just aren't on
the announce@ list. We could email every customer who is running
32-bit PV mode, but it would probably just serve to annoy those of
you who are never planning to update your kernels.
Finally, although switching to PVH should allow you to continue for
many years to come, nobody should be running 32-bit. We strongly
advise you to be planning your switch to amd64 as soon as possible
[advice since 2015].
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
Hello,
Going through a total reboot as we are just now, it naturally leads
to quite a few VMs sitting waiting for the customer to unlock their
LUKS (encrypted block devices) before they will boot. That's a major
downside of encrypting your disk.
Various methods to automatically unlock LUKS have existed for many
years, but can be somewhat fiddly to configure. That's obviously
less secure as well, unless you really put some effort and some
infrastructure into it.
Mandos is an example of something you might use if you had another
server somewhere:
https://www.recompile.se/mandos/man/intro.8mandos
Basically it stores an OpenPGP private key inside your initrd, and
uses that to decrypt the LUKS passphrase that is stored on the
mandos-server.
The threat model for that is that an attacker with BitFolk-level
access can read your (unencrypted) initrd to get the OpenPGP key out
and also read the encrypted passphrase out of the mandos-server, put
the two together to decrypt the passphrase and unlock your LUKS.
If your mandos-server was not at BitFolk then that would be slightly
more robust, but there is still the issue that an attacker at
BitFolk could use a copy of your initrd to impersonate your VM
during a maintenance window.
You can of course instead do something a lot simpler like literally
just have a script in your initrd contact a remote server to get the
passphrase. The threat model is still about how your VM identifies
itself to the secrets server in a way that can't be faked.
For example if your script does an SSH to your secrets server, it
means storing the private SSH key unencrypted inside your initrd, so
an attacker with BitFolk-level access can read those SSH keys at any
time without your knowledge and impersonate your VM to your secrets
server to get a plain text copy of your passphrase.
I can think of various ways you might harden that (or Mandos). Like
you might require that the request comes from your IP address and
within a certain time window, like within a few hours either side of
a known maintenance window. I'm sure you can see how it would still
be possible for an attacker with BitFolk-level access to impersonate
your VM during a maintenance window in order to get a copy of your
secrets to use at a later date.
I'm guessing that of those customers who use disk encryption:
- You mostly do it to try to hamper access by a fairly unskilled
attacker who gains access to BitFolk's systems, or who physically
steals the storage, or to guard against improper disposal of
storage.
- You haven't set up automatic unlocking because it's too fiddly, or
you lack other servers elsewhere to serve the secrets, or you can
see no way to make it as secure as you want.
I was wondering if there was anything BitFolk could do to make it
easier for those of you using LUKS, to have some form of automated
unlock, perhaps only during maintenance events? If any of you have
any thoughts I'd be interested to hear.
It all really hinges on whether you care about someone impersonating
your VM to get the LUKS passphrase, and if you do, if it is even
possible to come up with a way for your initrd to identify itself
that can't be subverted.
Note that while it would be possible for BitFolk to run a secrets
service and provide you with a minimally invasive package that adds
a script to your initrd to unlock it, there isn't really any way to
secure that against an attacker with BitFolk-level access.
They could for example just take a snapshot of your block device and
boot it during the maintenance window, unlock your LUKS, copy out
your data, shut it down and then boot your real VM. The only record
would be in BitFolk's secrets server that it received two requests
for unlock. Which could be made to lie to you if you asked it how
many requests it had seen.
So really, BitFolk probably should not provide a secrets server. I'd
be happy to investigate a few options for interacting with remote
servers that you provide yourself, though. Mostly a documentation
exercise.
== On threat models ==
I think it is important to be realistic about what using LUKS on a
virtual server actually can protect you against.
When I say "an attacker with BitFolk-level access" I mean
either BitFolk staff acting inappropriately, or outside attackers
who have gained root access, or agents of the state who can compel
BitFolk to do almost anything with your data.
Against one of those sorts of attackers who is really motivated and
specifically interested in you, you have no chance. It's trivial to
dump your VM's memory image to disk, and probably quite feasible to find
the LUKS keys within that - I've seen it done with KVM a few years
ago. Then snapshot the block devices and take away a copy to decrypt
at leisure.
Even if you *were* running on bare metal, there are techniques to
take dumps of the contents of DIMMs that are still in a running
server but we don't have to speculate about that because you're not
on bare metal and dumping VM memory is easy with root access to the
hypervisor.
Most attackers aren't like that though, not even the state.
Criminals pick easier targets; the state prefers to either sniff
traffic for long periods of time or just seize all the hardware.
If you're doing something where you think the state might go as far
as reading your LUKS keys out of a memory dump, you probably should
not be doing it on a virtual machine in the UK!
Another thing to consider is your Xen console. The kind of attacker
we're talking about can read and write to it, so if you leave
yourself logged in they can just be root straight away, and if you
ever log in they can read the root password that you type, then
reuse it later. When it comes to unlocking LUKS you are actually
safer SSHing in to an sshd that's running in your initrd, not using
the Xen console.
It is really hard to secure VMs against hypervisor-level attackers
with a particular interest, and so most uses of LUKS on VMs should
only be for protecting against simpler attacks. It's still a LOT
more secure than not using encrypted storage though…
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
