14 Apr
2024
14 Apr
'24
12:33 a.m.
Hi,
In light of the recent XZ/lzma backdoor we should perhaps think
harder about how complex sshd is and the wisdom of exposing that to
the entire Internet.
At BitFolk, this currently needs to be exposed to the Internet so
that you can connect to your Xen Shell (console).
More info on the Xen Shell for those new to it:
https://tools.bitfolk.com/wiki/Xen_Shell
When thinking about ways to tighten this up, they are all going to
involve taking away your ability to just SSH to
you(a)you.console.bitfolk.com.
There's a couple of ways this could go:
1. Remove direct SSH capability, replace with web
All current Xen Shell features to be put on a web interface.
Console to be used from a web interface.
I don't relish this, but I have seen some pretty nifty web
terminals so maybe it wouldn't be that bad.
2. Firewall off SSH from the Internet, poke holes temporarily
You'd have to supply to the Panel some list of net blocks that
you will SSH from and then there'd be a button to punch holes in
the firewall for SSH from those net blocks for 6 hours (for
example).
There would have to be a limit on the size of the net blocks.
Let's say a /16 for IPv4 and a /32 for IPv6.
Option (2) is lots easier to implement so could happen fairly
quickly, but it is more fiddly to use: You're in an arbitrary place
and you suddenly need to connect to your Xen Shell; you've then got
to log in to Panel, work out your IP address¹, add it to the allow
list, then hit the button. Finally you can SSH.
Option (1) will require a lot of work but maybe it's less friction
to use a web-based terminal if you have to visit a web site anyway?
Option (2) does not preclude option (1). Providing a web-based
terminal that doesn't need the "allow SSH" button to be pressed can
be implemented after option (2) is done, as a sort of stretch goal.
Or maybe the web terminal is such an attractive and required feature
that it should be done first, before option (2) would be enacted?
I am disregarding the idea of flipping the default to having
password auth disabled. It is currently by default enabled but you
can disable it on a per-VM basis. I do recommend that you do that,
but what I am really more concerned about is an(other) exploit in
sshd, and that doesn't really have to be anything to do with
authentication (the XZ thing wasn't). Yes it would be a very bad day
for you if someone got in to your Xen Shell, but that's as far as
that goes, so I am still okay with leaving that determination up to
the customer.
I can't see a way of avoiding there having to be a list of net
blocks. With 50+ VMs on a host, if SSH is opened up from anywhere
for any one of them then it is open for all so really even with a
timeout it would be open a lot of the time.
I guess something interesting could be done for those not on legacy
Internet: assign unique IPv6 console address that can only be used
for connecting to that VM's Xen Shell. 😀
I'd be interested in hearing your thoughts on this.
Thanks,
Andy
¹ Which you may not know because you might be behind one or more
layers of NAT.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
14 Apr
14 Apr
1:30 a.m.
Hello,
On Sun 14 Apr 2024 00:33:36 GMT, Andy Smith via BitFolk Users wrote:
2. Firewall off SSH from the Internet, poke holes
temporarily
You'd have to supply to the Panel some list of net blocks that
you will SSH from and then there'd be a button to punch holes in
the firewall for SSH from those net blocks for 6 hours (for
example).
There would have to be a limit on the size of the net blocks.
Let's say a /16 for IPv4 and a /32 for IPv6.
Sounds good to me, I really don’t like all this web-based stuff.
I guess something interesting could be done for those
not on legacy
Internet: assign unique IPv6 console address that can only be used
for connecting to that VM's Xen Shell. 😀
THAT sounds fun, I want to try it :)
--
Alarig
7:50 a.m.
Hey folks,
Just some thoughts on Andy's suggestions:
1. Remove direct SSH capability, replace with web
All current Xen Shell features to be put on a web interface.
Console to be used from a web interface.
I don't relish this, but I have seen some pretty nifty web
terminals so maybe it wouldn't be that bad.
That sounds like a possible solution, although I'd be intrigued to know how
SSH keys work in that case (ideally I'd still like my log in process to the
Xen shell to use a key, not a password)
2. Firewall off SSH from the Internet, poke holes temporarily
[...]
Option (2) is lots easier to implement so could happen
fairly
quickly, but it is more fiddly to use: You're in an arbitrary place
and you suddenly need to connect to your Xen Shell; you've then got
to log in to Panel, work out your IP address¹, add it to the allow
list, then hit the button. Finally you can SSH.
I agree that this would be fiddly - especially if you're away and don't
know your current IP. Additionally there might be a static IP you'd want to
allowlist constantly, and having to come in every time to allow it would be
a pain (or in my case, I have a DDNS entry that points to my home IP if it
changes - ideally I'd like to allowlist that so it updates)
Option (3) might be to use a VPN of some kind? I use Wireguard at home to
access my internal network - would something like that be preferable? I say
Wireguard because I think it's fairly(!) easy to manage once it's running
and doesn't require a huge amount of config on either side. Other VPN
solutions do exist, of course.
I guess something interesting could be done for those not on legacy
Internet: assign unique IPv6 console address that can
only be used
for connecting to that VM's Xen Shell. 😀
While that does sound fancy, I think that might cause more problems for
those of us still on the legacy Internet... or if you need emergency access
when travelling for example!
Cheers,
James
9:10 a.m.
On Sun 14 Apr 2024 08:50:37 GMT, James Gregory-Monk via BitFolk Users wrote:
Option (3) might be to use a VPN of some kind? I use
Wireguard at home to
access my internal network - would something like that be preferable? I say
Wireguard because I think it's fairly(!) easy to manage once it's running
and doesn't require a huge amount of config on either side. Other VPN
solutions do exist, of course.
Not directly bitfolk related, but I’m still trying to figure out how to
distribute private keys without people yelling about it being insecure.
Giving an openvpn password via email doesn’t bother so much people,
giving a wg private key does. It’s the same thing at the end, it sets a
VPN up, so I don’t really get the point…
--
Alarig
9:52 a.m.
On 14 April 2024 10:11:01 Alarig Le Lay wrote:
Not directly bitfolk related, but I’m still trying to figure out how to
distribute private keys without people yelling about it being insecure.
Giving an openvpn password via email doesn’t bother so much people,
giving a wg private key does. It’s the same thing at the end, it sets a
VPN up, so I don’t really get the point…
You need the other side to have a private key already, really.
Then you either:
• trust that key directly
• generate a new private key and encrypt it with their existing private key
An alternative, though, if it really really matters: put the key into a
YubiKey. Post it to them. Get the recipient to confirm receipt and that the
serial number matches, then disclose the PIN to the YubiKey. OpenVPN can
load keys via PKCS#11 (apparently).
As for passwords, it's a lot easier to let people set a new password than
to let them set a new private key, so private keys tend to stay set.
Tim
9:53 a.m.
Hey Alarig,
Not directly bitfolk related, but I’m still trying to figure out how to
distribute private keys without people yelling about
it being insecure.
Giving an openvpn password via email doesn’t bother so much people,
giving a wg private key does. It’s the same thing at the end, it sets a
VPN up, so I don’t really get the point…
My understanding is that each peer doesn't need to share their private key
- they only need to share their public keys. I'd imagine in this situation
that Bitfolk should have a public key we'd all see (maybe even a pre shared
one per peer?) and that we'd need to provide our public part to Bitfolk.
Admittedly it's been a little bit since I last set up Wireguard so I might
be wrong...
Cheers,
James
8:03 a.m.
On 14 April 2024 01:34:14 Andy Smith wrote:
There's a couple of ways this could go:
1. Remove direct SSH capability, replace with web
That change is likely to make the trusted computing based for shell VM
access much larger than it is now.
2. Firewall off SSH from the Internet, poke holes temporarily
Ideally with an API for defining the holes, or even better a VPN option
(WireGuard?).
In that story the VPN only allows (Xen) shell access and maybe also the
Bitfolk website(s).
…
What I'd like is to be able to configure MFA for the Xen shell. Maybe I use
a private key and also get prompted for OATH HOTP?
The chance of another xz like supply chain attack feels low, because people
will be looking for these now. My opinion - there is no special need to put
in effort planning to fight the last war again.
But remote access is tricky for sure and someone getting it can bypass many
layers of security. So general defence in depth is very worthwhile.
I guess something interesting could be done for those
not on legacy
Internet: assign unique IPv6 console address that can only be used
for connecting to that VM's Xen Shell. 😀
I like this idea, especially if combined with the option of bringing up a
VPN first. If you had VPN trouble, you could maybe have a setting to switch
from "VPN required" to "allow connections from this address range for n
minutes. Unsure about settings? Range could be ::0/0 or the IPv4
equivalent, and minutes could default to 1440. That is still more protected
than the status quo.
Tim
1:32 p.m.
Hello,
On Sun, Apr 14, 2024 at 09:03:00AM +0100, Tim Bannister via BitFolk Users wrote:
even better a VPN option (WireGuard?).
I'm fairly confident that almost no one will bother to use a VPN
unless they are forced to, in which case they just will refuse to be
customers at all.
Is there any existing VPS provider that forces customers to use a
VPN to access their console?
What I'd like is to be able to configure MFA for
the Xen shell. Maybe I use
a private key and also get prompted for OATH HOTP?
Are you aware that we support TOTP for both logging into
panel.bitfolk.com and for SSH to *.console.bitfolk.com?
https://tools.bitfolk.com/wiki/Two-factor_authentication
Using an SSH key does bypass this however.
I have just noticed that this article is out of date when it says
"There is currently no way to restrict Xen Shell SSH login to key
only", as that feature was also added.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
8:38 a.m.
Hi Andy,
Your argument was about ’how complex sshd is and the wisdom of exposing that to the entire
Internet.‘ and yet all of the options provided by you and others seem to be nothing more
than substituting in an equally, if not more complex system that will be exposed to the
entire internet.
With respect, there are far bigger people than Bitfolk who rely on SSH, so if they’re not
worried then I don’t see why we should be. My advice would be to wait and see how the
likes of GitHub et al. respond. If they are worried then whatever solution they come up
with will likely be far better than whatever we cobble together here.
Regards,
Chris
--
Chris Smith <space.dandy(a)icloud.com>
On 14 Apr 2024, at 01:34, Andy Smith via BitFolk Users
<users(a)mailman.bitfolk.com> wrote:
Hi,
In light of the recent XZ/lzma backdoor we should perhaps think
harder about how complex sshd is and the wisdom of exposing that to
the entire Internet.
At BitFolk, this currently needs to be exposed to the Internet so
that you can connect to your Xen Shell (console).
More info on the Xen Shell for those new to it:
https://tools.bitfolk.com/wiki/Xen_Shell
When thinking about ways to tighten this up, they are all going to
involve taking away your ability to just SSH to
you(a)you.console.bitfolk.com.
There's a couple of ways this could go:
1. Remove direct SSH capability, replace with web
All current Xen Shell features to be put on a web interface.
Console to be used from a web interface.
I don't relish this, but I have seen some pretty nifty web
terminals so maybe it wouldn't be that bad.
2. Firewall off SSH from the Internet, poke holes temporarily
You'd have to supply to the Panel some list of net blocks that
you will SSH from and then there'd be a button to punch holes in
the firewall for SSH from those net blocks for 6 hours (for
example).
There would have to be a limit on the size of the net blocks.
Let's say a /16 for IPv4 and a /32 for IPv6.
Option (2) is lots easier to implement so could happen fairly
quickly, but it is more fiddly to use: You're in an arbitrary place
and you suddenly need to connect to your Xen Shell; you've then got
to log in to Panel, work out your IP address¹, add it to the allow
list, then hit the button. Finally you can SSH.
Option (1) will require a lot of work but maybe it's less friction
to use a web-based terminal if you have to visit a web site anyway?
Option (2) does not preclude option (1). Providing a web-based
terminal that doesn't need the "allow SSH" button to be pressed can
be implemented after option (2) is done, as a sort of stretch goal.
Or maybe the web terminal is such an attractive and required feature
that it should be done first, before option (2) would be enacted?
I am disregarding the idea of flipping the default to having
password auth disabled. It is currently by default enabled but you
can disable it on a per-VM basis. I do recommend that you do that,
but what I am really more concerned about is an(other) exploit in
sshd, and that doesn't really have to be anything to do with
authentication (the XZ thing wasn't). Yes it would be a very bad day
for you if someone got in to your Xen Shell, but that's as far as
that goes, so I am still okay with leaving that determination up to
the customer.
I can't see a way of avoiding there having to be a list of net
blocks. With 50+ VMs on a host, if SSH is opened up from anywhere
for any one of them then it is open for all so really even with a
timeout it would be open a lot of the time.
I guess something interesting could be done for those not on legacy
Internet: assign unique IPv6 console address that can only be used
for connecting to that VM's Xen Shell. 😀
I'd be interested in hearing your thoughts on this.
Thanks,
Andy
¹ Which you may not know because you might be behind one or more
layers of NAT.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
BitFolk Users mailing list <users(a)mailman.bitfolk.com>
You're subscribed as <space.dandy(a)icloud.com>
Unsubscribe:
<https://mailman.bitfolk.com/mailman/postorius/lists/users.mailman.bitfolk.com/>
or send an email to <users-leave(a)mailman.bitfolk.com>
<signature.asc>
9:31 a.m.
I don't want to be paniced by this. Yes, a backdoor was sneaked into SSH. But it was
found and removed.
I'm pretty confident that a backdoor attempt will be made to something else in the
future, but my guess is that SSH will not be the target: too many people will be watching.
It might be useful to be able to block SSH access from certain regions (for example,
China, Russia and MAGAland*) but to replace a simple, reliable system like SSH with a
complex and therefore insecure web-page-based kludge feels like a leap in the wrong
direction.
It ain't broke: please don't fix it.
*It isn't very likely, but it is possible that the USA will end up with either Trump
as president, or in a modern civil war, some of which will be fought on the web.
On 14 Apr 2024 at 01:33, Andy Smith via BitFolk Users <users(a)mailman.bitfolk.com>
wrote:
Hi,
In light of the recent XZ/lzma backdoor we should perhaps think
harder about how complex sshd is and the wisdom of exposing that to
the entire Internet.
9:44 a.m.
Agree with Chris and Iain, OpenSSH is reasonably well audited, along with
its dependencies, it’s just the link with systemd that causes this issue. I
haven’t used
Xen for about ten years, does it use systemd?
If it does can you remove the link to stop future attacks so start it with
an init script?
Kamal
On Sun, 14 Apr 2024 at 19:31, iain via BitFolk Users <
users(a)mailman.bitfolk.com> wrote:
I don't want to be paniced by this. Yes, a
backdoor was sneaked into SSH.
But it was found and removed.
I'm pretty confident that a backdoor attempt will be made to something
else in the future, but my guess is that SSH will not be the target: too
many people will be watching.
It might be useful to be able to block SSH access from certain regions
(for example, China, Russia and MAGAland*) but to replace a simple,
reliable system like SSH with a complex and therefore insecure
web-page-based kludge feels like a leap in the wrong direction.
It ain't broke: please don't fix it.
*It isn't very likely, but it is possible that the USA will end up with
either Trump as president, or in a modern civil war, some of which will be
fought on the web.
On 14 Apr 2024 at 01:33, Andy Smith via BitFolk Users <
users(a)mailman.bitfolk.com> wrote:
Hi,
In light of the recent XZ/lzma backdoor we should perhaps think
harder about how complex sshd is and the wisdom of exposing that to
the entire Internet.
_______________________________________________
BitFolk Users mailing list <users(a)mailman.bitfolk.com>
You're subscribed as <shakerky(a)googlemail.com>
Unsubscribe: <
https://mailman.bitfolk.com/mailman/postorius/lists/users.mailman.bitfolk.c…
or send an email to
<users-leave(a)mailman.bitfolk.com>
11:56 a.m.
You'd have to supply to the Panel some list of net blocks that
you will SSH from and then there'd be a button
to punch holes in
the firewall for SSH from those net blocks for 6 hours (for
example).
There would have to be a limit on the size of the net blocks.
Let's say a /16 for IPv4 and a /32 for IPv6.
Perhaps a slight modification to the this option:
On some systems I set up a system where nftables opens the ssh port
temporarily and only after successful authentication via wget/https to
either port 24 or port 443.
I use port 24 because it is not one I usually use and less likely to be
fiddled with nat/broken firewalls.
The server uses the IP address 'wget' is coming from to open port 22 to
that source ip. (v4 and/or v6).
I can think of scenarios where weird nat/firewalls break that, but I
have not encountered such network in the past 5 odd years when I
started using it.
The timeouts I am using is 3 minutes until initial connection is made
and 1 hour after that.
When I travel and are on dynamic IPs, I sometimes start a shell script
to wget every 2 minutes to keep my IP white-listed.
FWIW I also use wireguard and openvpn as backup methods. (I am aware of
some networks where this algorithm would fail, I just don't happen to
be on any of those recently).
Conrad
15 Apr
15 Apr
3:24 p.m.
Hi,
On Sun, Apr 14, 2024 at 12:56:24PM +0100, Conrad Wood via BitFolk Users wrote:
The server uses the IP address 'wget' is
coming from to open port 22 to
that source ip. (v4 and/or v6).
It is an interesting idea to guess the customer's current IP from
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
9:07 p.m.
Hi,
On Sun, Apr 14, 2024 at 12:56:24PM +0100, Conrad Wood
via
BitFolk Users wrote:
Yes!
There could even be some nice UI in the panel that allows adding that IP
address (plus perhaps a suitably sized netmask) as a new allow-rule
immediately in one click.
Best wishes,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
The server uses the IP address 'wget' is
coming from to open port 22 to
that source ip. (v4 and/or v6).
It is an interesting idea to guess the customer's current IP from
3:34 p.m.
Hi,
I really have to push back on a couple of points here:
- The idea that all of the dependencies¹ of openssh are audited, let
alone well audited. Unless you know this to be the case, I think
it's very unlikely that anyone has been paying attention, because
having nobody to pay attention is the norm.
- That this is somehow a systemd issue. All of the code in every .so
below can do stuff as root inside the address space of the sshd
process.
What makes sshd different is that it's run almost everywhere (hugely
attractive target) and it's often exposed to the whole Internet
(hugely attractive target) and a lot of it runs as root.
No matter how much anyone hates systemd, this class of problem does
not go away by refusing to use systemd. Systemd had already made
changes to avoid it needing to be linked to sshd, before any of this
stuff happened - this may have been a reason why the attackers
accelerated their efforts to get this deployed.
Thanks,
Andy
¹ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffe34a98000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f7fc73b2000)
libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f7fc73a6000)
libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007f7fc7375000)
libpam.so.0 => /lib/x86_64-linux-gnu/libpam.so.0 (0x00007f7fc7363000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f7fc7293000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f7fc7263000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007f7fc7211000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f7fc7137000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f7fc7131000)
libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f7fc6c00000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7fc7112000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7fc6a1f000)
libnsl.so.2 => /lib/x86_64-linux-gnu/libnsl.so.2 (0x00007f7fc70f5000)
libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007f7fc70ed000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f7fc70e1000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f7fc68d8000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f7fc70b2000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f7fc681c000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f7fc708a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7fc7545000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f7fc6782000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007f7fc6755000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007f7fc6747000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007f7fc6740000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f7fc672f000)
libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007f7fc6701000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x00007f7fc66d9000)
On Sun, Apr 14, 2024 at 07:44:20PM +1000, Kamal Shaker via BitFolk Users wrote:
Agree with Chris and Iain, OpenSSH is reasonably well
audited, along with
its dependencies, it’s just the link with systemd that causes this issue. I
haven’t used
Xen for about ten years, does it use systemd?
If it does can you remove the link to stop future attacks so start it with
an init script?
Kamal
--
https://bitfolk.com/ -- No-nonsense VPS hosting
3:46 p.m.
On 15 Apr 2024, at 16:34, Andy Smith via BitFolk Users <users(a)mailman.bitfolk.com>
wrote:
What makes sshd different is that it's run almost
everywhere (hugely
attractive target) and it's often exposed to the whole Internet
(hugely attractive target) and a lot of it runs as root.
Exactly why I haven’t exposed sshd to anything other than a few of my own servers for the
past few years. If I’m out and about and desperately need access then I can VPN into my
home network and ssh from there.
Cheers,
Mike
3:56 p.m.
On 15 Apr 2024, at 16:34, Andy Smith via BitFolk Users <users(a)mailman.bitfolk.com>
wrote:
- That this is somehow a systemd issue. All of the
code in every .so
below can do stuff as root inside the address space of the sshd
process.
Granted.
What makes sshd different is that it's run almost
everywhere (hugely
attractive target) and it's often exposed to the whole Internet
(hugely attractive target) and a lot of it runs as root.
But despite that I expect it is still run in as many places as whatever web server you
will use to implement your alternative solution. You may argue that your web server
doesn’t run as root, but if you intend it to perform this intended service then a portion
of it will have to, so you’re back at square one. What does your attack surface look like
if you do ‘ldd /my/web/server’? It still seems to me like you're trading one fairly
complex, reasonably well understood mechanism for a more complex, less well understood
alternative.
Regards,
Chris
—
Chris Smith <space.dandy(a)icloud.com>
6:39 p.m.
Hello,
On Mon, Apr 15, 2024 at 04:56:12PM +0100, Chris Smith via BitFolk Users wrote:
On 15 Apr 2024, at 16:34, Andy Smith via BitFolk Users
<users(a)mailman.bitfolk.com> wrote:
I think it's fairly unlikely that there would be as many
installations of haproxy as there are of OpenSSH, since there are
several big-name reverse proxies and web servers while there is
really only OpenSSH used at scale. Outside of Windows and OpenBSD I
would expect almost every web and proxy server to also run OpenSSH.
What makes sshd different is that it's run
almost everywhere (hugely
attractive target) and it's often exposed to the whole Internet
(hugely attractive target) and a lot of it runs as root.
But despite that I expect it is still run in as many places as
whatever web server you will use to implement your alternative
solution. You may argue that your web server doesn’t run as
root,
but if you intend it to perform this intended service then a
portion of it will have to, so you’re back at square one.
I don't agree as what we're actually talking about here is a
collection of things: bits of the Panel site to manage the list of
net blocks in the customer database; something else to adjust the
firewall rules on the hypervisors. That's all very disaggregated and
limited in how they speak to each other; such communication is
mostly only internal; the bits that aren't internal are only
touched by authenticated users; and so on.
It doesn't seem like an easy or attractive target for anyone to
spend a lot of effort on.
I get that you aren't a fan of the idea though. Your point about
GitHub being a service that relies upon world-accessible SSH is well
taken and it may be a situation like that.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
7:29 p.m.
I have ssh on my phone. It's a pain to use, but it works well. The app I use is Called
Termius
On 15 Apr 2024 at 20:15, Chris Smith via BitFolk Users <users(a)mailman.bitfolk.com>
wrote:
On 15 Apr 2024, at 19:40, Andy Smith via BitFolk
Users <users(a)mailman.bitfolk.com> wrote:
I get that you aren't a fan of the idea though. Your point about
GitHub being a service that relies upon world-accessible SSH is well
taken and it may be a situation like that.
To be fair, as a feature in itself I think it has a lot of merit. As Sod’s Law would
have it, on the rare occasion I’ve really needed Xen access I’ve been abroad on holiday
without my laptop, so SSH was useless to me anyway. Having a web-based console would have
been useful.
I’m just not sold on it (yet) as a solution to the hypothetical problem. It’s certainly
not something at which I’m going to bother brandishing my pitchfork — it’s quite heavy and
I’m out of shape.
Regards,
Chris
—
Chris Smith <space.dandy(a)icloud.com>
_______________________________________________
BitFolk Users mailing list <users(a)mailman.bitfolk.com>
You're subscribed as <iain(a)hairydog.co.uk>
Unsubscribe:
<https://mailman.bitfolk.com/mailman/postorius/lists/users.mailman.bitfolk.com/>
or send an email to <users-leave(a)mailman.bitfolk.com>
8:50 p.m.
On 2024-04-15 20:29+0100, iain via BitFolk Users wrote:
I have ssh on my phone. It's a pain to use, but it
works well. The app
I use is Called Termius
Well, juicessh is useful and has some useful additions to the on screen
keyboard like esc (does the same as alt, alt+. is the same as esc, then
.).
Personally I don't mind if the console is changed, so long as I can get
to the console in a pinch. I don't travel with my bitfolk login details
in my pocket, I keep these elsewhere. SSH interface is a lot easier for
me than navigating the panel.
To that point, is the panel a stronger security zone than SSH itself? I
have more faith in SSH being reliable than anything that's on web.
IMO, it is more probable that an internet available admin panel includes
a backdoored perl/php/python library. Who's vetting the code that turns
login form posts into variables, or just spawn a thing ...
Ed
8:58 p.m.
Hi,
I really have to push back on a couple of points
here:
- The idea that all of the dependencies¹ of openssh are audited, let
alone well audited. Unless you know this to be the case, I think
it's very unlikely that anyone has been paying attention, because
having nobody to pay attention is the norm.
- That this is somehow a systemd issue. All of the code in every .so
below can do stuff as root inside the address space of the sshd
process.
Agreed.
What makes sshd different is that it's run almost
everywhere (hugely
attractive target) and it's often exposed to the whole Internet
(hugely attractive target) and a lot of it runs as root.
Yes: this is a worrying attack surface for sure.
I'm sure there have been a few holes over the years but only this current
attack and the ssh-1 break really stick in my mind as being major, major
problems. So my feeling is that the track record is pretty good over the
long term, even compared to other software in the same class.
However, it's also true that as sshd gets bigger and bigger the attack
surface slowly increases too.
It would be a shame to have to hide the service behind some other service.
It would be similar-but-different to the inconvenience of running sshd on a
weird port (which I do for my machine, but is generally unacceptable for a
public service for lots of reasons to numerous to include here).
A few questions:
Do you know how available different options such as VPNs, etc are from
behind common firewall and NAT configurations? We know that HTTP(S) is
available almost everywhere, and I imagine sshd is a close second. But lots
of other services are frequently filtered by providers and I (for one)
don't know the likelyhood of various other options being available in which
places.
Do you have a feel for whether people in general are starting to think that
ssh is not suitable for direct exposure to the Internet? If this feeling is
mounting then it's probably best to follow suit. Of course, there are
always the risk averse who always use bastion hosts, but there has always
been a hard-core of (especially) FOSS and academic people who have a risk
profile that allows it.
Do you have some more detail about the threat model you're trying to
protect against?
A suggestion regarding bastion hosts:
(Just an idea for discussion; not really fully formed yet)
Depending on the threat model, perhaps it's enough to provide an ssh
endpoint that simply leads to a shell or environment that only allows a
further ssh into the regular Xen Shells? That 2nd hop would still require
the same auth as it currently does. The purpose of the initial auth would
be to provide a DMZ. Even a full root compromise of that environment (for
example, via the irregularly authenticated RCE exploit in the XZ backdoor)
wouldn't get you any persistent user data or shell access. At worst the
attacker would be able to sniff in-progress console sessions and collect
credentials for some hosts and there are surely ways to monitor
long-running connections to the bastion host via a router or firewall to
detect successful attacks?
¹ $ ldd /usr/sbin/sshd
Ooo! Don't run ldd on an untrusted binary. :-p
https://jmmv.dev/2023/07/ldd-untrusted-binaries.html
:-)
Best wishes,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
9:19 p.m.
Hello,
On Mon, Apr 15, 2024 at 09:58:03PM +0100, Andy Bennett via BitFolk Users wrote:
It would be similar-but-different to the inconvenience
of running sshd on a
weird port (which I do for my machine, but is generally unacceptable for a
public service for lots of reasons to numerous to include here).
It's also not very effective any more. For more than 10 years I have
run SSH on a different port, and it still gets found almost
immediately. People just routinely scan all ports and look for SSH
banners. The volume of spam connections is less but it's not hugely
less.
Do you know how available different options such as
VPNs, etc are from
behind common firewall and NAT configurations?
I don't know, but I don't want to spend time on thinking about VPNs
for customer use because I just don't think the people can agree on
VPN software to install and bother to use.
People use VPNs. I use VPNs. But would even I install a VPN product
that wasn't my chosen one? Probably not.
Do you have a feel for whether people in general are
starting to think that
ssh is not suitable for direct exposure to the Internet? If this feeling is
mounting then it's probably best to follow suit.
It has always been best practice to put SSH behind a VPN or at least
some bastion jump host, for this exact reason. But in situations
where the user base is potentially "everyone from everywhere",
companies (like GitHub) don't do that for obvious reasons.
Maybe this is still one of those situations.
Do you have some more detail about the threat model
you're trying to protect
against?
As said, a zero-day in SSH, which is one of the few services that is
network exposed on the hypervisors.
(Just an idea for discussion; not really fully formed
yet)
Depending on the threat model, perhaps it's enough to provide an ssh
endpoint that simply leads to a shell or environment that only allows a
further ssh into the regular Xen Shells?
Yes as a half measure we could go to a bastion host arrangement.
You can push the ssh agent through that so keys would still work for
a seamless single connection, although that is slightly more risky
of course.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
9:52 p.m.
Hi,
That's quite a broad, unknowable attack surface tho', and perhaps the
reason this thread has become quite bushy already. :-)
I guess you're primarily trying to protect things in roughly this order:
+ Bitfolk business info such as customer lists and billing details
+ Customer data you have persistently on disk
+ VM state in memory
+ Customer credentials
+ Use of compute resources by people who are not customers
+ Use of other resources by people who are not customers
Given that the XZ backdoor shows signs of having had a lot of resources,
both technical and financial, and the exploit itself was carefully written
so that only the holder of the private key could use it, it seems that, in
that instance, your vulnerability may have been mostly to the last two
items on the list.
i.e. someone selling your machines as part of a botnet or using them for
some purpose unrelated to Bitfolk.
If it was a nation state, (I assume) it's unlikely you would be a general
target for the earlier items on the list. Perhaps some specific customers
in certain circumstances. But (IMHO) if the XZ backdoor was a nation state
funded thing then it smells more like the bad end of the stuxnet situation
where someone is trying to get a backdoor into a very specific installation
for a very specific reason. Once the backdoor is in, they can use other
hosts to hide the origin of their connections to the target.
</wild-speculation>
If it was a well funded private attacker then sure, they'll probably hoover
up everthing they can get but there's still a lot more analysis to do here.
Is it ransomware? Is it data mining? Is it resource usage for sale or for
crypto-currency mining? Is it as a more anonymous jumping off point for
more specific attacks?
Deliberate zero-days cost a lot and provide specific capabilities.
Accidental ones are probably more relevant to our discussion. The history
of them suggests that they are patched quickly (in the kind of daemon
software we are talking about) and often fragile or difficult to exploit.
So perhaps other sandboxing or hardening techniques for the ssh daemon are
appropriate?
Of course, I can only speculate on your priorities but I think it'd be
valuable to discuss the threat model in more detail because there is
potentially a bunch of options (such as sandboxing or hardening) that can
mitigate many vectors and won't change the workflow much (if at all) for
customers.
What do you think?
Best wishes,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
Do you have
some more detail about the threat model you're
trying to protect
against?
As said, a zero-day in SSH, which is one of the few services that is
network exposed on the hypervisors.
9:16 p.m.
On Tue, 16 Apr 2024 at 01:34, Andy Smith via BitFolk Users <
users(a)mailman.bitfolk.com> wrote:
Hi,
I really have to push back on a couple of points here:
- The idea that all of the dependencies¹ of openssh are audited, let
alone well audited. Unless you know this to be the case, I think
it's very unlikely that anyone has been paying attention, because
having nobody to pay attention is the norm.
Hmm, I could have sworn I saw something about it being audited, but my
Googling found nothing. But my memory stretches back to pre-Google. I'll
concede the auditing point. :-)
- That this is somehow a systemd issue. All of the
code in every .so
below can do stuff as root inside the address space of the sshd
process.
What makes sshd different is that it's run almost everywhere (hugely
attractive target) and it's often exposed to the whole Internet
(hugely attractive target) and a lot of it runs as root.
No matter how much anyone hates systemd, this class of problem does
not go away by refusing to use systemd. Systemd had already made
changes to avoid it needing to be linked to sshd, before any of this
stuff happened - this may have been a reason why the attackers
accelerated their efforts to get this deployed.
I didn't say get rid of systemd, I just said not to use it for sshd. As
having systemd control sshd drags in a whole bunch of extra dependencies. I
know it's untenable to create statically linked binaries for demons any
more so you have to trust the system libraries, but the smaller the attack
surface the better.
Similar to creating a web interface wrapper around SSH would most likely
just add more vectors.
18 Apr
18 Apr
3:10 p.m.
On Mon 15 Apr 2024 15:34:07 GMT, Andy Smith via BitFolk Users wrote:
¹ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffe34a98000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f7fc73b2000)
libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f7fc73a6000)
libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007f7fc7375000)
libpam.so.0 => /lib/x86_64-linux-gnu/libpam.so.0 (0x00007f7fc7363000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0
(0x00007f7fc7293000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1
(0x00007f7fc7263000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007f7fc7211000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f7fc7137000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
(0x00007f7fc7131000)
libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f7fc6c00000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7fc7112000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7fc6a1f000)
libnsl.so.2 => /lib/x86_64-linux-gnu/libnsl.so.2 (0x00007f7fc70f5000)
libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007f7fc70ed000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f7fc70e1000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20
(0x00007f7fc68d8000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f7fc70b2000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f7fc681c000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f7fc708a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7fc7545000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0
(0x00007f7fc6782000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007f7fc6755000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007f7fc6747000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007f7fc6740000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f7fc672f000)
libtirpc.so.3 => /lib/x86_64-linux-gnu/libtirpc.so.3 (0x00007f7fc6701000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x00007f7fc66d9000)
You really have a weird sshd
alarig@msi ~ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffd42dec000)
libcrypt.so.2 => /usr/lib64/libcrypt.so.2 (0x00007fb41f12b000)
libpam.so.0 => /usr/lib64/libpam.so.0 (0x00007fb41f11a000)
libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x00007fb41ece5000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb41eccb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb41eb0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb41f261000)
--
Alarig
3:24 p.m.
On 18 Apr 2024, at 16:10, Alarig Le Lay via BitFolk Users
<users(a)mailman.bitfolk.com> wrote:
You really have a weird sshd
alarig@msi ~ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffd42dec000)
libcrypt.so.2 => /usr/lib64/libcrypt.so.2 (0x00007fb41f12b000)
libpam.so.0 => /usr/lib64/libpam.so.0 (0x00007fb41f11a000)
libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x00007fb41ece5000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb41eccb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb41eb0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb41f261000)
I have the same /usr/sbin/sshd as Andy - looks like the one that comes with Debian 12.
Mike
3:47 p.m.
Some of the Kerberos dependencies can almost certainly be trimmed from a
modern ssh, I think.
Of all the discussion on the list, the one that resonated most was
hardening the main ssh by dropping unnecessary dependencies. Saying "you
can't use Kerberos to log in to the shell, and it won't be logged to
systemd" seems like it would inconvenience no one and gain a tiny bit of
extra peace of mind.
On the other hand, debugging why ssh won't let you log in when the cause is
a mismatch in crypto algorithms supported is a nightmare...
--scott
On Thu, Apr 18, 2024, 8:25 AM Mike Zanker via BitFolk Users <
users(a)mailman.bitfolk.com> wrote:
On 18 Apr 2024, at 16:10, Le Lay via BitFolk Users
<
users(a)mailman.bitfolk.com> wrote:
or send an email to
<users-leave(a)mailman.bitfolk.com>
You really have a weird sshd
alarig@msi ~ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffd42dec000)
libcrypt.so.2 => /usr/lib64/libcrypt.so.2 (0x00007fb41f12b000)
libpam.so.0 => /usr/lib64/libpam.so.0 (0x00007fb41f11a000)
libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x00007fb41ece5000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb41eccb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb41eb0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb41f261000)
I have the same /usr/sbin/sshd as Andy - looks like the one that comes
with Debian 12.
Mike
_______________________________________________
BitFolk Users mailing list <users(a)mailman.bitfolk.com>
You're subscribed as <cscott(a)cscott.net>
Unsubscribe: <
https://mailman.bitfolk.com/mailman/postorius/lists/users.mailman.bitfolk.c…
19 Apr
19 Apr
8:11 p.m.
Hi Scott,
This will involve compiling your own sshd. It means the end of binary
software distribution. As such it is a radical departure from Linux
distributions as most of us know them.
The Debian OpenSSH maintainers have been discussing how to get rid
of some of the dependencies or move them out into optional packages
(that most wouldn't use) and I think Kerberos was one of the
proposals.
Thanks,
Andy
On Thu, Apr 18, 2024 at 08:47:18AM -0700, C. Scott Ananian via BitFolk Users wrote:
Some of the Kerberos dependencies can almost certainly
be trimmed from a
modern ssh, I think.
Of all the discussion on the list, the one that resonated most was
hardening the main ssh by dropping unnecessary dependencies. Saying "you
can't use Kerberos to log in to the shell, and it won't be logged to
systemd" seems like it would inconvenience no one and gain a tiny bit of
extra peace of mind.
On the other hand, debugging why ssh won't let you log in when the cause is
a mismatch in crypto algorithms supported is a nightmare...
--scott
--
https://bitfolk.com/ -- No-nonsense VPS hosting
18 Apr
18 Apr
3:39 p.m.
On Thu, 18 Apr 2024 at 16:11, Alarig Le Lay via BitFolk Users
<users(a)mailman.bitfolk.com> wrote:
alarig@msi ~ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffd42dec000)
libcrypt.so.2 => /usr/lib64/libcrypt.so.2 (0x00007fb41f12b000)
libpam.so.0 => /usr/lib64/libpam.so.0 (0x00007fb41f11a000)
libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x00007fb41ece5000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb41eccb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb41eb0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb41f261000)
Is that something that can actually be a useful Xen dom0?
Graham
19 Apr
19 Apr
8:15 p.m.
Hello,
On Thu, Apr 18, 2024 at 05:10:48PM +0200, Alarig Le Lay via BitFolk Users wrote:
You really have a weird sshd
Binary distributions tend to have to include the whole kitchen sink
or explode the number of binary packages out to the matrix of every
possible combination of features.
Yes, this is me saying that sometimes Gentoo is better. 😀
alarig@msi ~ $ ldd /usr/sbin/sshd
linux-vdso.so.1 (0x00007ffd42dec000)
libcrypt.so.2 => /usr/lib64/libcrypt.so.2 (0x00007fb41f12b000)
libpam.so.0 => /usr/lib64/libpam.so.0 (0x00007fb41f11a000)
libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x00007fb41ece5000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb41eccb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb41eb0d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb41f261000)
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
15 Apr
15 Apr
3:44 p.m.
Hi,
On Sun, Apr 14, 2024 at 10:31:20AM +0100, iain via BitFolk Users wrote:
It might be useful to be able to block SSH access from
certain
regions (for example, China, Russia and MAGAland*)
BitFolk has customers in China, Russia and the USA so unless it
becomes illegal to have such customers, I do not expect to be
firewaling off those regions.
Hi to the China- and Russia-based customers reading.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
14 Apr
14 Apr
2:11 p.m.
Firewalling to restrict to my IP addresses would work well for me. My home Internet
connection is static IP, and when away from home I have a tablet with data-only SIM card
which also has a static ip address. I already do this with nftables on my VPS, and SSH
brute force attempts are blissfully absent from the logs, and of course it's
completely frictionless to use.
I realise this might not work for everyone, though. Some form of 2FA might do the trick,
better still, the option to do it either way, if that's possible. I wouldn't
object to a Web based console if it could be done well and safely but it does seem a
rather clunky solution.
The IPv6 only idea (if that's what it is) looks interesting, but I'm not sure I
fully understand it.
--
Anahata
Via Android tablet
01535 501017 - 07976 263827 - https://treewind.co.uk
On 14 April 2024 01:33:36 BST, Andy Smith via BitFolk Users
<users(a)mailman.bitfolk.com> wrote:
Hi,
In light of the recent XZ/lzma backdoor we should perhaps think
harder about how complex sshd is and the wisdom of exposing that to
the entire Internet.
At BitFolk, this currently needs to be exposed to the Internet so
that you can connect to your Xen Shell (console).
...
22
days inactive
27
days old
33 comments
14 participants
participants (14)
-
Alarig Le Lay -
Anahata -
Andy Bennett -
Andy Smith -
C. Scott Ananian -
Chris Smith -
Conrad Wood -
Ed Neville -
Graham Bleach -
iain -
James Gregory-Monk -
Kamal Shaker -
Mike Zanker -
Tim Bannister