Re: [bitfolk] The perils of opening tcp/22 to the Internet

Hi Andy, Thank you for this. Although I would consider myself an experience Linux administrator, I often ignore some of the advice I dole out to other people. This email (well, mainly the fact that if my VPS is compromised, I might not get it reinstated) has urged me to look at the issues raised in respect to my own VPS. I have always used keys, and always log on as root. I also always left password auth available to root, in case I was ever at a PC where I didn't have my keys available. I have now changed this so that root only accepts keys, and primarily I log onto another user account (albeit my first action is the "su -", but hey... one step at a time!). I have installed fail2ban, which I'm slowly configuring for my own needs. I have also instigated a full audit of everything I do on my VPS, to ensure that I only run those processes/jobs as root if they need to be - I used to run everything as root out of sheer laziness! So, thank you for the prompt, and for the suggestions. As far as my own input goes, please see below. Please note that I am a non-standard user with Bitfolk, as I use CentOS, not Debian, so some of my suggestions may be distro-specific (or at least, harder to implement on Debian). 1) Strong passwords can be generated at the shell using "mkpasswd", e.g. "mkpasswd -l 12 -d 3 -c 3 -C 3 -s 3 -2" will create a 12-digit password, with at least 3 numerical characters, 3 lowercase letters, 3 uppercase letters and 3 special characters. The "-2" option causes the characters to be chosen so that they alternate between the left and right hands, making for easier (and quicker!) typing. You can also use mkpasswd set the password immediately, which means it might be possible to script it into a web interface (e.g. when provisioning VPSes or resetting passwords). 2) I can see this deterring customers quite quickly, although I don't exactly know what an 'average' customer is - there seem to be people on this list with a lot less knowledge than others. In general, anything that's forced on you can cause angst - for example, if I had a 2-digit strong password, I could argue that I'm at far lower risk that most people from SSH hacking, and the benefit from forcing me to use keys is much smaller (albeit it still there) that for others, and it might just put my back up. 3) Disabling root access over SSH wouldn't be too bad - there's console access for those times when you *really* screw things up, and the rest of the time you can just "su -" or "sudo". While I'm touching on that - I really don't like sudo being suggested as a protective mechanism in this sense - it was originally designed to allow some administrative functionality to be given to non-admins, without giving them full-blown root access. These days, it's more of a protection from yourself (e.g. "Do I really want to sudo rm -rf /* ?"). If a user account has been compromised, then the hacker probably has the password, which means they can probably "sudo /bin/bash" and now have root access. Not good. Of course, "su -" is only better if your user and root passwords are different (and strong)! The problem with this idea is that it might lull you into a false sense of security - a new user might log in and immediately re-enable the root account login, without following any of the other security guidelines. I think it's best to provide try and tighten security against 'defaults' without using too many non-standard practices, otherwise you're back to square one when those practices are reverted to standard! 5) A good idea, although I've never used them before. I'll see how it goes on my box before I cast judgement. As an aside, I've been running some firewall rules for quite a while which are meant to limit the number of incoming SSH connections, as I used to see thousands of attempts per day (but no breaches luckily!): iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP 6) I don't like this. I agree it removes you as "low hanging fruit", but there are a lot of firewalls that block all outgoing ports <1024, apart from common ones. This would be a PITA for that situation. It would be ever more annoying as often you could use an SSH tunnel as a way of getting around these types of firewalls (e.g. for un-proxied web traffic), but this action would shoot yourself in the foot. That's my $0.02! :-) Paul -----Original Message----- From: users-bounces+bitfolk=pjlewis.org(a)lists.bitfolk.com [mailto:users-bounces+bitfolk=pjlewis.org@lists.bitfolk.com] On Behalf Of Andy Smith Sent: 14 March 2010 08:51 To: users(a)lists.bitfolk.com Subject: [bitfolk] The perils of opening tcp/22 to the Internet Hello, This very long email is about possible pro-active measures I could take to prevent customers being compromised by SSH dictionary attacks. The first part is just a recap of how we got here and what happens. If you want to make it shorter by skipping that, then skip to line 59 which begins with "Being compromised by an SSH dictionary attack..." Every so often I get an alert about the number of outbound SSH connections being much higher than usual. e.g. 100,000 instead of 4. I then investigate, and most of the time it's harmless -- someone doing a pentest or some sort of remote monitoring. But once every month or two, it's a customer's VPS which has been compromised and turned into an SSH scanning bot. As it happens, there's been two instances of this in the last month. When BitFolk first started, this sort of thing happened about three times before I decided I'd have to have monitoring of it so it wouldn't go on for days on end. When this happens and I catch it in the act, I have to turn off the customer's network, because their VPS is actively attacking remote sites, often at a rate of several hundred thousand SSH connections per hour. Aside from the hosts they may compromise, there will be abuse reports to deal with and possible damage to BitFolk's reputation, not to mention very large bandwidth usage that the customer is unlikely to want to pay for. I can't just block tcp/22 outbound because at this point the VPS is compromised and the priority (after stopping it) is understanding how it happened. Leaving the network working allows attackers more opportunity to clean up after themselves and/or do more damage. So, if you're the unlucky customer then at this point your VPS has no network, you've received an email (assuming your email is accessible from outside your VPS) about what's happened and you're going to have to log in to the console and/or rescue environment to try to work out how it happened. The rescue environment has access to your block devices and has unrestricted network access, but I'm not going to be able to turn networking back on for your VPS itself. If the customer can work out what happened, how it happened, and convince me that root access was not obtained, then I can turn their network back on. If they can't convince me that root access was not obtained then there's generally no option but to re-image the VPS. I'm really loathe to re-image a VPS that's been compromised where we don't know how, though, because it will likely just happen again. The customer has to work it out, and that's maybe something they're not used to doing, and/or don't want to be doing, while their service is down the whole time. Unfortunately I can't do it for them or help in any great detail because this is a time consuming task, and is the sysadmin's responsibility i.e. the customer. Being compromised by an SSH dictionary attack is far and away the most common thing that I see, and reacting to it by monitoring the rate of outbound SSH connections is no longer good enough. Reacting to it and cleaning up after it is wasting too much time, and when I have customers that have it happen to them repeatedly at some point I have to say enough is enough and not turn their service back on at all. Do you think there's any pro-active measures that would be acceptable to VPS customers? Typical ways to foil SSH dictionary attacks: 1) Only use strong passwords. 2) Don't use passwords at all, only keys. 3) Disable root login. 4) Restrict the list of usernames that are valid, in combination with (1) and (3). 5) Install DenyHosts or Fail2Ban. 6) Move sshd to another port. More? I can't do anything about (1), since the password the VPS comes with is strong and it's presumably made weaker later on. A lot of people have trouble setting up SSH keys and I would guess that very few customers have them before they get a VPS, so setting it up out of the box to require keys would be rather limiting. So that's (2) out. (3) is already the case for Ubuntu of course, but not any of the other distributions offered. I haven't kept track of how many compromises have been of root and not some other user but disabling root access by SSH and requiring some other username seems a reasonable starting point, would at least limit the damage. (4) sounds too confusing unless I stated in the setup email that SSH has been locked down to only the one non-root user they had when set up. Doing (5) in the base images would be controversial but I think actually very effective. Would it be an outrage for you to find DenyHosts or Fail2Ban installed for you? Moving the SSH port is probably the most effective of all, but I think it would be really confusing and not accepted by people, to find that their SSH was provisioned e.g on port 2022 to begin with. Any thoughts? -- http://bitfolk.com/ -- No-nonsense VPS hosting "It is I, Simon Quinlank. The chief conductor on the bus that is called hobby." -- Simon Quinlank