[bitfolk] The perils of opening tcp/22 to the Internet

Hello, This very long email is about possible pro-active measures I could take to prevent customers being compromised by SSH dictionary attacks. The first part is just a recap of how we got here and what happens. If you want to make it shorter by skipping that, then skip to line 59 which begins with "Being compromised by an SSH dictionary attack..." Every so often I get an alert about the number of outbound SSH connections being much higher than usual. e.g. 100,000 instead of 4. I then investigate, and most of the time it's harmless -- someone doing a pentest or some sort of remote monitoring. But once every month or two, it's a customer's VPS which has been compromised and turned into an SSH scanning bot. As it happens, there's been two instances of this in the last month. When BitFolk first started, this sort of thing happened about three times before I decided I'd have to have monitoring of it so it wouldn't go on for days on end. When this happens and I catch it in the act, I have to turn off the customer's network, because their VPS is actively attacking remote sites, often at a rate of several hundred thousand SSH connections per hour. Aside from the hosts they may compromise, there will be abuse reports to deal with and possible damage to BitFolk's reputation, not to mention very large bandwidth usage that the customer is unlikely to want to pay for. I can't just block tcp/22 outbound because at this point the VPS is compromised and the priority (after stopping it) is understanding how it happened. Leaving the network working allows attackers more opportunity to clean up after themselves and/or do more damage. So, if you're the unlucky customer then at this point your VPS has no network, you've received an email (assuming your email is accessible from outside your VPS) about what's happened and you're going to have to log in to the console and/or rescue environment to try to work out how it happened. The rescue environment has access to your block devices and has unrestricted network access, but I'm not going to be able to turn networking back on for your VPS itself. If the customer can work out what happened, how it happened, and convince me that root access was not obtained, then I can turn their network back on. If they can't convince me that root access was not obtained then there's generally no option but to re-image the VPS. I'm really loathe to re-image a VPS that's been compromised where we don't know how, though, because it will likely just happen again. The customer has to work it out, and that's maybe something they're not used to doing, and/or don't want to be doing, while their service is down the whole time. Unfortunately I can't do it for them or help in any great detail because this is a time consuming task, and is the sysadmin's responsibility i.e. the customer. Being compromised by an SSH dictionary attack is far and away the most common thing that I see, and reacting to it by monitoring the rate of outbound SSH connections is no longer good enough. Reacting to it and cleaning up after it is wasting too much time, and when I have customers that have it happen to them repeatedly at some point I have to say enough is enough and not turn their service back on at all. Do you think there's any pro-active measures that would be acceptable to VPS customers? Typical ways to foil SSH dictionary attacks: 1) Only use strong passwords. 2) Don't use passwords at all, only keys. 3) Disable root login. 4) Restrict the list of usernames that are valid, in combination with (1) and (3). 5) Install DenyHosts or Fail2Ban. 6) Move sshd to another port. More? I can't do anything about (1), since the password the VPS comes with is strong and it's presumably made weaker later on. A lot of people have trouble setting up SSH keys and I would guess that very few customers have them before they get a VPS, so setting it up out of the box to require keys would be rather limiting. So that's (2) out. (3) is already the case for Ubuntu of course, but not any of the other distributions offered. I haven't kept track of how many compromises have been of root and not some other user but disabling root access by SSH and requiring some other username seems a reasonable starting point, would at least limit the damage. (4) sounds too confusing unless I stated in the setup email that SSH has been locked down to only the one non-root user they had when set up. Doing (5) in the base images would be controversial but I think actually very effective. Would it be an outrage for you to find DenyHosts or Fail2Ban installed for you? Moving the SSH port is probably the most effective of all, but I think it would be really confusing and not accepted by people, to find that their SSH was provisioned e.g on port 2022 to begin with. Any thoughts? -- http://bitfolk.com/ -- No-nonsense VPS hosting "It is I, Simon Quinlank. The chief conductor on the bus that is called hobby." -- Simon Quinlank