Andy Smith wrote: [snip] Having had to install fail2ban, I feel that installing that in the image would be a positive bonus. Customers would need to be told about the "changes from standard" that have been applied to their image. I would like to see more "Newbie's Guides" in the support information. I remember struggling when I first signed up because I did not know enough about linux to know what I needed to do about security. I could do the basics, but I was no sysadmin (still am not). Perhaps add a wiki to the customer information that only customers can edit? Then recommendations and standard procedures can be documented and used without involving support. I do understand that adding to the existing pages involves a lot of detail work, for no clear benefit. The sort of thing that is needed, are some very basic guides covering the absolute fundamentals. The content would appear to be "obvious" and therefore unnecessary. Let me assure people that, when you don't know, it isn't obvious, and it often isn't clear how to find out either. Having a place to browse would be very helpful. If you know most if it, you are reassured that you haven't forgotten anything obvious. Regards Ian P.S. A "thumbs up" to the IRC support. It was very helpful and much appreciated when I needed it. Thanks guys.

Thanks Andy; this served as a reminder to double check / tweak some of my configuration. Andy Smith wrote: As you say, probably not acceptable to some folks, but perhaps you can at least default to asking only for an ssh key, and let people request a password explicitly or set a password themselves if they can't cope with keys? Certainly seems sensible; I thought most OSes denied it by default by I guess I'm wrong. At the very least: PermitRootLogin without-password should be acceptable (and perhaps better, as people might otherwise just change the 'no' to 'yes') No strong feeling on that one. I'd be quite happy with this. It's something you'd probably want to point out in the welcome email as well. (Is either of these better than the other, and do they generally need any tweaking from the default debian configuration?) One possible downside to that is that a number of ISPs are doing traffic prioritisation these days, and the default or normal settings for these systems seem to put uncommon ports into low or lowest priorities. That results in slow ssh connections if you're not careful in your port choice. (I think, for example, plus.net will put traffic to port 2222 at a low priority, but ssh traffic on the default jabber port seems fine.) Joseph

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 14/03/2010 08:51, Andy Smith wrote: I find this to be quite the best way of severely reducing, if not removing, the incidence of untargeted SSH dictionary attacks (I do keys / valid username restriction / no root login too). I can't remember the last time I saw a failed SSH attempt in the logs. Cheers Dominic

** Andy Smith &lt;andy(a)bitfolk.com&gt; [2010-03-14 16:22]: <snip> Is there any option to provide a facility to generate passwords to encourage customers to use better passwords. Either an https based web system or something like apg and some documentation. Perhaps a password change utility that generates a password as part of the process. Anyone less familiar with hosting would possible tend to use the documented system and hence get better passwords. It may be less popular, but would it work by creating a key that is available through the admin console with instructions on how to download it and use it to access the server for the first time using openssh or putty? I had been thinking about a slightly higher cost for a VPS with password access rather than key, but it is too easy to go for one and switch to the other, and a major nightmare to 'police'. A more practical suggestion may be an extra setup cost for anyone wanting password access on installation, but key by default. That would encourage learning how to use keys, and perhaps lead to continued usage. The sales pitch is increased security and with decent documentation it shouldn't be too much hassle. Should be standard to my mind, although as has been said, a compromised Ubuntu account has sudo access with the password that has already been compromised. Possibly not workable, again it depends on how easy it is for non-technical users to add usernames. SSH access my well be something that only the original account needs for non-technical customers. Perhaps looking into a control panel that allows some of these features to be set to make them easier to manage. Not a quick fix though. As a comparison, I have to manually create mail addresses allowed to be sent out through my ISP mail server relay for any domains not part of the sub-domain they provide. The system could be much better in that you can't use wildcards before the @ or delete entries without tech. support, but I assume they don't have too much complaint about it. I'd go for Fail2Ban as default personally, and it should be fairly easy to promote this as a benefit to hosting for the less technical customers. Those that are technical can easily disable it if preferred - again with some good documentation :) Not something I'm a big fan of. I suspect that many attacks will check for something as basic as this, although I've not seen any evidence or analysis of this theory! Perhaps some of this could be incorporated into the admin console, and this is the first place you log into. How easy would it be to run a script on first login to choose these options, with the defaults being the more secure choices? With the right 'blurb' I would expect many of the less experienced would choose those options described as more secure. Perhaps with a bit of gentle but firm 'encouragement' along with some custom documentation you can improve customer setups without upsetting anyone! ** end quote [Andy Smith] -- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001 ====================================================================== Registered in England | Company No: 4905028 | Registered Office: Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU

Andy Smith wrote: Well, I understand your problem. I only really like options 2, 3 and 5 (I like 1 but 2 is better) Fundamentally if you really want to admin a server part of the deal is about being a good neighbor. I understand you don't want to scare customers away and education is really, really difficult. Maybe there are some customers who don't really need admin access to a box, or rather, do need admin access to do what they require but don't have enough knowledge to do it safely. I have been thinking about this for a while and see a couple of alternatives in addition to promoting 1, 2, 3 and 5 above. - Allow password or key based provisioning but have some form of incremental tightening of security on boxes that have been compromised as part of the TOS. - Provide a financial incentive for the customer to request key-based provisioning and points 1, 2, 3, and 5 above. Think of it like insurance, your premiums are higher if you are more of a risk. - Offer sysadmin services for customers who require fine control over a machine but do not have the requisite knowledge to administer the machine. n NB: Yes, I had my machine provisioned via password but the first thing I did was lock it down with 2 and 3 above.

Hi Andy, Thank you for this. Although I would consider myself an experience Linux administrator, I often ignore some of the advice I dole out to other people. This email (well, mainly the fact that if my VPS is compromised, I might not get it reinstated) has urged me to look at the issues raised in respect to my own VPS. I have always used keys, and always log on as root. I also always left password auth available to root, in case I was ever at a PC where I didn't have my keys available. I have now changed this so that root only accepts keys, and primarily I log onto another user account (albeit my first action is the "su -", but hey... one step at a time!). I have installed fail2ban, which I'm slowly configuring for my own needs. I have also instigated a full audit of everything I do on my VPS, to ensure that I only run those processes/jobs as root if they need to be - I used to run everything as root out of sheer laziness! So, thank you for the prompt, and for the suggestions. As far as my own input goes, please see below. Please note that I am a non-standard user with Bitfolk, as I use CentOS, not Debian, so some of my suggestions may be distro-specific (or at least, harder to implement on Debian). 1) Strong passwords can be generated at the shell using "mkpasswd", e.g. "mkpasswd -l 12 -d 3 -c 3 -C 3 -s 3 -2" will create a 12-digit password, with at least 3 numerical characters, 3 lowercase letters, 3 uppercase letters and 3 special characters. The "-2" option causes the characters to be chosen so that they alternate between the left and right hands, making for easier (and quicker!) typing. You can also use mkpasswd set the password immediately, which means it might be possible to script it into a web interface (e.g. when provisioning VPSes or resetting passwords). 2) I can see this deterring customers quite quickly, although I don't exactly know what an 'average' customer is - there seem to be people on this list with a lot less knowledge than others. In general, anything that's forced on you can cause angst - for example, if I had a 2-digit strong password, I could argue that I'm at far lower risk that most people from SSH hacking, and the benefit from forcing me to use keys is much smaller (albeit it still there) that for others, and it might just put my back up. 3) Disabling root access over SSH wouldn't be too bad - there's console access for those times when you *really* screw things up, and the rest of the time you can just "su -" or "sudo". While I'm touching on that - I really don't like sudo being suggested as a protective mechanism in this sense - it was originally designed to allow some administrative functionality to be given to non-admins, without giving them full-blown root access. These days, it's more of a protection from yourself (e.g. "Do I really want to sudo rm -rf /* ?"). If a user account has been compromised, then the hacker probably has the password, which means they can probably "sudo /bin/bash" and now have root access. Not good. Of course, "su -" is only better if your user and root passwords are different (and strong)! The problem with this idea is that it might lull you into a false sense of security - a new user might log in and immediately re-enable the root account login, without following any of the other security guidelines. I think it's best to provide try and tighten security against 'defaults' without using too many non-standard practices, otherwise you're back to square one when those practices are reverted to standard! 5) A good idea, although I've never used them before. I'll see how it goes on my box before I cast judgement. As an aside, I've been running some firewall rules for quite a while which are meant to limit the number of incoming SSH connections, as I used to see thousands of attempts per day (but no breaches luckily!): iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP 6) I don't like this. I agree it removes you as "low hanging fruit", but there are a lot of firewalls that block all outgoing ports <1024, apart from common ones. This would be a PITA for that situation. It would be ever more annoying as often you could use an SSH tunnel as a way of getting around these types of firewalls (e.g. for un-proxied web traffic), but this action would shoot yourself in the foot. That's my $0.02! :-) Paul -----Original Message----- From: users-bounces+bitfolk=pjlewis.org(a)lists.bitfolk.com [mailto:users-bounces+bitfolk=pjlewis.org@lists.bitfolk.com] On Behalf Of Andy Smith Sent: 14 March 2010 08:51 To: users(a)lists.bitfolk.com Subject: [bitfolk] The perils of opening tcp/22 to the Internet Hello, This very long email is about possible pro-active measures I could take to prevent customers being compromised by SSH dictionary attacks. The first part is just a recap of how we got here and what happens. If you want to make it shorter by skipping that, then skip to line 59 which begins with "Being compromised by an SSH dictionary attack..." Every so often I get an alert about the number of outbound SSH connections being much higher than usual. e.g. 100,000 instead of 4. I then investigate, and most of the time it's harmless -- someone doing a pentest or some sort of remote monitoring. But once every month or two, it's a customer's VPS which has been compromised and turned into an SSH scanning bot. As it happens, there's been two instances of this in the last month. When BitFolk first started, this sort of thing happened about three times before I decided I'd have to have monitoring of it so it wouldn't go on for days on end. When this happens and I catch it in the act, I have to turn off the customer's network, because their VPS is actively attacking remote sites, often at a rate of several hundred thousand SSH connections per hour. Aside from the hosts they may compromise, there will be abuse reports to deal with and possible damage to BitFolk's reputation, not to mention very large bandwidth usage that the customer is unlikely to want to pay for. I can't just block tcp/22 outbound because at this point the VPS is compromised and the priority (after stopping it) is understanding how it happened. Leaving the network working allows attackers more opportunity to clean up after themselves and/or do more damage. So, if you're the unlucky customer then at this point your VPS has no network, you've received an email (assuming your email is accessible from outside your VPS) about what's happened and you're going to have to log in to the console and/or rescue environment to try to work out how it happened. The rescue environment has access to your block devices and has unrestricted network access, but I'm not going to be able to turn networking back on for your VPS itself. If the customer can work out what happened, how it happened, and convince me that root access was not obtained, then I can turn their network back on. If they can't convince me that root access was not obtained then there's generally no option but to re-image the VPS. I'm really loathe to re-image a VPS that's been compromised where we don't know how, though, because it will likely just happen again. The customer has to work it out, and that's maybe something they're not used to doing, and/or don't want to be doing, while their service is down the whole time. Unfortunately I can't do it for them or help in any great detail because this is a time consuming task, and is the sysadmin's responsibility i.e. the customer. Being compromised by an SSH dictionary attack is far and away the most common thing that I see, and reacting to it by monitoring the rate of outbound SSH connections is no longer good enough. Reacting to it and cleaning up after it is wasting too much time, and when I have customers that have it happen to them repeatedly at some point I have to say enough is enough and not turn their service back on at all. Do you think there's any pro-active measures that would be acceptable to VPS customers? Typical ways to foil SSH dictionary attacks: 1) Only use strong passwords. 2) Don't use passwords at all, only keys. 3) Disable root login. 4) Restrict the list of usernames that are valid, in combination with (1) and (3). 5) Install DenyHosts or Fail2Ban. 6) Move sshd to another port. More? I can't do anything about (1), since the password the VPS comes with is strong and it's presumably made weaker later on. A lot of people have trouble setting up SSH keys and I would guess that very few customers have them before they get a VPS, so setting it up out of the box to require keys would be rather limiting. So that's (2) out. (3) is already the case for Ubuntu of course, but not any of the other distributions offered. I haven't kept track of how many compromises have been of root and not some other user but disabling root access by SSH and requiring some other username seems a reasonable starting point, would at least limit the damage. (4) sounds too confusing unless I stated in the setup email that SSH has been locked down to only the one non-root user they had when set up. Doing (5) in the base images would be controversial but I think actually very effective. Would it be an outrage for you to find DenyHosts or Fail2Ban installed for you? Moving the SSH port is probably the most effective of all, but I think it would be really confusing and not accepted by people, to find that their SSH was provisioned e.g on port 2022 to begin with. Any thoughts? -- http://bitfolk.com/ -- No-nonsense VPS hosting "It is I, Simon Quinlank. The chief conductor on the bus that is called hobby." -- Simon Quinlank