Hi all,
I seem to have started getting this error in the last 24 hours:
W: GPG error: http://apt-cacher.lon.bitfolk.com lenny/volatile Release:
The following signatures were invalid: KEYEXPIRED 1269969909 KEYEXPIRED
1269969909 KEYEXPIRED 1269969909
W: You may want to run apt-get update to correct these problems
apt-get update doesn't fix it, and the obvious google seemed only to
suggest clock syncronisation problems (the clock appears to be correct).
Any insight appreciated :)
Thanks
Joseph
Hello,
This very long email is about possible pro-active measures I could
take to prevent customers being compromised by SSH dictionary
attacks. The first part is just a recap of how we got here and what
happens. If you want to make it shorter by skipping that, then skip
to line 59 which begins with "Being compromised by an SSH dictionary
attack..."
Every so often I get an alert about the number of outbound SSH
connections being much higher than usual. e.g. 100,000 instead of 4.
I then investigate, and most of the time it's harmless -- someone
doing a pentest or some sort of remote monitoring.
But once every month or two, it's a customer's VPS which has been
compromised and turned into an SSH scanning bot. As it happens,
there's been two instances of this in the last month.
When BitFolk first started, this sort of thing happened about three
times before I decided I'd have to have monitoring of it so it
wouldn't go on for days on end.
When this happens and I catch it in the act, I have to turn off the
customer's network, because their VPS is actively attacking remote
sites, often at a rate of several hundred thousand SSH connections
per hour. Aside from the hosts they may compromise, there will be abuse
reports to deal with and possible damage to BitFolk's reputation,
not to mention very large bandwidth usage that the customer is
unlikely to want to pay for.
I can't just block tcp/22 outbound because at this point the VPS is
compromised and the priority (after stopping it) is understanding
how it happened. Leaving the network working allows attackers more
opportunity to clean up after themselves and/or do more damage.
So, if you're the unlucky customer then at this point your VPS has
no network, you've received an email (assuming your email is
accessible from outside your VPS) about what's happened and you're
going to have to log in to the console and/or rescue environment to
try to work out how it happened. The rescue environment has access to
your block devices and has unrestricted network access, but I'm not
going to be able to turn networking back on for your VPS itself.
If the customer can work out what happened, how it happened, and
convince me that root access was not obtained, then I can turn their
network back on. If they can't convince me that root access was not
obtained then there's generally no option but to re-image the VPS.
I'm really loathe to re-image a VPS that's been compromised where we
don't know how, though, because it will likely just happen again.
The customer has to work it out, and that's maybe something they're
not used to doing, and/or don't want to be doing, while their
service is down the whole time.
Unfortunately I can't do it for them or help in any great detail
because this is a time consuming task, and is the sysadmin's
responsibility i.e. the customer.
Being compromised by an SSH dictionary attack is far and away the
most common thing that I see, and reacting to it by monitoring the
rate of outbound SSH connections is no longer good enough. Reacting
to it and cleaning up after it is wasting too much time, and when
I have customers that have it happen to them repeatedly at some
point I have to say enough is enough and not turn their service
back on at all.
Do you think there's any pro-active measures that would be
acceptable to VPS customers? Typical ways to foil SSH dictionary
attacks:
1) Only use strong passwords.
2) Don't use passwords at all, only keys.
3) Disable root login.
4) Restrict the list of usernames that are valid, in combination
with (1) and (3).
5) Install DenyHosts or Fail2Ban.
6) Move sshd to another port.
More?
I can't do anything about (1), since the password the VPS comes with
is strong and it's presumably made weaker later on.
A lot of people have trouble setting up SSH keys and I would guess
that very few customers have them before they get a VPS, so setting
it up out of the box to require keys would be rather limiting. So
that's (2) out.
(3) is already the case for Ubuntu of course, but not any of the
other distributions offered. I haven't kept track of how many
compromises have been of root and not some other user but disabling
root access by SSH and requiring some other username seems a
reasonable starting point, would at least limit the damage.
(4) sounds too confusing unless I stated in the setup email that SSH
has been locked down to only the one non-root user they had when set
up.
Doing (5) in the base images would be controversial but I think
actually very effective. Would it be an outrage for you to find
DenyHosts or Fail2Ban installed for you?
Moving the SSH port is probably the most effective of all, but I
think it would be really confusing and not accepted by people, to
find that their SSH was provisioned e.g on port 2022 to begin with.
Any thoughts?
--
http://bitfolk.com/ -- No-nonsense VPS hosting
"It is I, Simon Quinlank. The chief conductor on the bus that is called
hobby." -- Simon Quinlank
Hi Andy.
I'm asking this on the mailing list so it can benefit anyone else that
needs to know, but contact me directly if needed.
One of my customers had an issue with data protection recently (not
website or IT related) and have had to review everything (a right PITA!).
This covers me as I host the their website with a database of registered
users.
I understand you have access to my server, both to the file system, and
root on the server itself with the backup SSH key.
While I trust you and accept that you won't do anything malicious,
Could you give something "official" or update your DPA page covering any
access you have to the VPS and its associated data?
http://www.bitfolk.com/policy/dpa.html
Much appreciated.
--
Dee Earley (dee(a)earlsoft.co.uk)
irc: irc://irc.blitzed.org/
web: http://www.earlsoft.co.uk
phone: +44 (0)780 8369596
Hello,
Some people don't like that this list has no Reply-To: header
directed back at the list.
Whilst I believe the way it's set up is technically correct and is
what I personally prefer, I'm willing to go with whatever makes the
most people happy.
Here's a poll: http://doodle.com/bmfbsa3paah2w97a
If you care about how the list is set up, please take a moment to
submit that web form.
I'd prefer if only people who have ever actually posted to this list
participated.
And please no discussion on this matter on-list - I think everything
that's ever been said on the subject is contained in the two links
listed on the poll, and no one's mind is going to be changed. If you
absolutely must make a comment, there's a comment thing on the poll.
Thanks!
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
"I'd be happy to buy all variations of sex to ensure I got what I wanted."
-- Gary Coates (talking about cabling)
I'm thinking of changing my vps to debian from ubuntu
I'm not sure the best way to do it
I was thinking of maybe paying for an extra months hosting and having a 2nd
server setup with a clean install and leaving me a month to get everything
transferred over and tested before taking the old 1 down
i was wondering if anyone has done anything like this or can see any
problems with doing it like that or has a better idea
Being a bit of a Bitfolk newbie and a gentoo rice racer, I was wondering if
anyone could tell me what the best -march setting would be? I understand
that its only really a concern if the VM was migrated to another host with
differing CPU features. Does this ever happen or are VM's fairly tightly
coupled with the host?
Cheers,
Ashley Nicholls
--
---------
Open all hailing frequencies and broadcast in all known languages. Including
Welsh.
- Arnold J. Rimmer
Has anyone managed to get a Debian Xen kernel working on the Ubuntu 8.04 LTS
release on a Bitfolk VPS? I'm having problems with stability, but haven't
managed to get the Debian kernel to boot yet to see if it is any better. It
would save me some down time and experimentation if somebody has already done
it :) My next step will have to be abandoning Ubuntu and heading back to
Debian, but it would be nicer to be standardised on Ubuntu for my servers and
then there's all the hassle of another rebuild and configure.
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
======================================================================
Registered in England | Company No: 4905028 | Registered Office:
Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU
Hello,
I've added a section in the panel for managing the SSH public keys
of your Xen Shell console. That is what you connect to when you do:
ssh username(a)username.console.bitfolk.com
You'll find it at:
https://panel.bitfolk.com/account/security/
If you already had keys added, it would be good if you could have a
look at it and make sure that the keys you expect to be there are
there.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
You dont have to be illiterate to use the Internet, but it help's.
-- Mike Bristow
Hi,
At approximately 1550, dunkel lost power because someone doing
unrelated work in the rack accidentally dislodged the power cord. I
was alerted a few minutes later and took a few more minutes to
assess the situation.
The last few VPSes are in the process of starting now.
If you're on dunkel and you're still seeing problems after about
4.30pm, please contact support.
Please accept my apologies for the disruption. I will be passing on
some service credit, the amount of which will depend on how much I
get back from the provider but it will be more than zero.
Cheers,
Andy
