Hello,
This weekend I'm off to OggCamp - http://oggcamp.org/
- if you're there too then do say hello! :)
Whilst there will be support cover and I should be available in
emergencies, this is also awkward timing because the next Ubuntu LTS
release is due for tomorrow.
Firstly I would say if you do have a support issue you need doing
before next week you should put it in now.
Secondly, if you are considering upgrading an Ubuntu VPS to 10.04 I
would recommend waiting until after the weekend. I haven't tried it
yet myself, though I do hope to do so. I think it should work, but
there might be some gotchas, and a lot of people breaking their
VPSes at once will stretch things this weekend. Plus if it does
break all I'll likely be able to do is put you back to 8.04 or
Debian.
Don't forget that I can snapshot your filesystem before you do a
major upgrade so if it all goes wrong we can roll it back to before
you started fairly easily. But you have to ask first.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
I don't live in UK and I maybe want to stream BBC web streams over ssh.
It was illegal before the law (te-hee), but now, according to the law, "ISPs
that fail to apply technical measures against subscribers can be fined up to
£250,000".
So, what now? BitFolk will start to monitor traffic and when certain clients
try to access BBC web streams, then they will get very angry letter saying
that they have been naughty?
Or, I'm exaggerating?
Hello,
As you may know, ext3 filesystem is by default set to require a fsck
at boot time based on both elapsed time since last fsck and number
of mounts since last fsck.
The time-based fsck can be painful, because when one of BitFolk's
servers is rebooted a high proportion of the VPSes on it won't have
been rebooted inside this time period (typically 6 months).
Therefore almost every VPS will fsck its filesystems at the same
time, causing massive IO load and a slow boot for everyone.
Having now realised this, I'm considering disabling the time-based
fsck by default.
Would it bother you to discover your VPS had been provided with
time-based fsck disabled?
Would it bother you if you one day discovered that time-based fsck
had been disabled for you without your knowledge since you aren't on
this mailing list?
In case you're interested, you can see the current settings like
this:
$ sudo tune2fs -l /dev/xvda | grep -i 'check\|mount count'
Mount count: 2
Maximum mount count: 34
Last checked: Sat Oct 17 09:10:33 2009
Check interval: 15552000 (6 months)
Next check after: Thu Apr 15 09:10:33 2010
And you can disable time-based fsck like this:
$ sudo tune2fs -i 0 /dev/xvda
tune2fs 1.41.3 (12-Oct-2008)
Setting interval between checks to 0 seconds
$ sudo tune2fs -l /dev/xvda | grep -i 'check\|mount count'
Mount count: 2
Maximum mount count: 34
Last checked: Sat Oct 17 09:10:33 2009
Check interval: 0 (<none>)
(replace /dev/xvda with whatever your partitions are -- see
/proc/partitions)
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
You dont have to be illiterate to use the Internet, but it help's.
-- Mike Bristow
Hi,
Apologies in advance if these are stupid questions...
1) With previous VPS providers, I've found that I've had to symlink
/dev/random to /dev/urandom to avoid an issue where SSL/TLS smtp
connections would hang for a long time waiting for sufficient entropy to
setup the secure connection. Is this something I need to do at Bitfolk too?
2) I've setup postfix (with TLS), and a self-signed certificate. This is
fine for my purposes, but I wondered if there was a risk that other
relay hosts would be unable to deliver email to my box if they weren't
able to validate the certificate? If so, is there a way of forcing other
relays to use non-secure connections, while retaining the ability to do
authenticated SMTP over TLS?
thanks in advance for any help with this!
Matt.
Hi all,
I seem to have started getting this error in the last 24 hours:
W: GPG error: http://apt-cacher.lon.bitfolk.com lenny/volatile Release:
The following signatures were invalid: KEYEXPIRED 1269969909 KEYEXPIRED
1269969909 KEYEXPIRED 1269969909
W: You may want to run apt-get update to correct these problems
apt-get update doesn't fix it, and the obvious google seemed only to
suggest clock syncronisation problems (the clock appears to be correct).
Any insight appreciated :)
Thanks
Joseph
Hello,
This very long email is about possible pro-active measures I could
take to prevent customers being compromised by SSH dictionary
attacks. The first part is just a recap of how we got here and what
happens. If you want to make it shorter by skipping that, then skip
to line 59 which begins with "Being compromised by an SSH dictionary
attack..."
Every so often I get an alert about the number of outbound SSH
connections being much higher than usual. e.g. 100,000 instead of 4.
I then investigate, and most of the time it's harmless -- someone
doing a pentest or some sort of remote monitoring.
But once every month or two, it's a customer's VPS which has been
compromised and turned into an SSH scanning bot. As it happens,
there's been two instances of this in the last month.
When BitFolk first started, this sort of thing happened about three
times before I decided I'd have to have monitoring of it so it
wouldn't go on for days on end.
When this happens and I catch it in the act, I have to turn off the
customer's network, because their VPS is actively attacking remote
sites, often at a rate of several hundred thousand SSH connections
per hour. Aside from the hosts they may compromise, there will be abuse
reports to deal with and possible damage to BitFolk's reputation,
not to mention very large bandwidth usage that the customer is
unlikely to want to pay for.
I can't just block tcp/22 outbound because at this point the VPS is
compromised and the priority (after stopping it) is understanding
how it happened. Leaving the network working allows attackers more
opportunity to clean up after themselves and/or do more damage.
So, if you're the unlucky customer then at this point your VPS has
no network, you've received an email (assuming your email is
accessible from outside your VPS) about what's happened and you're
going to have to log in to the console and/or rescue environment to
try to work out how it happened. The rescue environment has access to
your block devices and has unrestricted network access, but I'm not
going to be able to turn networking back on for your VPS itself.
If the customer can work out what happened, how it happened, and
convince me that root access was not obtained, then I can turn their
network back on. If they can't convince me that root access was not
obtained then there's generally no option but to re-image the VPS.
I'm really loathe to re-image a VPS that's been compromised where we
don't know how, though, because it will likely just happen again.
The customer has to work it out, and that's maybe something they're
not used to doing, and/or don't want to be doing, while their
service is down the whole time.
Unfortunately I can't do it for them or help in any great detail
because this is a time consuming task, and is the sysadmin's
responsibility i.e. the customer.
Being compromised by an SSH dictionary attack is far and away the
most common thing that I see, and reacting to it by monitoring the
rate of outbound SSH connections is no longer good enough. Reacting
to it and cleaning up after it is wasting too much time, and when
I have customers that have it happen to them repeatedly at some
point I have to say enough is enough and not turn their service
back on at all.
Do you think there's any pro-active measures that would be
acceptable to VPS customers? Typical ways to foil SSH dictionary
attacks:
1) Only use strong passwords.
2) Don't use passwords at all, only keys.
3) Disable root login.
4) Restrict the list of usernames that are valid, in combination
with (1) and (3).
5) Install DenyHosts or Fail2Ban.
6) Move sshd to another port.
More?
I can't do anything about (1), since the password the VPS comes with
is strong and it's presumably made weaker later on.
A lot of people have trouble setting up SSH keys and I would guess
that very few customers have them before they get a VPS, so setting
it up out of the box to require keys would be rather limiting. So
that's (2) out.
(3) is already the case for Ubuntu of course, but not any of the
other distributions offered. I haven't kept track of how many
compromises have been of root and not some other user but disabling
root access by SSH and requiring some other username seems a
reasonable starting point, would at least limit the damage.
(4) sounds too confusing unless I stated in the setup email that SSH
has been locked down to only the one non-root user they had when set
up.
Doing (5) in the base images would be controversial but I think
actually very effective. Would it be an outrage for you to find
DenyHosts or Fail2Ban installed for you?
Moving the SSH port is probably the most effective of all, but I
think it would be really confusing and not accepted by people, to
find that their SSH was provisioned e.g on port 2022 to begin with.
Any thoughts?
--
http://bitfolk.com/ -- No-nonsense VPS hosting
"It is I, Simon Quinlank. The chief conductor on the bus that is called
hobby." -- Simon Quinlank
Hi Andy.
I'm asking this on the mailing list so it can benefit anyone else that
needs to know, but contact me directly if needed.
One of my customers had an issue with data protection recently (not
website or IT related) and have had to review everything (a right PITA!).
This covers me as I host the their website with a database of registered
users.
I understand you have access to my server, both to the file system, and
root on the server itself with the backup SSH key.
While I trust you and accept that you won't do anything malicious,
Could you give something "official" or update your DPA page covering any
access you have to the VPS and its associated data?
http://www.bitfolk.com/policy/dpa.html
Much appreciated.
--
Dee Earley (dee(a)earlsoft.co.uk)
irc: irc://irc.blitzed.org/
web: http://www.earlsoft.co.uk
phone: +44 (0)780 8369596
>
> Yes I do think it is a problem with the keys as everything else is set up
> Sorry folks
>
>
> Message: 2
> Date: Thu, 15 Apr 2010 14:29:13 +0100
> From: "Tomalak Geret'kal" <tom(a)kera.name>
> Subject: Re: [bitfolk] Re VPS Security
> To: BitFolk Users <users(a)lists.bitfolk.com>
> Message-ID: <4BC714A9.4010306(a)kera.name>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>
> Sorry to be a pain Keith, but just as a quick aside is there
> any chance that you could please thread your replies, by
> including relevant quotations above your reply text and
> keeping the subject line intact, usually with a "Re: " prefix?
> Tom
> Keith - please use the "Reply" option in Gmail rather than composing a
> fresh e-mail each time. Then threading will work fine (even if you
> change the subject line).
>
> Cheers,
> John
>
>
>
> ------------------------------
>
> _______________________________________________
> users mailing list
> users(a)lists.bitfolk.com
> https://lists.bitfolk.com/mailman/listinfo/users
>
>
> End of users Digest, Vol 18, Issue 13
> *************************************
>
--
Keith
The most dangerous strategy is to jump a chasm in two leaps.
www.westnorfolkrspca.org.uk
James and John,
Thanks for your comments.
I have windows on my laptop, have to I'm afraid because of some software
that is used by a group of which I am a member. But I do have a Linux box
running Ubuntu, so perhaps I would do better using that, at least for the
sys administration thing before I get onto my web work.
Can't for the life of me think why I set the execute permissions - I put it
down to age.
Yes I do think it is a problem with the keys as everything else is set up
properly, barring any silly typo type errors. One of which I did discover
was naming the file authorised_keys (English rather than American spelling)
but I discovered that early on.
It is frustrating as it is a simple procedure so it must be a simple error'
Keith
--
Keith
The most dangerous strategy is to jump a chasm in two leaps.
www.westnorfolkrspca.org.uk
I do apologise if this seems a stupid question.
I am struggling with this and probably missing something very basic. Have
done the initials - changed password to strong random one. Set up iptables -
all ports closed off except port 22 and 80. Fail2Ban running and configured,
I will change ssh to another port later. Set up non-root user, let's call
him fred - with again a strong password.
Checked sshd config file - using protocol 2. Used puttygen to generate
keypair, private key secured with a near-gibberish longish passphrase,
loaded private key into pageant. Saved public key to
/home/fred/.ssh/authorised_keys then chmod that to 744 and .ssh directory
(tried them also at 700 same effect). Changed PasswordAuthorisation in sshd
config file to no
Fire up Putty set VPS ip goto to data page and put in fred for username then
to ssh>auth page to set attempt using pageant and select the private key
file to use. Press open and lo and behold, it asks me for password. I put
that in and I am logged on. Why is it using passwords, why ignoring keys?
At my age you can't afford to lose too much hair, but I am pulling it out
over this
Keith
--
Keith
The most dangerous strategy is to jump a chasm in two leaps.
www.westnorfolkrspca.org.uk
