7 Dec
2012
7 Dec
'12
2:19 a.m.
Hello,
From time to time BitFolk customer VPSes occasionally become subject
to various kinds of compromise. Frustratingly, the kinds of
compromise encountered are generally the result of run of the mill,
completely preventable and unremarkable root causes.
I would like to find a way to raise awareness of these very simple
security concerns amongst the customer base, in order to hopefully
cut down on how often they happen.
I was thinking that if customers saw how often these things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
So I was contemplating posting an email thread to this ("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
It might look something like this:
Today at around 04:30 we became aware of a customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may only become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
If the customer is unable/unwilling to do this then we may never
know why their VPS began misbehaving. We don't examine customer data
unless given permission to do so, and even then this is often too
time-consuming to undertake on an unpaid basis. I would consider the
above an example of the maximum amount of detail we would go into.
No identifying information regarding the affected customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Would this sort of posting be welcomed or would it be unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
Please also note that those with an extremely low tolerance for
email noise may wish to quit this list and instead join the
"announce" list, as it contains only announcements from BitFolk with
no customer discussion whatsoever:
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Thoughts?
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
7 Dec
7 Dec
4:50 a.m.
I think it's a great idea. Hearing what is actually going on in the real
world helps awareness for the rest of us sharing it. I reserve the right
to eat these words if it all turns out to be too depressing...!
Ross
5:44 a.m.
Agree, think it's a good idea.
On 7 December 2012 06:50, Ross Younger <ross(a)impropriety.org.uk> wrote:
I think it's a great idea. Hearing what is
actually going on in the real
world helps awareness for the rest of us sharing it. I reserve the right to
eat these words if it all turns out to be too depressing...!
Ross
______________________________**_________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/**mailman/listinfo/users<https://lists.bitfolk…
6:05 a.m.
On 2012/12/07 4:19 AM, Andy Smith wrote:
I was thinking that if customers saw how often these
things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
You could also consider creating another mailing list. Perhaps
"security(a)bitfolk.com" or "compromise(a)bitfolk.com"?
Whether you do this or use users@, I would definitely be interested,
even though most of these won't affect me[1].
It might look something like this:
Today at around 04:30 we became aware of a customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may only
become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
Of course.
No identifying information regarding the affected
customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Of course :)
Would this sort of posting be welcomed or would it be
unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
I would welcome it.
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Heh. Even our company's announce lists have got 100s of mails this year.
Some 1000s.
[1] I allow incoming :1194UDP (openvpn) and :80TCP(web) publicly on my
vps. Without the static openvpn key you can't do anything but browse the
single domain hosted on it. All other access happen via a VPN tunnel.
That said every service is still secured as if it was public (SSH only
via authorized_keys, etc). So even if openvpn gets compromised you still
need to get through that.
7:43 a.m.
Great idea
On 7 December 2012 06:05, Peet Grobler <peet(a)peet.za.net> wrote:
On 2012/12/07 4:19 AM, Andy Smith wrote:
--
Keith Williams
www.PhilsArt.co.uk
"Time is an illusion. Lunchtime doubly so." Douglas Adams
He's done it again! www.justgiving.com/France-The-Wrong-Way
Tailor Made English www.tmenglish.org
I was thinking that if customers saw how often
these things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
You could also consider creating another mailing list. Perhaps
"security(a)bitfolk.com" or "compromise(a)bitfolk.com"?
Whether you do this or use users@, I would definitely be interested,
even though most of these won't affect me[1].
It might look something like this:
Today at around 04:30 we became aware of a customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may
only become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
Of course.
No identifying information regarding the affected
customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Of course :)
Would this sort of posting be welcomed or would
it be unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
I would welcome it.
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Heh. Even our company's announce lists have got 100s of mails this year.
Some 1000s.
[1] I allow incoming :1194UDP (openvpn) and :80TCP(web) publicly on my
vps. Without the static openvpn key you can't do anything but browse the
single domain hosted on it. All other access happen via a VPN tunnel.
That said every service is still secured as if it was public (SSH only
via authorized_keys, etc). So even if openvpn gets compromised you still
need to get through that.
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
8:37 a.m.
I think the users@ list is sufficiently low volume and the compromise
rate is (I hope) sufficiently low that users@ would be the best place to
do this. If there ends up being enough traffic to warrant a separate
mailing list (heaven forbid), I suggest an announcement on users@ and
the discussion continuing on a separate list would be the way to go.
--
Phil
On 07/12/2012 07:43, Keith Williams wrote:
Great idea
On 7 December 2012 06:05, Peet Grobler <peet(a)peet.za.net
<mailto:peet@peet.za.net>> wrote:
On 2012/12/07 4:19 AM, Andy Smith wrote:
I was thinking that if customers saw how often
these things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
You could also consider creating another mailing list. Perhaps
"security(a)bitfolk.com <mailto:security@bitfolk.com>" or
"compromise(a)bitfolk.com <mailto:compromise@bitfolk.com>"?
Whether you do this or use users@, I would definitely be interested,
even though most of these won't affect me[1].
It might look something like this:
Today at around 04:30 we became aware of a customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may
only become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
Of course.
No identifying information regarding the affected
customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Of course :)
Would this sort of posting be welcomed or would
it be unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
I would welcome it.
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Heh. Even our company's announce lists have got 100s of mails this
year.
Some 1000s.
[1] I allow incoming :1194UDP (openvpn) and :80TCP(web) publicly on my
vps. Without the static openvpn key you can't do anything but
browse the
single domain hosted on it. All other access happen via a VPN tunnel.
That said every service is still secured as if it was public (SSH only
via authorized_keys, etc). So even if openvpn gets compromised you
still
need to get through that.
_______________________________________________
users mailing list
users(a)lists.bitfolk.com <mailto:users@lists.bitfolk.com>
https://lists.bitfolk.com/mailman/listinfo/users
--
Keith Williams
www.PhilsArt.co.uk <http://www.PhilsArt.co.uk>
"Time is an illusion. Lunchtime doubly so." Douglas Adams
He's done it again! www.justgiving.com/France-The-Wrong-Way
<http://www.justgiving.com/France-The-Wrong-Way>
Tailor Made English www.tmenglish.org <http://www.tmenglish.org>
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
8:51 a.m.
Hi,
On Fri, Dec 07, 2012 at 02:19:42AM +0000, Andy Smith wrote:
So I was contemplating posting an email thread to this
("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
Someone asked off-list how many security incidents there had been in
the last year.
It's a little difficult to answer because I haven't been keeping a
structured record. There are support tickets for each one that we
know of, but they aren't in a standard format so are a little
difficult to search for.
After a quick search with terms that immediately spring to mind I can
see:
- 1 Customer VPS doing FTP dictionary attacks.
Source of compromise as-yet unknown (customer has allowed me to
take a look but I haven't yet got around to it).
- 5 customer VPSes doing SSH dictionary attacks, which were
themselves compromised by SSH dictionary attacks. One of these
was a root account. In all cases the customer was running sshd on
port 22 with password authentication enabled.
- 2 cases of customer VPSes engaged in direct denial of service
attacks (packeting).
One was broken into through an insecure Joomla template.
The other was cause unknown since the customer never responded,
but was probably fraudulently purchased from the beginning since
the PayPal transaction was also disputed.
- 2 customers participating in a DDoS through their open DNS
resolvers:
http://bitfolk.com/orns.html
- 4 customers hosting bank phishing sites (i.e. they're hosting
pages that are made to look like bank sites which people are
directed to, and then after their info is filled in, the info is
sent to the attacker).
Cause as-yet unknown for one of these because it happened only
last night.
Cause for one was a Wordpress exploit that allowed the attacker to
upload a page of their own content. Not sure whether that was a
case of an out of date Wordpress install or a bad plugin.
Cause for one was an unknown CMS which customer left admin
unpassworded.
Cause for last one was never determined and customer did not give
permission for us to examine. VPS was reinstalled.
- At least 20 reports of drones connecting to their C&C¹ channels
through Tor exit nodes hosted by BitFolk customers.
It's still abuse but there's nothing that can be done since it's
Tor.
There'll probably be a couple more that I was unable to find, but
that's the gist of it.
Cheers,
Andy
¹ Command and Control -
http://www.shadowserver.org/wiki/pmwiki.php/Information/Botnets#commandandc…
--
http://bitfolk.com/ -- No-nonsense VPS hosting
"I'd be happy to buy all variations of sex to ensure I got what I wanted."
— Gary Coates (talking about cabling)
9:49 a.m.
On Fri, Dec 07, 2012 at 08:51:17AM +0000, Andy Smith wrote:
There'll probably be a couple more that I was
unable to find, but
that's the gist of it.
Hi.
I got hit by a dictionary attack on SMTP AUTH (well it looked like that
was how they got the login) that was then used to send out spam.
Thankfully the volume was fairly low.
My lessons:
a) SMTP AUTH actually does get attacked - keep an eye on it and use
solid passwords.
b) It's fairly easy to setup fail2ban to monitor exim.
Michael
2:56 p.m.
Hi,
I got hit by a dictionary attack on SMTP AUTH (well it
looked like that
was how they got the login) that was then used to send out spam.
Thankfully the volume was fairly low.
My lessons:
a) SMTP AUTH actually does get attacked - keep an eye on it and use
solid passwords.
Yes: these are real. I've also seen "reverse dictionary attacks" where
an attacker chooses a handful of popular passwords and then tries to
brute-force the username.
Almost all the attacks I have seen involve "unqualified" usernames such
as "andyjpb" rather than "andyjpb(a)ashurst.eu.org".org". For this and many
other reasons, I'd recommend using fully qualified usernames for SMTP AUTH.
Regards,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
9:01 a.m.
On 07/12/2012 02:19, Andy Smith wrote:
So I was contemplating posting an email thread to this
("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
Great idea - it can only help improve our own security measures to see
how other users are being compromised.
Mike
9:52 a.m.
I think it is an excellent idea Andy!
If the volume is low (as your later post suggests), personally I se no
need to create yet another e-mail list for this. A subject line
starting with a tag like [general security alert] would probably help
people like me. Where the word "general" is the key. If I receive an
e-mail saying [security alert] or such it would require immediate
attention, whilst a general security alert is of a slightly lesser
urgency . But that's just semantics. I'd be happy with whatever
solution you come up with. This kind of info is, just like you write,
quite interesting and enlightening.
Cheers,
__
/ony
-------
Friday, December 7, 2012, 2:19:42 AM, Andy wrote:
Return-Path:
<users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com>
X-Original-To: BitFolkList(a)tony-andersson.com
Delivered-To: BitFolkList(a)tony-andersson.com
Received: by tony-andersson.com (Postfix, from userid 500)
id F090B24008; Fri, 7 Dec 2012 02:19:46 +0000 (GMT)
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
spamd3.lon.bitfolk.com
X-Spam-Level:
X-Spam-ASN:
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT
shortcircuit=ham autolearn=disabled version=3.3.1
X-Spam-Report:
* -0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited rule
* -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
Received: from mail.bitfolk.com (bitfolk.com [85.119.80.223])
by tony-andersson.com (Postfix) with ESMTPS id CDB5524007
for <BitFolkList(a)tony-andersson.com>om>; Fri, 7 Dec 2012 02:19:46 +0000
(GMT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bitfolk.com; s=alpha;
h=Sender:Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version:Message-ID:To:From:Date;
bh=vRbIloMoG9gJ141i3a7pQTJwQvEPRJCMNXFddRhCqVw=;
b=NxPuc0+iwzaEN71o7gWpkatFlLBIa6VbsG3NyqWcaNeYmSPICkTDeE7lSNBNxJTkYf6Qjd5aA7LejgILtndux+t/cLXeYgjQpCIVUBp1/19AkTs9HrWRPAUWF6cDYGv6;
Received: from localhost ([127.0.0.1] helo=bitfolk.com)
by mail.bitfolk.com with esmtp (Exim 4.72)
(envelope-from
<users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com>)
id 1TgnXW-0001Mr-K4
for BitFolkList(a)tony-andersson.com; Fri, 07 Dec 2012 02:19:46 +0000
Received: from andy by mail.bitfolk.com with local (Exim 4.72)
(envelope-from <andy(a)bitfolk.com>) id 1TgnXS-0001Lk-6E
for users(a)lists.bitfolk.com; Fri, 07 Dec 2012 02:19:42 +0000
Date: Fri, 7 Dec 2012 02:19:42 +0000
From: Andy Smith <andy(a)bitfolk.com>
To: users(a)lists.bitfolk.com
Message-ID: <20121207021942.GT3867(a)bitfolk.com>
MIME-Version: 1.0
OpenPGP: id=BF15490B; url=http://strugglers.net/~andy/pubkey.asc
X-URL: http://strugglers.net/wiki/User:Andy
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Fri,
07 Dec 2012 02:19:42 +0000
Subject: [bitfolk] Proposal: Security incidents postings
X-BeenThere: users(a)lists.bitfolk.com
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
<mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
List-Post: <mailto:users@lists.bitfolk.com>
List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
<mailto:users-request@lists.bitfolk.com?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1702776325=="
Sender: users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com
Errors-To:
users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Fri, 07 Dec 2012 02:19:46
+0000
X-SA-Exim-Connect-IP: 127.0.0.1
X-SA-Exim-Mail-From:
users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com
X-SA-Exim-Scanned: No (on mail.bitfolk.com); SAEximRunCond expanded to false
Hello,
From time to time BitFolk customer VPSes occasionally
become subject
to various kinds of compromise. Frustratingly, the kinds of
compromise encountered are generally the result of run of the mill,
completely preventable and unremarkable root causes.
I would like to find a way to raise awareness of these
very simple
security concerns amongst the customer base, in order to hopefully
cut down on how often they happen.
I was thinking that if customers saw how often these
things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
So I was contemplating posting an email thread to this
("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
It might look something like this:
Today at around 04:30 we became aware of a
customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the
VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may only
become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
If the customer is unable/unwilling to do this then we
may never
know why their VPS began misbehaving. We don't examine customer data
unless given permission to do so, and even then this is often too
time-consuming to undertake on an unpaid basis. I would consider the
above an example of the maximum amount of detail we would go into.
No identifying information regarding the affected
customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Would this sort of posting be welcomed or would it be
unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
Please also note that those with an extremely low
tolerance for
email noise may wish to quit this list and instead join the
"announce" list, as it contains only announcements from BitFolk with
no customer discussion whatsoever:
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Thoughts?
Cheers,
Andy
9:57 a.m.
Sorry about the excessive headers, that's a user error. Silly me! I'll
go to my room and think about what I have done. Sorry!
__
/ony
-------
Friday, December 7, 2012, 9:52:51 AM, Tony wrote:
I think it is an excellent idea Andy!
If the volume is low (as your later post suggests), personally I se no
need to create yet another e-mail list for this. A subject line
starting with a tag like [general security alert] would probably help
people like me. Where the word "general" is the key. If I receive an
e-mail saying [security alert] or such it would require immediate
attention, whilst a general security alert is of a slightly lesser
urgency . But that's just semantics. I'd be happy with whatever
solution you come up with. This kind of info is, just like you write,
quite interesting and enlightening.
Cheers,
__
/ony
-------
Friday, December 7, 2012, 2:19:42 AM, Andy wrote:
> Return-Path:
> <users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com>
> X-Original-To: BitFolkList(a)tony-andersson.com
> Delivered-To: BitFolkList(a)tony-andersson.com
> Received: by tony-andersson.com (Postfix, from userid 500)
> id F090B24008; Fri, 7 Dec 2012 02:19:46 +0000 (GMT)
> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
> spamd3.lon.bitfolk.com
> X-Spam-Level:
> X-Spam-ASN:
> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT
> shortcircuit=ham autolearn=disabled version=3.3.1
> X-Spam-Report:
> * -0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited rule
> * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
> Received: from mail.bitfolk.com (bitfolk.com [85.119.80.223])
> by tony-andersson.com (Postfix) with ESMTPS id CDB5524007
> for <BitFolkList(a)tony-andersson.com>om>; Fri, 7 Dec 2012 02:19:46 +0000
(GMT)
> DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bitfolk.com;
s=alpha;
>
>
h=Sender:Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version:Message-ID:To:From:Date;
> bh=vRbIloMoG9gJ141i3a7pQTJwQvEPRJCMNXFddRhCqVw=;
>
>
b=NxPuc0+iwzaEN71o7gWpkatFlLBIa6VbsG3NyqWcaNeYmSPICkTDeE7lSNBNxJTkYf6Qjd5aA7LejgILtndux+t/cLXeYgjQpCIVUBp1/19AkTs9HrWRPAUWF6cDYGv6;
> Received: from localhost ([127.0.0.1] helo=bitfolk.com)
> by mail.bitfolk.com with esmtp (Exim 4.72)
> (envelope-from
> <users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com>)
> id 1TgnXW-0001Mr-K4
> for BitFolkList(a)tony-andersson.com; Fri, 07 Dec 2012 02:19:46 +0000
> Received: from andy by mail.bitfolk.com with local (Exim 4.72)
> (envelope-from <andy(a)bitfolk.com>) id 1TgnXS-0001Lk-6E
> for users(a)lists.bitfolk.com; Fri, 07 Dec 2012 02:19:42 +0000
> Date: Fri, 7 Dec 2012 02:19:42 +0000
> From: Andy Smith <andy(a)bitfolk.com>
> To: users(a)lists.bitfolk.com
> Message-ID: <20121207021942.GT3867(a)bitfolk.com>
> MIME-Version: 1.0
> OpenPGP: id=BF15490B; url=http://strugglers.net/~andy/pubkey.asc
> X-URL: http://strugglers.net/wiki/User:Andy
> User-Agent: Mutt/1.5.20 (2009-06-14)
> X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Fri,
> 07 Dec 2012 02:19:42 +0000
> Subject: [bitfolk] Proposal: Security incidents postings
> X-BeenThere: users(a)lists.bitfolk.com
> X-Mailman-Version: 2.1.13
> Precedence: list
> List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
> List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
> <mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
> List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
> List-Post: <mailto:users@lists.bitfolk.com>
> List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
> List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
> <mailto:users-request@lists.bitfolk.com?subject=subscribe>
> Content-Type: multipart/mixed; boundary="===============1702776325=="
> Sender: users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com
> Errors-To:
> users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com
> X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Fri, 07 Dec 2012 02:19:46
+0000
> X-SA-Exim-Connect-IP: 127.0.0.1
> X-SA-Exim-Mail-From:
> users-bounces+bitfolklist=tony-andersson.com(a)lists.bitfolk.com
> X-SA-Exim-Scanned: No (on mail.bitfolk.com); SAEximRunCond expanded to false
> Hello,
> From time to time BitFolk customer VPSes
occasionally become subject
> to various kinds of compromise. Frustratingly, the kinds of
> compromise encountered are generally the result of run of the mill,
> completely preventable and unremarkable root causes.
> I would like to find a way to raise awareness of
these very simple
> security concerns amongst the customer base, in order to hopefully
> cut down on how often they happen.
> I was thinking that if customers saw how often
these things happen
> to people very much like themselves then it might help remove some
> of the "yeah I've heard of that but it will never happen to me"
> mindset that we all regrettably can fall into.
> So I was contemplating posting an email thread to
this ("users")
> list every time we become aware of a customer compromise, and I was
> wondering what you thought of that idea.
> It might look something like this:
> Today at around 04:30 we became aware of a
customer VPS
> initiating an abnormal amount of outbound SSH connections (~200
> per second). The VPS's network access was suspended and customer
> contacted.
> It was later determined that a user account on
the VPS had been
> accessed starting 3 days ago, via an SSH dictionary attack. The
> attacker installed another copy of the SSH dictionary attack
> software and set it going. We do not believe that root access
> was obtained.
> The amount of detail would vary because we may
only become aware of
> a compromise when the customer's VPS itself starts perpetrating
> abusive activity, and then we rely on the customer to investigate
> why that is.
> If the customer is unable/unwilling to do this
then we may never
> know why their VPS began misbehaving. We don't examine customer data
> unless given permission to do so, and even then this is often too
> time-consuming to undertake on an unpaid basis. I would consider the
> above an example of the maximum amount of detail we would go into.
> No identifying information regarding the affected
customer would be
> shared. We already share non-identifying information similar to the
> above to peers within the industry to aid deterrence and detection
> of future abuses.
> Would this sort of posting be welcomed or would it
be unwelcome
> noise? If the consensus is that it would be unwelcome noise then I
> may create a new list specifically for it, but I would rather not do
> so as then that is just another list that we have to raise awareness
> of.
> Please also note that those with an extremely low
tolerance for
> email noise may wish to quit this list and instead join the
> "announce" list, as it contains only announcements from BitFolk with
> no customer discussion whatsoever:
>
https://lists.bitfolk.com/mailman/listinfo/announce
> http://lists.bitfolk.com/lurker/list/announce.html
> (just 19 threads this year)
> Thoughts?
> Cheers,
> Andy
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
9:58 a.m.
On 07/12/12 09:57, Tony Andersson wrote:
Another vote from me. Sounds good.
In the unlikely event that it gets too much trafficwise (which I
strongly doubt) then just setup a new list.
Cheers,
--
Matthew Moore
Surgical Materials Testing Laboratory
System Administrator
Telephone: +44 (0)1656 752165
Email: matt(a)smtl.co.uk
> So I was contemplating posting an email
thread to this ("users")
> list every time we become aware of a customer compromise, and I was
> wondering what you thought of that idea.
10:21 a.m.
I too think that this is a great idea, however, it might be easier to create a separate
security@ mailing list now (and automatically?) subscribe users rather than posting
everything to the users@ mailing list; even if there is little traffic right now, this may
increase/change and it'd be easier to grow the service if its segregated from the
start.
I also can't remember what the new customer signup form for Bitfolk is like, but
recently filled in a form (from another organisation) with a single checkbox labelled
"…We like to keep you informed about services, campaigns, events, publications and
new initiatives…" - obviously more granular control is better for the user and I
think that given the option most users will opt-in (or can be auto-enrolled in accordance
with terms & conditions), knowing full-well that they actually want what they're
signing-up for rather than facing the daunting single tick box which gives the user
"all or nothing".
On 7 Dec 2012, at 02:19, Andy Smith <andy(a)bitfolk.com> wrote:
So I was contemplating posting an email thread to this
("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
1:09 p.m.
On 2012-12-07 10:21, Richard Green wrote:
I too think that this is a great idea, however, it
might be easier to
create a separate security@ mailing list now (and automatically?)
subscribe users rather than posting everything to the users@ mailing
list; even if there is little traffic right now, this may
increase/change and it'd be easier to grow the service if its
segregated from the start.
I feel inclined to agree with the idea of having a seperate "security"
or "advisory" list that could be used for this purpose, although at the
end of the day the real important thing is that we get the information
in the first place. So whatever the implementation details turns out to
be, I heartily supports it.
Regards,
Jan Henkins
1:09 p.m.
I am ok with that kind of information.
Cheers
Sämi
2012/12/7 Andy Smith <andy(a)bitfolk.com>
Hello,
From time to time BitFolk customer VPSes occasionally become subject
to various kinds of compromise. Frustratingly, the kinds of
compromise encountered are generally the result of run of the mill,
completely preventable and unremarkable root causes.
I would like to find a way to raise awareness of these very simple
security concerns amongst the customer base, in order to hopefully
cut down on how often they happen.
I was thinking that if customers saw how often these things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
So I was contemplating posting an email thread to this ("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
It might look something like this:
Today at around 04:30 we became aware of a customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may only become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
If the customer is unable/unwilling to do this then we may never
know why their VPS began misbehaving. We don't examine customer data
unless given permission to do so, and even then this is often too
time-consuming to undertake on an unpaid basis. I would consider the
above an example of the maximum amount of detail we would go into.
No identifying information regarding the affected customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Would this sort of posting be welcomed or would it be unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
Please also note that those with an extremely low tolerance for
email noise may wish to quit this list and instead join the
"announce" list, as it contains only announcements from BitFolk with
no customer discussion whatsoever:
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Thoughts?
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
--
Samuel Bächler
Obere Bläsistrasse 1
8049 Zürich
Web: boeser.ch
Tel: +41(0)43 817 46 28
Mob: +41(0)79 478 49 42
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEAREDAAYFAlDBUj4ACgkQIJm2TL8VSQsqvACgwIgInU6KIOtadzOhGfxJbzq2
IMwAoKpBPCQW2HYD1Dgs6RPF38QNycai
=xqsl
-----END PGP SIGNATURE-----
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
2:22 p.m.
Andy,
I think this is a good idea and am happy with it.
On 7 December 2012 02:19, Andy Smith <andy(a)bitfolk.com> wrote:
Hello,
From time to time BitFolk customer VPSes occasionally become subject
to various kinds of compromise. Frustratingly, the kinds of
compromise encountered are generally the result of run of the mill,
completely preventable and unremarkable root causes.
--
Alastair Sherringham
http://www.sherringham.net
2:24 p.m.
On 7/12/2012 4:19 πμ, Andy Smith wrote:
I was thinking that if customers saw how often these
things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
So I was contemplating posting an email thread to this ("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
Completely agree with this idea.
I do not wish to follow a different mailing list. I also believe
educating users a responsibility, especially considering the product
bitfolk sells.
In fact, I grant permission to Andy to publish anonymous breakins into
any of my machines right now.
--
Με εκτίμηση,
Γ. Μηλιώτης
Elementality Υπηρεσίες Πληροφορικής
corfiot(a)elementality.biz
8 Dec
8 Dec
9:36 p.m.
On Fri, Dec 07, 2012 at 02:19:42AM +0000, Andy Smith wrote:
Hello,
From time to time BitFolk customer VPSes occasionally become subject
to various kinds of compromise. Frustratingly, the kinds of
compromise encountered are generally the result of run of the mill,
completely preventable and unremarkable root causes.
As someone who worked in the web hosting industry for 5 years, I feel
your pain, sir.
I would like to find a way to raise awareness of these
very simple
security concerns amongst the customer base, in order to hopefully
cut down on how often they happen.
I was thinking that if customers saw how often these things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
So I was contemplating posting an email thread to this ("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
I would say not so much every time there's a compromise, but maybe every
time there's a compromise and you or the customer was able to do a full
root cause analysis.
Just seeing "this box got rooted and sent out tons of spam" isn't all
that useful, as we're all aware (or, should be, maybe I'm arguing
against myself here) that machines get exploited all the time and do Bad
Things, but seeing WHY, and maybe even some of the process for tracing
down the root cause would be handy.
At any rate, I'm a mailing list whore, so I would read every post
anyways, bring it on :)
Additionally, I must say that I'm very happy I chose bitfolk for my vps
provider. Things like this (and even the discussion itself) make me feel
very good about the people running things behind the scenes.
Also, it's always fun to find a fellow mutt user in the wild ;)
-Jeremy
9:39 p.m.
On Sat, Dec 08, 2012 at 01:36:08PM -0800, Jeremy Kitchen wrote:
[snip]
Also, it's always fun to find a fellow mutt user
in the wild ;)
Wild? I was *livid*! [1]
Hugo (also a mutt user).
[1] https://www.youtube.com/watch?v=beCYGm1vMJ0
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- We demand rigidly defined areas of doubt and uncertainty! ---
10 Dec
10 Dec
1 p.m.
** Hugo Mills <hugo-bf(a)carfax.org.uk> [2012-12-08 21:40]:
On Sat, Dec 08, 2012 at 01:36:08PM -0800, Jeremy
Kitchen wrote:
[snip]
** end quote [Hugo Mills]
I didn't realise we were that endangered as to be only found in captivity!
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
======================================================================
Registered in England | Company No: 4905028 | Registered Office:
Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU
Also, it's always fun to find a fellow mutt
user in the wild ;)
Wild? I was *livid*! [1]
Hugo (also a mutt user).
[1] https://www.youtube.com/watch?v=beCYGm1vMJ0
9:30 p.m.
New subject: mutt users! (Was: Re: Proposal: Security incidents postings)
On Mon, Dec 10, 2012 at 01:00:28PM +0000, Paul Tansom wrote:
** Hugo Mills <hugo-bf(a)carfax.org.uk>
[2012-12-08 21:40]:
Given that I haven't come across another mutt user at work in the past
10 years and I work in very tech/unix-heavy positions, we are pretty
rare.
Then again, I *just* started using it a few months ago after wanting to
use it for years and just not sitting down and putting the time into it.
It's definitely the ultimate hacker's email client.
-Jeremy
On Sat, Dec 08, 2012 at 01:36:08PM -0800, Jeremy
Kitchen wrote:
[snip]
** end quote [Hugo Mills]
I didn't realise we were that endangered as to be only found in captivity! Also, it's always fun to find a fellow mutt
user in the wild ;)
Wild? I was *livid*! [1]
Hugo (also a mutt user).
[1] https://www.youtube.com/watch?v=beCYGm1vMJ0 
11 Dec
11 Dec
12:57 a.m.
New subject: mutt users! (Was: Re: Proposal: Security incidents postings)
On Mon, Dec 10, 2012 at 9:30 PM, Jeremy Kitchen <kitchen(a)kitchen.io> wrote:
On Mon, Dec 10, 2012 at 01:00:28PM +0000, Paul Tansom
wrote:
In some departments of my company the non-mutt users are the rare
ones; I bet we have at least 50 mutt users in total, maybe even
100 ... but I guess SUSE is not your average company ;-)
** Hugo Mills <hugo-bf(a)carfax.org.uk>
[2012-12-08 21:40]:
Given that I haven't come across another mutt user at work in the past
10 years and I work in very tech/unix-heavy positions, we are pretty
rare. On Sat, Dec 08, 2012 at 01:36:08PM -0800, Jeremy
Kitchen wrote:
[snip]
** end quote [Hugo Mills]
I didn't realise we were that endangered as to be only found in captivity! Also, it's always fun to find a fellow mutt
user in the wild ;)
Wild? I was *livid*! [1]
Hugo (also a mutt user).
[1] https://www.youtube.com/watch?v=beCYGm1vMJ0 Then again, I *just* started using it a few months ago
after wanting to
use it for years and just not sitting down and putting the time into it.
Yikes, just calculated I've been using it for around 15 years.
It's definitely the ultimate hacker's email
client.
Definitely; I have a fairly elaborate mutt setup ;-)
$ find ~/.mutt | wc -l
116
If that makes you go WTF then you can see the non-confidential parts
here:
https://github.com/aspiers/mutt/tree/master/.mutt
https://github.com/aspiers/mutt.pub/tree/master/.mutt
12 Dec
12 Dec
5:20 p.m.
New subject: mutt users! (Was: Re: Proposal: Security incidents postings)
On 2012/12/10 11:30 PM, Jeremy Kitchen wrote:
Given that I haven't come across another mutt user
at work in the past
10 years and I work in very tech/unix-heavy positions, we are pretty
rare.
Then again, I *just* started using it a few months ago after wanting to
use it for years and just not sitting down and putting the time into it.
It's definitely the ultimate hacker's email client.
I used mutt via an ssh tunnel for personal mail for about 10 years.
Nowadays I still use it on various servers to read mail, especially when
I'm away from my mac (it's all imap). It's much faster on a slow
connection, e.g 3G on a tablet/cellphone, as you don't have to download
all the crud (even if just the headers) to read 1 important message.
4:53 p.m.
Preamble: Glad to see that this proposal has been adopted.
More below.
On 08/12/12 21:36, Jeremy Kitchen wrote:
Just seeing "this box got rooted and sent out
tons of spam" isn't
all that useful, as we're all aware (or, should be, maybe I'm
arguing against myself here) that machines get exploited all the time
and do Bad Things, but seeing WHY, and maybe even some of the process
for tracing down the root cause would be handy.
Yes, but the "this box got rooted" post might still be a useful
reminder to Pay Attention, and might provoke useful discussion
on how best to avoid these things.
One other thing might be useful - knowing the "population size",
if that is possible.
Additionally, I must say that I'm very happy I
chose bitfolk for my
vps provider. Things like this (and even the discussion itself) make
me feel very good about the people running things behind the scenes.
+1 to that.
5:09 p.m.
Hi,
On Wed, Dec 12, 2012 at 04:53:23PM +0000, Dom Latter wrote:
On 08/12/12 21:36, Jeremy Kitchen wrote:
Yeah that is sort of what I was thinking.
I feel like people need regular reminders, and there will be new
customers too, so a single historical posting or a wiki page that is
never looked at won't do it.
I was thinking that seeing that an actual customer got compromised
and that the threat is not hypothetical might provide a useful shock
to the system as it were..
I don't want to annoy people however.
Just seeing "this box got rooted and sent
out tons of spam" isn't
all that useful, as we're all aware (or, should be, maybe I'm
arguing against myself here) that machines get exploited all the time
and do Bad Things, but seeing WHY, and maybe even some of the process
for tracing down the root cause would be handy.
Yes, but the "this box got rooted" post might still be a useful
reminder to Pay Attention, and might provoke useful discussion
on how best to avoid these things. One other thing might be useful - knowing the
"population size",
if that is possible.
There are currently 422 active customer VPSes if that is what you
mean.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
9 Dec
9 Dec
2:46 a.m.
+1
Good idea.
On 06/12/12 20:19, Andy Smith wrote:
Hello,
From time to time BitFolk customer VPSes occasionally become subject
to various kinds of compromise. Frustratingly, the kinds of
compromise encountered are generally the result of run of the mill,
completely preventable and unremarkable root causes.
I would like to find a way to raise awareness of these very simple
security concerns amongst the customer base, in order to hopefully
cut down on how often they happen.
I was thinking that if customers saw how often these things happen
to people very much like themselves then it might help remove some
of the "yeah I've heard of that but it will never happen to me"
mindset that we all regrettably can fall into.
So I was contemplating posting an email thread to this ("users")
list every time we become aware of a customer compromise, and I was
wondering what you thought of that idea.
It might look something like this:
Today at around 04:30 we became aware of a customer VPS
initiating an abnormal amount of outbound SSH connections (~200
per second). The VPS's network access was suspended and customer
contacted.
It was later determined that a user account on the VPS had been
accessed starting 3 days ago, via an SSH dictionary attack. The
attacker installed another copy of the SSH dictionary attack
software and set it going. We do not believe that root access
was obtained.
The amount of detail would vary because we may only become aware of
a compromise when the customer's VPS itself starts perpetrating
abusive activity, and then we rely on the customer to investigate
why that is.
If the customer is unable/unwilling to do this then we may never
know why their VPS began misbehaving. We don't examine customer data
unless given permission to do so, and even then this is often too
time-consuming to undertake on an unpaid basis. I would consider the
above an example of the maximum amount of detail we would go into.
No identifying information regarding the affected customer would be
shared. We already share non-identifying information similar to the
above to peers within the industry to aid deterrence and detection
of future abuses.
Would this sort of posting be welcomed or would it be unwelcome
noise? If the consensus is that it would be unwelcome noise then I
may create a new list specifically for it, but I would rather not do
so as then that is just another list that we have to raise awareness
of.
Please also note that those with an extremely low tolerance for
email noise may wish to quit this list and instead join the
"announce" list, as it contains only announcements from BitFolk with
no customer discussion whatsoever:
https://lists.bitfolk.com/mailman/listinfo/announce
http://lists.bitfolk.com/lurker/list/announce.html
(just 19 threads this year)
Thoughts?
Cheers,
Andy
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
4152
days inactive
4158
days old
30 comments
25 participants
participants (25)
-
Adam Spiers -
Alastair Sherringham -
Andy Bennett -
Andy Smith -
Brendan -
Dom Latter -
Duggie -
G. Miliotis -
Geoffrey Teale -
Hugo Mills -
jan＠henkins.za.net -
Jeremy Kitchen -
Keith Williams -
Matthew Moore -
Michael Stevens -
Mike Zanker -
Nigel Rantor -
Ole-Morten Duesund -
Paul Tansom -
Peet Grobler -
Phil Stewart -
Richard Green -
Ross Younger -
Samuel Bächler -
Tony Andersson