31 Dec
2013
31 Dec
'13
2:11 p.m.
Tuesday, December 31, 2013, 1:00:07 PM, Andy wrote:
On Tue, Dec 31, 2013 at 12:11:34AM +0000, Tony
Andersson wrote:
> Have a strange attack happening to one of my domains, on the web
> server. It is a small privatish phpBB forum with nothing exciting,
> interesting or valuable going on at all. And it is the only one
> attacked out of a handful web sites on the server.
When you say "attack" can you give more
information as to what the
negative effects are?
e.g. is it killing your web server, or is it just
polluting your
logs?
It is just polluting the logs now. Before it added extra workload on
the machine. MySQL and Apache took the system load average up to 0.8 -
0.9 due to the amount of requests (and probably because phpBB isn't
written with performance in mind). After filtering out the "attacking"
connection (by stopping the POST requests to '/') this is no longer an
issue. Load average is now around 0.10, which is more than OK.
Logs are rotated, so I am not concerned about that.
> The site has had a lot of incorrect requests to
the server since
> before Christmas. I get POST requests in the region of two per second.
> There's noting in the post request and it is to the root of the
> domain. Like this:
When you say there's noting (nothing?) in the POST
request do you
mean that you have looked at it in tcpdump etc and there is no POST
data?
No, have not done any deeper analysis. Only looked at it from the web
servers point of view since the requests are accepted as normal POST
requests and no system or security log is mentioning them at all as
being malformed.
Good question though! The request is between 40 and 60 kB in size, so
it contains something. I would guess it is similar to what is seen in
the postfix log, with a lot of garbage that is not recognised as
proper POST data. What I am really saying I could not see anything as
name value pairs from inside PHP in the POST request. Nothing else.
Although all the requests you have shown have 301
return code which
is presumably not being followed up by the client. I am not sure if
Apache shows the request size (the "-" after the 301) when the
return code is 301, or if these requests are really empty.
> 184.57.181.141 - - [30/Dec/2013:23:32:24 +0000]
"POST / HTTP/1.1" 301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1)"
> 108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Before I set the site up to return a 301 the request size was shown as
being between 40 and 60 kB roughly.
My first guess on seeing logs like this would be that
it's just a
botnet probing for signatures of apps that are vulnerable to a known
exploit.
That was my first thought too. But since it has been ongoing since
just before Christmas with multiple requests per second from so many
different IP addresses I am starting to worry a bit. The occasional
bot scan I am not so concerned about.
> I have set up a filter in fail2ban for anyone
POSTing to '/' so they
> should be completely banned (using action 'iptables-allports'). But
> due to the sheer amount of different addresses attacking it seems to
> have little effect. Plus the fact I quite often see this in the
> fail2ban log:
> 2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
Another thing you could try if Fail2Ban isn't
doing it for you is to
block all POST requests to a given URI using mod_security2 (which is
available for Debian).
I haven't tested this but it should be something
like:
<IfModule mod_security2.c>
SecRule REQUEST_METHOD "^POST$"
"phase:1,nolog,chain,t:none,id:100"
SecRule REQUEST_URI "^/$" "drop"
</IfModule>
I think that will drop (immediately close TCP
connection) on any
POST request to /.
That sounds like a very good idea! Will look in to that this
afternoon.
I did think it should be possible with core Apache
config but I
don't think that:
<Limit POST>
</Limit>
can be combined with a Location block.
Probably not doable, especially since I have to allow post to sub directories
and files in the root directory that are "attacked".
If you didn't want to install mod_security2 then
you could probably
achieve similar with mod_rewrite:
RewriteCond %{REQUEST_METHOD} POST
RewriteRule ^/$ - [forbidden]
Which would be not much different from my 301 now.
Thing is, if you're sure the POSTs aren't
doing anything harmful
then blocking them like this maybe isn't buying you very much. It
may allow Apache to free up the request quicker if you're really in
a denial of service situation.
True. I just have to stop these requests from being processed as early
as possible, to both limit the workload on the machine and also to
block any potentially harmful requests.
> So it seems that despite being banned they can
still send a request to
> the Apache server? Not sure why, the iptables -L seems to list an
> awful lot of IP addresses and domain names. So the fail2ban filter is
> working as it should with setting up rules in iptables.
Is it possible they are concurrent requests? iptables
probably
allows an ongoing TCP conversation due to allowing "related and
established" traffic.
Ah! Yes! That could well be! Have turned off http keep-alive, but it
could be concurrent or persistent on a lower level...
> At the same time, postfix is getting a large
amount of requests on
> port 25 too:
Are you quite sure this is related? Did it only start
at the same
time as the Apache POST request flood?
I am ashamed to say I do not know. Checking the mail logs, it looks
like it started some time after 11am on the 22/12. Checking the access
logs for Apache I see that type of "attack" I have described above,
started on the 27th. Which is when I first patched a hole in the user
registration module (after upgrading phpBB to the latest version my
settings were messed up so the user registration module was open for
anyone without any verification.) And now the bigger picture is
surfacing here. By turning off all new member registration, the bots /
attackers / bastards are being redirected directly to '/' when they do
their direct POST request to the from the registration module.
Apache's access log is not showing this internal redirect for reasons
still uninvestigated. But now it makes sense! That is why there's
nothing in the POST request. It does not make any sense why they are
still persisting though. Probably because they are not intelligent
enough to check for anything more complicated than a successful http
request, which it kind of looks like to them. A redirect is expected
when posting things like a new member registration, if I think about
it. So I have to just close the connection on the tcp level to show
they are not successful. Or completely block any incoming connection.
Which is what fail2ban is doing. Problem there seems to be the sheer
amount of machines doing this, with different IP addresses.
> Now I am worried all this will consume up my
bandwidth allowance (as
> well as eating into system resources of course),
If I am looking at the right VPS's bandwidth
graphs, your traffic
usage has indeed increased from almost nothing before Friday, but it
has only increased to about 16kbps out and 24kbps in, so you don't
need to worry about going through your data transfer allowance.
The vps is 'ta2' and the numbers looks reasonable. I will stop worry
about the bandwidth usage then! And start worrying about what fireworks
to get for tonight instead. ;-)
(After I have implemented the mod_security2 block of course)
Many thanks,
__
/ony