31 Dec
2013
31 Dec
'13
12:11 a.m.
Hi all,
Have a strange attack happening to one of my domains, on the web
server. It is a small privatish phpBB forum with nothing exciting,
interesting or valuable going on at all. And it is the only one
attacked out of a handful web sites on the server.
The site has had a lot of incorrect requests to the server since
before Christmas. I get POST requests in the region of two per second.
There's noting in the post request and it is to the root of the
domain. Like this:
184.57.181.141 - - [30/Dec/2013:23:32:24 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
68.118.233.245 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
173.51.226.12 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
71.10.0.254 - - [30/Dec/2013:23:32:27 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
75.91.250.137 - - [30/Dec/2013:23:32:29 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.200.239.239 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
64.40.129.122 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
128.210.19.134 - - [30/Dec/2013:23:32:32 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
12.51.89.194 - - [30/Dec/2013:23:32:33 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
184.57.181.141 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
208.96.191.152 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
The 301 response is something I set up when I discovered this. There
should be no POST requests to /, so I do a 301 permanent redirect back
to the client's own IP address. But that seems to have had no effect
at all. The requests are still constantly coming in.
I have set up a filter in fail2ban for anyone POSTing to '/' so they
should be completely banned (using action 'iptables-allports'). But
due to the sheer amount of different addresses attacking it seems to
have little effect. Plus the fact I quite often see this in the
fail2ban log:
2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
So it seems that despite being banned they can still send a request to
the Apache server? Not sure why, the iptables -L seems to list an
awful lot of IP addresses and domain names. So the fail2ban filter is
working as it should with setting up rules in iptables.
At the same time, postfix is getting a large amount of requests on
port 25 too:
Dec 30 23:54:45 bitfolk postfix/smtpd[14601]: connect from unknown[180.67.178.14]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: lost connection after UNKNOWN from
unknown[76.2.133.225]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: disconnect from unknown[76.2.133.225]
Dec 30 23:54:48 bitfolk postfix/smtpd[27968]: connect from unknown[24.151.82.226]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: lost connection after UNKNOWN from
unknown[173.220.57.214]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: disconnect from unknown[173.220.57.214]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: lost connection after UNKNOWN from
unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: disconnect from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: lost connection after UNKNOWN from
unknown[173.246.215.147]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: disconnect from unknown[173.246.215.147]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: lost connection after UNKNOWN from
unknown[180.67.178.14]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: disconnect from unknown[180.67.178.14]
And in the mail.warn log:
Dec 30 23:10:15 bitfolk postfix/smtpd[19391]: warning: non-SMTP command from
unknown[96.38.26.186]: UY:l??????????z??????\?
Dec 30 23:11:22 bitfolk postfix/smtpd[17880]: warning: non-SMTP command from
unknown[181.67.172.79]: U:??[6?
Dec 30 23:14:46 bitfolk postfix/smtpd[19522]: warning: non-SMTP command from
unknown[24.39.251.34]: @:??>T^R^?d???&U?V<??;W?p4?Gf#???t????,???E?
Dec 30 23:16:57 bitfolk postfix/smtpd[24688]: warning: non-SMTP command from
unknown[72.181.54.101]: gu:?R?M????
I can only conclude this is sent to the same domain name as is
attacked on port 80...
Now I am worried all this will consume up my bandwidth allowance (as
well as eating into system resources of course), and I have run out of ideas how
to stop this. Any suggestions are most welcome!
Thanks,
__
/ony
31 Dec
31 Dec
12:18 a.m.
Realised the second after I pressed the send button that the answer to
the ban issue is because those attacks are on ip v6
root@bitfolk:/etc/fail2ban# netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 1 85.119.82.79:80 121.168.45.218:1446 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 24.186.158.213:61301 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 67.180.245.251:17277 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 71.218.243.152:25311 FIN_WAIT1
Now, I have to figure out how to turn IP v6 off on the vps then...
__
/ony
-------
Tuesday, December 31, 2013, 12:11:34 AM, Tony wrote:
Hi all,
Have a strange attack happening to one of my domains,
on the web
server. It is a small privatish phpBB forum with nothing exciting,
interesting or valuable going on at all. And it is the only one
attacked out of a handful web sites on the server.
The site has had a lot of incorrect requests to the
server since
before Christmas. I get POST requests in the region of two per second.
There's noting in the post request and it is to the root of the
domain. Like this:
184.57.181.141 - - [30/Dec/2013:23:32:24 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
68.118.233.245 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
173.51.226.12 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
71.10.0.254 - - [30/Dec/2013:23:32:27 +0000] "POST / HTTP/1.1" 301
- "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
75.91.250.137 - - [30/Dec/2013:23:32:29 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.200.239.239 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
64.40.129.122 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
128.210.19.134 - - [30/Dec/2013:23:32:32 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
12.51.89.194 - - [30/Dec/2013:23:32:33 +0000] "POST / HTTP/1.1" 301
- "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
184.57.181.141 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
208.96.191.152 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
The 301 response is something I set up when I
discovered this. There
should be no POST requests to /, so I do a 301 permanent redirect back
to the client's own IP address. But that seems to have had no effect
at all. The requests are still constantly coming in.
I have set up a filter in fail2ban for anyone POSTing
to '/' so they
should be completely banned (using action 'iptables-allports'). But
due to the sheer amount of different addresses attacking it seems to
have little effect. Plus the fact I quite often see this in the
fail2ban log:
2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
So it seems that despite being banned they can still
send a request to
the Apache server? Not sure why, the iptables -L seems to list an
awful lot of IP addresses and domain names. So the fail2ban filter is
working as it should with setting up rules in iptables.
At the same time, postfix is getting a large amount of
requests on
port 25 too:
Dec 30 23:54:45 bitfolk postfix/smtpd[14601]: connect
from unknown[180.67.178.14]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: lost connection after
UNKNOWN from unknown[76.2.133.225]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: disconnect from unknown[76.2.133.225]
Dec 30 23:54:48 bitfolk postfix/smtpd[27968]: connect from unknown[24.151.82.226]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: lost connection after
UNKNOWN from unknown[173.220.57.214]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: disconnect from unknown[173.220.57.214]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: lost connection after
UNKNOWN from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: disconnect from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: lost connection after
UNKNOWN from unknown[173.246.215.147]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: disconnect from unknown[173.246.215.147]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: lost connection after
UNKNOWN from unknown[180.67.178.14]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: disconnect from unknown[180.67.178.14]
And in the mail.warn log:
Dec 30 23:10:15 bitfolk postfix/smtpd[19391]: warning:
non-SMTP
command from unknown[96.38.26.186]: UY:l??????????z??????\?
Dec 30 23:11:22 bitfolk postfix/smtpd[17880]: warning: non-SMTP
command from unknown[181.67.172.79]: U:??[6?
Dec 30 23:14:46 bitfolk postfix/smtpd[19522]: warning: non-SMTP
command from unknown[24.39.251.34]:
@:??>T^R^?d???&U?V<??;W?p4?Gf#???t????,???E?
Dec 30 23:16:57 bitfolk postfix/smtpd[24688]: warning: non-SMTP
command from unknown[72.181.54.101]: gu:?R?M????
I can only conclude this is sent to the same domain
name as is
attacked on port 80...
Now I am worried all this will consume up my bandwidth
allowance (as
well as eating into system resources of course), and I have run out of ideas how
to stop this. Any suggestions are most welcome!
Thanks,
__
/ony
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
12:34 a.m.
% cat >> /etc/hosts.deny
ALL: ALL except your.home.ip.here
might help, as might this:
% cat /var/log/apache2/access.log | awk '{printf "echo 'ALL:
%s'>>/etc/hosts.deny\n",$1}' | /bin/sh
this assumes you have installed tcpd
cheers
Le Mardi 31 décembre 2013 1h19, Tony Andersson <BitFolkList(a)tony-andersson.com> a
écrit :
Realised the second after I pressed the send button that the answer to
the ban issue is because those attacks are on ip v6
root@bitfolk:/etc/fail2ban# netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 1 85.119.82.79:80 121.168.45.218:1446 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 24.186.158.213:61301 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 67.180.245.251:17277 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 71.218.243.152:25311 FIN_WAIT1
Now, I have to figure out how to turn IP v6 off on the vps then...
__
/ony
-------
Tuesday, December 31, 2013, 12:11:34 AM, Tony wrote:
Hi all,
Have a strange attack happening to one of my domains,
on the web
server. It is a small privatish phpBB forum with nothing exciting,
interesting or valuable going on at all. And it is the only one
attacked out of a handful web sites on the server.
The site has had a lot of incorrect requests to the
server since
before Christmas. I get POST requests in the region of two per second.
There's noting in the post request and it is to the root of the
domain. Like this:
184.57.181.141 - - [30/Dec/2013:23:32:24 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
68.118.233.245 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
173.51.226.12 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
71.10.0.254 - - [30/Dec/2013:23:32:27 +0000] "POST / HTTP/1.1" 301
- "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
75.91.250.137 - - [30/Dec/2013:23:32:29 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.200.239.239 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
64.40.129.122 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
128.210.19.134 - - [30/Dec/2013:23:32:32 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
12.51.89.194 - - [30/Dec/2013:23:32:33 +0000] "POST / HTTP/1.1" 301
- "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
184.57.181.141 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
208.96.191.152 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
The 301 response is something I set up when I
discovered this. There
should be no POST requests to /, so I do a 301 permanent redirect back
to the client's own IP address. But that seems to have had no effect
at all. The requests are still constantly coming in.
I have set up a filter in fail2ban for anyone POSTing
to '/' so they
should be completely banned (using action 'iptables-allports'). But
due to the sheer amount of different addresses attacking it seems to
have little effect. Plus the fact I quite often see this in the
fail2ban log:
2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
So it seems that despite being banned they can still
send a request to
the Apache server? Not sure why, the iptables -L seems to list an
awful lot of IP addresses and domain names. So the fail2ban filter is
working as it should with setting up rules in iptables.
At the same time, postfix is getting a large amount of
requests on
port 25 too:
Dec 30 23:54:45 bitfolk postfix/smtpd[14601]: connect
from unknown[180.67.178.14]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: lost connection after
UNKNOWN from unknown[76.2.133.225]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: disconnect from unknown[76.2.133.225]
Dec 30 23:54:48 bitfolk postfix/smtpd[27968]: connect from unknown[24.151.82.226]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: lost connection after
UNKNOWN from unknown[173.220.57.214]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: disconnect from unknown[173.220.57.214]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: lost connection after
UNKNOWN from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: disconnect from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: lost connection after
UNKNOWN from unknown[173.246.215.147]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: disconnect from unknown[173.246.215.147]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: lost connection after
UNKNOWN from unknown[180.67.178.14]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: disconnect from unknown[180.67.178.14]
And in the mail.warn log:
Dec 30 23:10:15 bitfolk postfix/smtpd[19391]: warning:
non-SMTP
command from unknown[96.38.26.186]: UY:l??????????z??????\?
Dec 30 23:11:22 bitfolk postfix/smtpd[17880]: warning: non-SMTP
command from unknown[181.67.172.79]: U:??[6?
Dec 30 23:14:46 bitfolk postfix/smtpd[19522]: warning: non-SMTP
command from unknown[24.39.251.34]:
@:??>T^R^?d???&U?V<??;W?p4?Gf#???t????,???E?
Dec 30 23:16:57 bitfolk postfix/smtpd[24688]: warning: non-SMTP
command from unknown[72.181.54.101]: gu:?R?M????
I can only conclude this is sent to the same domain
name as is
attacked on port 80...
Now I am worried all this will consume up my bandwidth
allowance (as
well as eating into system resources of course), and I have run out of ideas how
to stop this. Any suggestions are most welcome!
Thanks,
__
/ony
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
2:32 a.m.
Hi Tony,
Well, a DDoS solution it is. :)
When they realise that they have no effect they are likely to move on. And you can
gradually open your server up as need be, once you have established which machines are
culpable.
Consider this the rough and ready farmer's method.
It's imperfect I know but it should work until you get to a garage.
Cheers,
Le Mardi 31 décembre 2013 2h06, Tony Andersson <BitFolkList(a)tony-andersson.com> a
écrit :
Re: [bitfolk] Strange DDOS attack?
Hi Max,
Thanks for the idea, but problem then would be that no-one can connect to any web site on
the server, nor can anyone actually send legit e-mails to the domain.
I could just turn the vps off completely instead in that case... :-(
have turned off IP v6, and the attacks on port 25 seems to have eased off, but on port 80
they are still ongoing.
Cheers,
__
/ony
-------
Tuesday, December 31, 2013, 12:34:58 AM, Max wrote:
% cat >> /etc/hosts.deny
ALL: ALL except your.home.ip.here
might help, as might this:
% cat /var/log/apache2/access.log | awk '{printf "echo 'ALL:
%s'>>/etc/hosts.deny\n",$1}' | /bin/sh
this assumes you have installed tcpd
cheers
Le Mardi 31 décembre 2013 1h19, Tony Andersson <BitFolkList(a)tony-andersson.com> a
écrit :
Realised the second after I pressed the send button that the answer to
the ban issue is because those attacks are on ip v6
root@bitfolk:/etc/fail2ban# netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 1 85.119.82.79:80 121.168.45.218:1446 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 24.186.158.213:61301 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 67.180.245.251:17277 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 71.218.243.152:25311 FIN_WAIT1
Now, I have to figure out how to turn IP v6 off on the vps then...
__
/ony
-------
Tuesday, December 31, 2013, 12:11:34 AM, Tony wrote:
Hi all,
Have a strange attack happening to one of my domains,
on the web
server. It is a small privatish phpBB forum with nothing exciting,
interesting or valuable going on at all. And it is the only one
attacked out of a handful web sites on the server.
The site has had a lot of incorrect requests to the
server since
before Christmas. I get POST requests in the region of two per second.
There's noting in the post request and it is to the root of the
domain. Like this:
184.57.181.141 - - [30/Dec/2013:23:32:24 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
68.118.233.245 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
173.51.226.12 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
71.10.0.254 - - [30/Dec/2013:23:32:27 +0000] "POST / HTTP/1.1" 301
- "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
75.91.250.137 - - [30/Dec/2013:23:32:29 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.200.239.239 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
64.40.129.122 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
128.210.19.134 - - [30/Dec/2013:23:32:32 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
12.51.89.194 - - [30/Dec/2013:23:32:33 +0000] "POST / HTTP/1.1" 301
- "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
184.57.181.141 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
208.96.191.152 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1"
301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
The 301 response is something I set up when I
discovered this. There
should be no POST requests to /, so I do a 301 permanent redirect back
to the client's own IP address. But that seems to have had no effect
at all. The requests are still constantly coming in.
I have set up a filter in fail2ban for anyone POSTing
to '/' so they
should be completely banned (using action 'iptables-allports'). But
due to the sheer amount of different addresses attacking it seems to
have little effect. Plus the fact I quite often see this in the
fail2ban log:
2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
So it seems that despite being banned they can still
send a request to
the Apache server? Not sure why, the iptables -L seems to list an
awful lot of IP addresses and domain names. So the fail2ban filter is
working as it should with setting up rules in iptables.
At the same time, postfix is getting a large amount of
requests on
port 25 too:
Dec 30 23:54:45 bitfolk postfix/smtpd[14601]: connect
from unknown[180.67.178.14]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: lost connection after
UNKNOWN from unknown[76.2.133.225]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: disconnect from unknown[76.2.133.225]
Dec 30 23:54:48 bitfolk postfix/smtpd[27968]: connect from unknown[24.151.82.226]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: lost connection after
UNKNOWN from unknown[173.220.57.214]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: disconnect from unknown[173.220.57.214]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: lost connection after
UNKNOWN from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: disconnect from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: lost connection after
UNKNOWN from unknown[173.246.215.147]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: disconnect from unknown[173.246.215.147]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: lost connection after
UNKNOWN from unknown[180.67.178.14]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: disconnect from unknown[180.67.178.14]
And in the mail.warn log:
Dec 30 23:10:15 bitfolk postfix/smtpd[19391]: warning:
non-SMTP
command from unknown[96.38.26.186]: UY:l??????????z??????\?
Dec 30 23:11:22 bitfolk postfix/smtpd[17880]: warning: non-SMTP
command from unknown[181.67.172.79]: U:??[6?
Dec 30 23:14:46 bitfolk postfix/smtpd[19522]: warning: non-SMTP
command from unknown[24.39.251.34]:
@:??>T^R^?d???&U?V<??;W?p4?Gf#???t????,???E?
Dec 30 23:16:57 bitfolk postfix/smtpd[24688]: warning: non-SMTP
command from unknown[72.181.54.101]: gu:?R?M????
I can only conclude this is sent to the same domain
name as is
attacked on port 80...
Now I am worried all this will consume up my bandwidth
allowance (as
well as eating into system resources of course), and I have run out of ideas how
to stop this. Any suggestions are most welcome!
Thanks,
__
/ony
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
9:37 a.m.
On Tue, Dec 31, 2013 at 12:18 AM, Tony Andersson
<BitFolkList(a)tony-andersson.com> wrote:
Realised the second after I pressed the send button
that the answer to
the ban issue is because those attacks are on ip v6
root@bitfolk:/etc/fail2ban# netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 1 85.119.82.79:80 121.168.45.218:1446 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 24.186.158.213:61301 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 67.180.245.251:17277 FIN_WAIT1
tcp6 0 1 85.119.82.79:80 71.218.243.152:25311 FIN_WAIT1
Those are IPv4 addresses, they should be firewalled by iptables as usual.
I think this happens when a service binds a socket to both ipv4 and
ipv6: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=453272
Cheers,
Graham
9:55 a.m.
On Tue, Dec 31, 2013 at 12:11 AM, Tony Andersson
<BitFolkList(a)tony-andersson.com> wrote:
Hi all,
Have a strange attack happening to one of my domains, on the web
server. It is a small privatish phpBB forum with nothing exciting,
interesting or valuable going on at all. And it is the only one
attacked out of a handful web sites on the server.
The site has had a lot of incorrect requests to the server since
before Christmas. I get POST requests in the region of two per second.
There's noting in the post request and it is to the root of the
domain. Like this:
184.57.181.141 - - [30/Dec/2013:23:32:24 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
68.118.233.245 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
173.51.226.12 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
71.10.0.254 - - [30/Dec/2013:23:32:27 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
75.91.250.137 - - [30/Dec/2013:23:32:29 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
108.200.239.239 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
64.40.129.122 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
128.210.19.134 - - [30/Dec/2013:23:32:32 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
12.51.89.194 - - [30/Dec/2013:23:32:33 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
184.57.181.141 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
208.96.191.152 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Because it's only the phpBB site that's being accessed in this way,
it's probably something phpBB-specific. It could be a mobile client
that does something unusual (although it's a strange user-agent string
for something legit), or a phpBB worm / mass hack. Previous phpBB
worms have searched google for phpBB specific urls / signatures and
then exploited them. I suggest that you post on a phpBB specific list,
as you're more likely to find people who know what it is.
Two requests/second isn't really a very effective DOS attack against
most applications.
The 301 response is something I set up when I
discovered this. There
should be no POST requests to /, so I do a 301 permanent redirect back
to the client's own IP address. But that seems to have had no effect
at all. The requests are still constantly coming in.
I have set up a filter in fail2ban for anyone POSTing to '/' so they
should be completely banned (using action 'iptables-allports'). But
due to the sheer amount of different addresses attacking it seems to
have little effect. Plus the fact I quite often see this in the
fail2ban log:
2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
So it seems that despite being banned they can still send a request to
the Apache server? Not sure why, the iptables -L seems to list an
awful lot of IP addresses and domain names. So the fail2ban filter is
working as it should with setting up rules in iptables.
fail2ban works aynchronously; the ban rules only get applied after the
requests appear in the logfiles and fail2ban processes them. So it's
almost certainly possible for several requests to be made from an IP
before fail2ban has a chance to block the IP for the first time. It
should be simple to look in the logs for the timestamps of the
requests and the ban attempt to work out what the sequence of events
was.
At the same time, postfix is getting a large amount of
requests on
port 25 too:
Dec 30 23:54:45 bitfolk postfix/smtpd[14601]: connect from unknown[180.67.178.14]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: lost connection after UNKNOWN from
unknown[76.2.133.225]
Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: disconnect from unknown[76.2.133.225]
Dec 30 23:54:48 bitfolk postfix/smtpd[27968]: connect from unknown[24.151.82.226]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: lost connection after UNKNOWN from
unknown[173.220.57.214]
Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: disconnect from unknown[173.220.57.214]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: lost connection after UNKNOWN from
unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: disconnect from unknown[72.135.3.145]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: lost connection after UNKNOWN from
unknown[173.246.215.147]
Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: disconnect from unknown[173.246.215.147]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: lost connection after UNKNOWN from
unknown[180.67.178.14]
Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: disconnect from unknown[180.67.178.14]
And in the mail.warn log:
Dec 30 23:10:15 bitfolk postfix/smtpd[19391]: warning: non-SMTP command from
unknown[96.38.26.186]: UY:l??????????z??????\?
Dec 30 23:11:22 bitfolk postfix/smtpd[17880]: warning: non-SMTP command from
unknown[181.67.172.79]: U:??[6?
Dec 30 23:14:46 bitfolk postfix/smtpd[19522]: warning: non-SMTP command from
unknown[24.39.251.34]: @:??>T^R^?d???&U?V<??;W?p4?Gf#???t????,???E?
Dec 30 23:16:57 bitfolk postfix/smtpd[24688]: warning: non-SMTP command from
unknown[72.181.54.101]: gu:?R?M????
I can only conclude this is sent to the same domain name as is
attacked on port 80...
Alternatively, it could just be the random scans and probes that
everyone who runs services on the Internet gets.
Now I am worried all this will consume up my bandwidth
allowance (as
well as eating into system resources of course), and I have run out of ideas how
to stop this. Any suggestions are most welcome!
The bandwidth consumption of these requests seems tiny. Since they've
been going since Christmas they'll have shown up in your weekly data
transfer reports by now, if they were going to cause you a problem.
Cheers,
Graham
11:47 a.m.
Hi Tony,
On Tue, December 31, 2013 12:18 am, Tony Andersson wrote:
Realised the second after I pressed the send button
that the answer to
the ban issue is because those attacks are on ip v6
[...]
tcp6 0 1 85.119.82.79:80
121.168.45.218:1446
They are actually IPv4 connections, as indicated by the address
structure/format. The 'tcp6' label is just how netstat illustrates a
service listening on both IPv4 and IPv6.
Mathew
11:55 a.m.
On Tue, Dec 31, 2013 at 12:34:58AM +0000, Max B wrote:
% cat >> /etc/hosts.deny
ALL: ALL except your.home.ip.here
Apache isn't usually built with tcpwrappers support, nor is any
other web server I'm aware of. You could do so but I think there are
better solutions depending on what you want to achieve.
If you wanted to block these hosts from polluting logs and are
already running Fail2Ban for that purpose for SSH connections, then
I would use Fail2Ban for that as well.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
1 p.m.
Hi Tony,
On Tue, Dec 31, 2013 at 12:11:34AM +0000, Tony Andersson wrote:
Have a strange attack happening to one of my domains,
on the web
server. It is a small privatish phpBB forum with nothing exciting,
interesting or valuable going on at all. And it is the only one
attacked out of a handful web sites on the server.
When you say "attack" can you give more information as to what the
negative effects are?
e.g. is it killing your web server, or is it just polluting your
logs?
The site has had a lot of incorrect requests to the
server since
before Christmas. I get POST requests in the region of two per second.
There's noting in the post request and it is to the root of the
domain. Like this:
When you say there's noting (nothing?) in the POST request do you
mean that you have looked at it in tcpdump etc and there is no POST
data?
Although all the requests you have shown have 301 return code which
is presumably not being followed up by the client. I am not sure if
Apache shows the request size (the "-" after the 301) when the
return code is 301, or if these requests are really empty.
184.57.181.141 - - [30/Dec/2013:23:32:24 +0000]
"POST / HTTP/1.1" 301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1)"
108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
My first guess on seeing logs like this would be that it's just a
botnet probing for signatures of apps that are vulnerable to a known
exploit.
I have set up a filter in fail2ban for anyone POSTing
to '/' so they
should be completely banned (using action 'iptables-allports'). But
due to the sheer amount of different addresses attacking it seems to
have little effect. Plus the fact I quite often see this in the
fail2ban log:
2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
Another thing you could try if Fail2Ban isn't doing it for you is to
block all POST requests to a given URI using mod_security2 (which is
available for Debian).
I haven't tested this but it should be something like:
<IfModule mod_security2.c>
SecRule REQUEST_METHOD "^POST$"
"phase:1,nolog,chain,t:none,id:100"
SecRule REQUEST_URI "^/$" "drop"
</IfModule>
I think that will drop (immediately close TCP connection) on any
POST request to /.
I did think it should be possible with core Apache config but I
don't think that:
<Limit POST>
</Limit>
can be combined with a Location block.
If you didn't want to install mod_security2 then you could probably
achieve similar with mod_rewrite:
RewriteCond %{REQUEST_METHOD} POST
RewriteRule ^/$ - [forbidden]
Thing is, if you're sure the POSTs aren't doing anything harmful
then blocking them like this maybe isn't buying you very much. It
may allow Apache to free up the request quicker if you're really in
a denial of service situation.
So it seems that despite being banned they can still
send a request to
the Apache server? Not sure why, the iptables -L seems to list an
awful lot of IP addresses and domain names. So the fail2ban filter is
working as it should with setting up rules in iptables.
Is it possible they are concurrent requests? iptables probably
allows an ongoing TCP conversation due to allowing "related and
established" traffic.
At the same time, postfix is getting a large amount of
requests on
port 25 too:
Are you quite sure this is related? Did it only start at the same
time as the Apache POST request flood?
Now I am worried all this will consume up my bandwidth
allowance (as
well as eating into system resources of course),
If I am looking at the right VPS's bandwidth graphs, your traffic
usage has indeed increased from almost nothing before Friday, but it
has only increased to about 16kbps out and 24kbps in, so you don't
need to worry about going through your data transfer allowance.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
1:19 p.m.
Aah! Didn't know that. But now when you say so, it makes sense. Have
no real experience with IPv6, so I thought it might be translated to
IP v4 addresses for some reason (and that causing the confusion for
fail2ban).
Thanks!
__
/ony
-------
Tuesday, December 31, 2013, 11:47:30 AM, Mathew wrote:
Hi Tony,
On Tue, December 31, 2013 12:18 am, Tony Andersson
wrote:
> Realised the second after I pressed the send
button that the answer to
> the ban issue is because those attacks are on ip v6
[...]
> tcp6 0 1 85.119.82.79:80
121.168.45.218:1446
They are actually IPv4 connections, as indicated by
the address
structure/format. The 'tcp6' label is just how netstat illustrates a
service listening on both IPv4 and IPv6.
Mathew
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
2:11 p.m.
Tuesday, December 31, 2013, 1:00:07 PM, Andy wrote:
On Tue, Dec 31, 2013 at 12:11:34AM +0000, Tony
Andersson wrote:
> Have a strange attack happening to one of my domains, on the web
> server. It is a small privatish phpBB forum with nothing exciting,
> interesting or valuable going on at all. And it is the only one
> attacked out of a handful web sites on the server.
When you say "attack" can you give more
information as to what the
negative effects are?
e.g. is it killing your web server, or is it just
polluting your
logs?
It is just polluting the logs now. Before it added extra workload on
the machine. MySQL and Apache took the system load average up to 0.8 -
0.9 due to the amount of requests (and probably because phpBB isn't
written with performance in mind). After filtering out the "attacking"
connection (by stopping the POST requests to '/') this is no longer an
issue. Load average is now around 0.10, which is more than OK.
Logs are rotated, so I am not concerned about that.
> The site has had a lot of incorrect requests to
the server since
> before Christmas. I get POST requests in the region of two per second.
> There's noting in the post request and it is to the root of the
> domain. Like this:
When you say there's noting (nothing?) in the POST
request do you
mean that you have looked at it in tcpdump etc and there is no POST
data?
No, have not done any deeper analysis. Only looked at it from the web
servers point of view since the requests are accepted as normal POST
requests and no system or security log is mentioning them at all as
being malformed.
Good question though! The request is between 40 and 60 kB in size, so
it contains something. I would guess it is similar to what is seen in
the postfix log, with a lot of garbage that is not recognised as
proper POST data. What I am really saying I could not see anything as
name value pairs from inside PHP in the POST request. Nothing else.
Although all the requests you have shown have 301
return code which
is presumably not being followed up by the client. I am not sure if
Apache shows the request size (the "-" after the 301) when the
return code is 301, or if these requests are really empty.
> 184.57.181.141 - - [30/Dec/2013:23:32:24 +0000]
"POST / HTTP/1.1" 301 - "-" "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1)"
> 108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Before I set the site up to return a 301 the request size was shown as
being between 40 and 60 kB roughly.
My first guess on seeing logs like this would be that
it's just a
botnet probing for signatures of apps that are vulnerable to a known
exploit.
That was my first thought too. But since it has been ongoing since
just before Christmas with multiple requests per second from so many
different IP addresses I am starting to worry a bit. The occasional
bot scan I am not so concerned about.
> I have set up a filter in fail2ban for anyone
POSTing to '/' so they
> should be completely banned (using action 'iptables-allports'). But
> due to the sheer amount of different addresses attacking it seems to
> have little effect. Plus the fact I quite often see this in the
> fail2ban log:
> 2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
Another thing you could try if Fail2Ban isn't
doing it for you is to
block all POST requests to a given URI using mod_security2 (which is
available for Debian).
I haven't tested this but it should be something
like:
<IfModule mod_security2.c>
SecRule REQUEST_METHOD "^POST$"
"phase:1,nolog,chain,t:none,id:100"
SecRule REQUEST_URI "^/$" "drop"
</IfModule>
I think that will drop (immediately close TCP
connection) on any
POST request to /.
That sounds like a very good idea! Will look in to that this
afternoon.
I did think it should be possible with core Apache
config but I
don't think that:
<Limit POST>
</Limit>
can be combined with a Location block.
Probably not doable, especially since I have to allow post to sub directories
and files in the root directory that are "attacked".
If you didn't want to install mod_security2 then
you could probably
achieve similar with mod_rewrite:
RewriteCond %{REQUEST_METHOD} POST
RewriteRule ^/$ - [forbidden]
Which would be not much different from my 301 now.
Thing is, if you're sure the POSTs aren't
doing anything harmful
then blocking them like this maybe isn't buying you very much. It
may allow Apache to free up the request quicker if you're really in
a denial of service situation.
True. I just have to stop these requests from being processed as early
as possible, to both limit the workload on the machine and also to
block any potentially harmful requests.
> So it seems that despite being banned they can
still send a request to
> the Apache server? Not sure why, the iptables -L seems to list an
> awful lot of IP addresses and domain names. So the fail2ban filter is
> working as it should with setting up rules in iptables.
Is it possible they are concurrent requests? iptables
probably
allows an ongoing TCP conversation due to allowing "related and
established" traffic.
Ah! Yes! That could well be! Have turned off http keep-alive, but it
could be concurrent or persistent on a lower level...
> At the same time, postfix is getting a large
amount of requests on
> port 25 too:
Are you quite sure this is related? Did it only start
at the same
time as the Apache POST request flood?
I am ashamed to say I do not know. Checking the mail logs, it looks
like it started some time after 11am on the 22/12. Checking the access
logs for Apache I see that type of "attack" I have described above,
started on the 27th. Which is when I first patched a hole in the user
registration module (after upgrading phpBB to the latest version my
settings were messed up so the user registration module was open for
anyone without any verification.) And now the bigger picture is
surfacing here. By turning off all new member registration, the bots /
attackers / bastards are being redirected directly to '/' when they do
their direct POST request to the from the registration module.
Apache's access log is not showing this internal redirect for reasons
still uninvestigated. But now it makes sense! That is why there's
nothing in the POST request. It does not make any sense why they are
still persisting though. Probably because they are not intelligent
enough to check for anything more complicated than a successful http
request, which it kind of looks like to them. A redirect is expected
when posting things like a new member registration, if I think about
it. So I have to just close the connection on the tcp level to show
they are not successful. Or completely block any incoming connection.
Which is what fail2ban is doing. Problem there seems to be the sheer
amount of machines doing this, with different IP addresses.
> Now I am worried all this will consume up my
bandwidth allowance (as
> well as eating into system resources of course),
If I am looking at the right VPS's bandwidth
graphs, your traffic
usage has indeed increased from almost nothing before Friday, but it
has only increased to about 16kbps out and 24kbps in, so you don't
need to worry about going through your data transfer allowance.
The vps is 'ta2' and the numbers looks reasonable. I will stop worry
about the bandwidth usage then! And start worrying about what fireworks
to get for tonight instead. ;-)
(After I have implemented the mod_security2 block of course)
Many thanks,
__
/ony
2:31 p.m.
Tuesday, December 31, 2013, 9:55:43 AM, Graham wrote:
Hi Graham,
On Tue, Dec 31, 2013 at 12:11 AM, Tony Andersson
<BitFolkList(a)tony-andersson.com> wrote:
>
> Have a strange attack happening to one of my domains, on the web
> server. It is a small privatish phpBB forum with nothing exciting,
> interesting or valuable going on at all. And it is the only one
> attacked out of a handful web sites on the server.
>
> The site has had a lot of incorrect requests to the server since
> before Christmas. I get POST requests in the region of two per second.
> There's noting in the post request and it is to the root of the
> domain. Like this:
> 184.57.181.141 - - [30/Dec/2013:23:32:24 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 108.205.136.80 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 68.118.233.245 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 173.51.226.12 - - [30/Dec/2013:23:32:25 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 71.10.0.254 - - [30/Dec/2013:23:32:27 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 75.91.250.137 - - [30/Dec/2013:23:32:29 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 108.200.239.239 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 64.40.129.122 - - [30/Dec/2013:23:32:31 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 128.210.19.134 - - [30/Dec/2013:23:32:32 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 12.51.89.194 - - [30/Dec/2013:23:32:33 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 184.57.181.141 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 208.96.191.152 - - [30/Dec/2013:23:32:35 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
> 70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Because it's only the phpBB site that's being
accessed in this way,
it's probably something phpBB-specific. It could be a mobile client
that does something unusual (although it's a strange user-agent string
for something legit), or a phpBB worm / mass hack. Previous phpBB
worms have searched google for phpBB specific urls / signatures and
then exploited them. I suggest that you post on a phpBB specific list,
as you're more likely to find people who know what it is.
Two requests/second isn't really a very effective
DOS attack against
most applications.
Good and valid points! And as I just wrote in reply to Andy, I think
the request looks strange because phpBB is redirecting the original
request, and that is not shown in the Apache access log. (Just like
you only see the 301 there now and noting else.) Fairly sure it is
from the new user registration module, which is completely disabled
now.
Reason for posting here first is that I see this list as a really good
source of experienced knowledge, and other places like the phpBB forum
a bit less so. (Had a problem with my phpBB upgrade and the responses
I got were on the lines of "Force a new installation on top of the old
one and don't worry about the failure of the upgrade process". Which
doesn't sit well with me. And here I am now: The authentication module
for new registrations stopped working for a week and now I have
multiple "attacks" per second compared to a couple per day before. :-P
> The 301 response is something I set up when I
discovered this. There
> should be no POST requests to /, so I do a 301 permanent redirect back
> to the client's own IP address. But that seems to have had no effect
> at all. The requests are still constantly coming in.
>
> I have set up a filter in fail2ban for anyone POSTing to '/' so they
> should be completely banned (using action 'iptables-allports'). But
> due to the sheer amount of different addresses attacking it seems to
> have little effect. Plus the fact I quite often see this in the
> fail2ban log:
> 2013-12-30 23:38:33,080 fail2ban.actions: WARNING [http-ddos] 37.142.202.18 already
banned
>
> So it seems that despite being banned they can still send a request to
> the Apache server? Not sure why, the iptables -L seems to list an
> awful lot of IP addresses and domain names. So the fail2ban filter is
> working as it should with setting up rules in iptables.
fail2ban works aynchronously; the ban rules only get
applied after the
requests appear in the logfiles and fail2ban processes them. So it's
almost certainly possible for several requests to be made from an IP
before fail2ban has a chance to block the IP for the first time. It
should be simple to look in the logs for the timestamps of the
requests and the ban attempt to work out what the sequence of events
was.
I did a scan of the logs, but at 2am I couldn't find anything that my
brain made sense of with regards to that. No two requests from the
same IP within a minute. But I am fairly happy now, 12 hours later and
the "IP already banned" entries have not increased in proportion in
the log, so it is a race condition of some kind, not a fail to ban by
fail2ban. ;-)
> At the same time, postfix is getting a large
amount of requests on
> port 25 too:
>
> Dec 30 23:54:45 bitfolk postfix/smtpd[14601]: connect from unknown[180.67.178.14]
> Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: lost connection after UNKNOWN from
unknown[76.2.133.225]
> Dec 30 23:54:45 bitfolk postfix/smtpd[14071]: disconnect from unknown[76.2.133.225]
> Dec 30 23:54:48 bitfolk postfix/smtpd[27968]: connect from unknown[24.151.82.226]
> Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: lost connection after UNKNOWN from
unknown[173.220.57.214]
> Dec 30 23:54:49 bitfolk postfix/smtpd[14107]: disconnect from
unknown[173.220.57.214]
> Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: lost connection after UNKNOWN from
unknown[72.135.3.145]
> Dec 30 23:54:50 bitfolk postfix/smtpd[10196]: disconnect from unknown[72.135.3.145]
> Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: lost connection after UNKNOWN from
unknown[173.246.215.147]
> Dec 30 23:54:50 bitfolk postfix/smtpd[10354]: disconnect from
unknown[173.246.215.147]
> Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: lost connection after UNKNOWN from
unknown[180.67.178.14]
> Dec 30 23:54:51 bitfolk postfix/smtpd[14601]: disconnect from unknown[180.67.178.14]
>
> And in the mail.warn log:
>
> Dec 30 23:10:15 bitfolk postfix/smtpd[19391]: warning: non-SMTP command from
unknown[96.38.26.186]: UY:l??????????z??????\?
> Dec 30 23:11:22 bitfolk postfix/smtpd[17880]: warning: non-SMTP command from
unknown[181.67.172.79]: U:??[6?
> Dec 30 23:14:46 bitfolk postfix/smtpd[19522]: warning: non-SMTP command from
unknown[24.39.251.34]: @:??>T^R^?d???&U?V<??;W?p4?Gf#???t????,???E?
> Dec 30 23:16:57 bitfolk postfix/smtpd[24688]: warning: non-SMTP command from
unknown[72.181.54.101]: gu:?R?M????
>
> I can only conclude this is sent to the same domain name as is
> attacked on port 80...
Alternatively, it could just be the random scans and
probes that
everyone who runs services on the Internet gets.
The increase in connections and the garbage it sends has increased
from maybe a couple of connection attempts per minute to every second,
and at the same time as I first experienced issues with the phpBB
site, so I suspect it is a "broad spectrum attack".
> Now I am worried all this will consume up my
bandwidth allowance (as
> well as eating into system resources of course), and I have run out of ideas how
> to stop this. Any suggestions are most welcome!
The bandwidth consumption of these requests seems
tiny. Since they've
been going since Christmas they'll have shown up in your weekly data
transfer reports by now, if they were going to cause you a problem.
Thanks Graham, valuable input all of it! Most appreciated. I don't get
any weekly data transfer reports though. Not sure why. I remember in a
distant past getting them, but not for a long time. But Andy says the
bandwidth usage is not high.
Cheers,
__
/ony
2:41 p.m.
Hi,
What's strange about it? It's just Internet Explorer 6 on Windows XP
SP2. A good one to copy if you're really a robot...
Regards,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
70.190.134.120
- - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1" 301 - "-"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Because it's only the phpBB site that's being accessed in this way,
it's probably something phpBB-specific. It could be a mobile client
that does something unusual (although it's a strange user-agent string
for something legit),
3:08 p.m.
On Tue, Dec 31, 2013 at 2:41 PM, Andy Bennett <andyjpb(a)ashurst.eu.org> wrote:
Hi,
What's strange about it? It's just Internet Explorer 6 on Windows XP
SP2. A good one to copy if you're really a robot...
I said "a strange user-agent string for something legit"; it's odd
(strange) to choose a string that obscures what your program is
actually for by claiming to be IE 6, unless you are up to no good.
70.190.134.120 - - [30/Dec/2013:23:32:37 +0000] "POST / HTTP/1.1" 301 -
"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Because it's only the phpBB site that's being accessed in this way,
it's probably something phpBB-specific. It could be a mobile client
that does something unusual (although it's a strange user-agent string
for something legit),
4:34 p.m.
Hi Tony,
On Tue, Dec 31, 2013 at 02:31:34PM +0000, Tony Andersson wrote:
I don't get any weekly data transfer reports
though. Not sure why.
I remember in a distant past getting them, but not for a long
time.
You can check if the weekly report emails are disabled, here:
https://panel.bitfolk.com/xfer/
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
3775
days inactive
3775
days old
15 comments
6 participants
participants (6)
-
Andy Bennett -
Andy Smith -
Graham Bleach -
Mathew Newton -
Max B -
Tony Andersson