17 Jul
2014
17 Jul
'14
3:58 p.m.
Is anyone else suffering from this pingback ddos at the moment? My server is
only a low spec one and keeps being brought to its knees by it. Now I've had
the time to actually look at the logs and work out what is happening I'm in the
process of putting some form of protection in there, although the quick fixes
seem to impact functionality. I was just wondering what anyone else had done,
assuming others have been impacted too. Not having read in full detail yet, I'm
wondering why this is just a Wordpress issue. Wouldn't you be able to do the
same thing with any blog or cms that uses pingback?
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
17 Jul
17 Jul
6:18 p.m.
Paul Tansom said:
Is anyone else suffering from this pingback ddos at
the moment? My server is
only a low spec one and keeps being brought to its knees by it. Now I've had
the time to actually look at the logs and work out what is happening I'm in the
process of putting some form of protection in there, although the quick fixes
seem to impact functionality. I was just wondering what anyone else had done,
assuming others have been impacted too. Not having read in full detail yet, I'm
wondering why this is just a Wordpress issue. Wouldn't you be able to do the
same thing with any blog or cms that uses pingback?
I haven't seen this, but that's because I turn off pingbacks on all WP
sites. There's a plugin to do it on sites that have had them,
<https://wordpress.org/plugins/disable-xml-rpc-pingback/>, without
totally disabling xml-rpc in case you need it for something else.
If you can't install plugins for some reason, but can access the
database, there's a couple of lines of SQL to turn it off on all
existing pages and posts.
You won't be missing out on anything, because the overwhelming majority
of pingbacks have been 'not good' for years. Turning them off is a part
of setting up a WP site...
There are more WordPress sites, so they get targeted first. The people
developing WP also think that because people like getting comments, they
should leave a new site completely vulnerable to spam comments and
pingbacks and until one issue was fixed, allowed attackers to do things
like probe your server with them.
Ian
9:44 p.m.
** Ian <ian(a)lovingboth.com> [2014-07-17 19:23]:
Paul Tansom said:
** end quote [Ian]
Yes, that's the plugin I'm trying to install. I've clearly been lax, but I
thought pingbacks were disabled by default, I've certainly never had any,
although from what I've read the option has been removed from the configuration
and it has been enabled by default. At the moment I'm blocking xmlrpc.php from
the .htaccess, but each time I enable Apache again the load goes from around
0.09 through to 30, 40 or 50 within a minute or so. It is very difficult to
test and diagnose if you can't get any response from the server because of the
load :( I'm just putting something together with fail2ban, then with any luck I
can put the plugin in place and experiment.
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
Is anyone else suffering from this pingback ddos
at the moment? My server is
only a low spec one and keeps being brought to its knees by it. Now I've had
the time to actually look at the logs and work out what is happening I'm in the
process of putting some form of protection in there, although the quick fixes
seem to impact functionality. I was just wondering what anyone else had done,
assuming others have been impacted too. Not having read in full detail yet, I'm
wondering why this is just a Wordpress issue. Wouldn't you be able to do the
same thing with any blog or cms that uses pingback?
I haven't seen this, but that's because I turn off pingbacks on all WP
sites. There's a plugin to do it on sites that have had them,
<https://wordpress.org/plugins/disable-xml-rpc-pingback/>, without
totally disabling xml-rpc in case you need it for something else.
If you can't install plugins for some reason, but can access the
database, there's a couple of lines of SQL to turn it off on all
existing pages and posts.
You won't be missing out on anything, because the overwhelming majority
of pingbacks have been 'not good' for years. Turning them off is a part
of setting up a WP site...
There are more WordPress sites, so they get targeted first. The people
developing WP also think that because people like getting comments, they
should leave a new site completely vulnerable to spam comments and
pingbacks and until one issue was fixed, allowed attackers to do things
like probe your server with them.
18 Jul
18 Jul
8:30 a.m.
Paul Tansom said:
Yes, that's the plugin I'm trying to install.
I've clearly been lax, but I
thought pingbacks were disabled by default,
They got enabled in July 2012 in one of the not really documented
changes for the then forthcoming version 3.5 released later that year, sigh.
At the moment I'm blocking xmlrpc.php from
the .htaccess, but each time I enable Apache again the load goes from around
0.09 through to 30, 40 or 50 within a minute or so. It is very difficult to
test and diagnose if you can't get any response from the server because of the
load :( I'm just putting something together with fail2ban, then with any luck I
can put the plugin in place and experiment.
If I look at the log files, the most any site here has had this month is
97 accesses.
I wonder if someone is doing a DDOS on you - the ability to do that via
pingbacks is the subject of a WP bug report that goes back about five years.
I'd expect using fail2ban to ban anyone who is trying to access that
file frequently should help.
Ian
8:38 a.m.
Paul Tansom said:
I thought pingbacks were disabled by default
<https://core.trac.wordpress.org/ticket/21509>
"Quite a bit has changed since we introduced off-by-default for XML-RPC.
Their code has improved, and it is no longer considered a second-class
citizen when it comes to API development, thanks to the work of a large
team of awesome contributors. Security is no greater a concern than the
rest of core.
"There is no longer a compelling reason to disable this by default. It's
time we should remove the option entirely."
Ian
23 Jul
23 Jul
10:29 p.m.
** Ian <ian(a)lovingboth.com> [2014-07-18 09:32]:
** end quote [Ian]
Yes, I think it is a DDOS, or rather several. I've had issues a few times over
the past couple of months; oddly since I upgraded to 12.04 or thereabouts.
Initially I thought it was increased load from the newer version of Ubuntu, but
wasn't entirely convinced. Then I found that stopping and restarting Apache
cleared the load and made note to investigate the logs. This time round
stopping and restarting Apache has made no difference as when I restart the
load increases within a few minutes. Clearly a higher spec VM would help, but
it seems silly throwing resources at it rather than sorting it properly.
Looking at the logs I've been getting multiple access for the xmlrpc.php file
from multiple sources. Using .htaccess to deny access to the file made no
difference, not sure why, I should have investigated, but instead I reached for
fail2ban. Obviously coming from multiple IPs this didn't work (but I tried
anyway). In the end I managed to install the WP plugin and things settled,
although I have noted a few occaisions when the load has increased
significantly (errors from a plugin I have that updates WP content from RSS
feeds alerts me to this).
I'll have to take a look at the logs, but other things took priority when it
settled (classic firefighting!). With a talk on Saturday and a Raspberry Jam on
Sunday (both work related - it's a hard life when a Raspberry Jam is work) I've
been flat out all weekend too.
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
Paul Tansom said:
Wow, that long ago, I missed the change. I've installed a few times since then,
but clearly have got so used to doing it I've not really taken a close look at
changes in setup for a while!
Yes, that's the plugin I'm trying to
install. I've clearly been lax, but I
thought pingbacks were disabled by default,
They got enabled in July 2012 in one of the not really documented
changes for the then forthcoming version 3.5 released later that year, sigh. At the moment
I'm blocking xmlrpc.php from
the .htaccess, but each time I enable Apache again the load goes from around
0.09 through to 30, 40 or 50 within a minute or so. It is very difficult to
test and diagnose if you can't get any response from the server because of the
load :( I'm just putting something together with fail2ban, then with any luck I
can put the plugin in place and experiment.
If I look at the log files, the most any site here has had this month is
97 accesses.
I wonder if someone is doing a DDOS on you - the ability to do that via
pingbacks is the subject of a WP bug report that goes back about five years.
I'd expect using fail2ban to ban anyone who is trying to access that
file frequently should help. 
24 Jul
24 Jul
2:43 a.m.
Hi Paul,
On Wed, Jul 23, 2014 at 11:29:36PM +0100, Paul Tansom wrote:
This time round stopping and restarting Apache has
made no
difference as when I restart the load increases within a few
minutes. Clearly a higher spec VM would help, but it seems silly
throwing resources at it rather than sorting it properly.
The very first thing you should do is make it so that large numbers
of simultaneous HTTP requests cannot bring down your entire VM.
That is most likely happening because too many Apache children are
running at once, using all the RAM and sending the VM into swap.
If so then the first step would be to work out how many Apache
children can actually be safely run simultaneously, and make sure
that MaxClients is set so this maximum value. When the limit is
reached the connections will stack up and your website will cease to
respond, but your VM won't cease to respond.
That will give you some chance to try some other approaches and see
what works best.
Further steps that don't involve increasing the RAM available might
include lighttpd and/or nginx. At some point you're going to need
something capable of interpreting PHP but if you can make as many
requests as possible avoid getting that far, serving things out of
cache, sending static requests to lighttd/nginx etc., then you get
further.
At the very least you can partition the sluggishness to just the
requests that need to hit PHP: you could have PHP being called as CGI
via a fixed number of FastCGI backend servers, while every other
request is served directly by lighttpd/nginx much faster, even under
load.
Of course, Apache is easy, everything works with Apache, all the
docs are for Apache. But Apache is a memory hog. It is a trade-off
to use more memory and have an easier life. :)
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
8:52 a.m.
On Thu, 2014-07-24 at 02:43 +0000, Andy Smith wrote:
*snip*
Further steps that don't involve increasing the
RAM available might
include lighttpd and/or nginx. At some point you're going to need
something capable of interpreting PHP but if you can make as many
requests as possible avoid getting that far, serving things out of
cache, sending static requests to lighttd/nginx etc., then you get
further.
*snip*
From personal experience, using FastCGI + Apache's threaded worker is
"enough" memory wise. It might be a good intermediary step as it won't
require configuring a new httpd.
10:55 p.m.
** Andy Smith <andy(a)bitfolk.com> [2014-07-24 03:44]:
On Wed, Jul 23, 2014 at 11:29:36PM +0100, Paul Tansom
wrote:
** end quote [Andy Smith]
Yes, I've sort of got used to Apache and trundled along with it for what must
be about 15 years now. I have been seriously thinking of trying nginx recently
though, so this may just be the thing that makes be explore it further. I'm
going to have to look through my Apache config more closely now things have
settled down, I have a suspicion that somewhere along the line my
customisations have been removed - either during a reinstall or an update - as
I did have the number of children cranked down (I don't get much traffic after
all). As ever it is your own systems that get neglected.
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
=============================================================================
Registered in England | Company No: 4905028 | Registered Office: Ralls House,
Parklands Business Park, Forrest Road, Denmead, Waterlooville, Hants, PO7 6XP
This time round stopping and restarting Apache
has made no
difference as when I restart the load increases within a few
minutes. Clearly a higher spec VM would help, but it seems silly
throwing resources at it rather than sorting it properly.
The very first thing you should do is make it so that large numbers
of simultaneous HTTP requests cannot bring down your entire VM.
That is most likely happening because too many Apache children are
running at once, using all the RAM and sending the VM into swap.
If so then the first step would be to work out how many Apache
children can actually be safely run simultaneously, and make sure
that MaxClients is set so this maximum value. When the limit is
reached the connections will stack up and your website will cease to
respond, but your VM won't cease to respond.
That will give you some chance to try some other approaches and see
what works best.
Further steps that don't involve increasing the RAM available might
include lighttpd and/or nginx. At some point you're going to need
something capable of interpreting PHP but if you can make as many
requests as possible avoid getting that far, serving things out of
cache, sending static requests to lighttd/nginx etc., then you get
further.
At the very least you can partition the sluggishness to just the
requests that need to hit PHP: you could have PHP being called as CGI
via a fixed number of FastCGI backend servers, while every other
request is served directly by lighttpd/nginx much faster, even under
load.
Of course, Apache is easy, everything works with Apache, all the
docs are for Apache. But Apache is a memory hog. It is a trade-off
to use more memory and have an easier life. :)
11:59 p.m.
Matt Molyneaux said:
From personal experience, using FastCGI + Apache's
threaded worker
is "enough" memory wise. It might be a good intermediary step as it
won't require configuring a new httpd.
One of the reasons I use mod-suphp is that it means you don't need
mod-php which really cuts down on the size of each resident apache2
process. It's slightly slower to load php5-cgi, but hardly anyone will
notice the difference, especially if you use caching.
There's an article on the wiki about Apache tuning.
Ian
3573
days inactive
3580
days old
9 comments
4 participants
participants (4)
-
Andy Smith -
Ian -
Matt Molyneaux -
Paul Tansom