3 Jan
2013
3 Jan
'13
6:24 a.m.
Hello,
What's the list's preferred techniques for preventing spidering of a
web application (in this case Mediawiki) by misguided web robots?
robots.txt already in place, but they ignore that of course.
Ideally Apache-based.
I don't particularly care if they are still able to download the
content or not, I just don't want them taking up every single
process slot thus impacting non-abusive 'real' web clients. So a
rate-limiting solution would be acceptable.
Cheers,
Andy
3 Jan
3 Jan
11:32 a.m.
On 03/01/13 06:24, Andy Smith wrote:
Hello,
What's the list's preferred techniques for preventing spidering of a
web application (in this case Mediawiki) by misguided web robots?
robots.txt already in place, but they ignore that of course.
Ideally Apache-based.
I don't particularly care if they are still able to download the
content or not, I just don't want them taking up every single
process slot thus impacting non-abusive 'real' web clients. So a
rate-limiting solution would be acceptable.
Cheers,
Andy
I have read some great stuff on perishablepress.com about blocking
misguided/bad robots, using apache and mod rewrite if I remember correctly.
--
4 Jan
4 Jan
7:53 a.m.
On 03/01/13 11:32, Simon wrote:
I have read some great stuff on perishablepress.com
about blocking
misguided/bad robots, using apache and mod rewrite if I remember
correctly.
Thanks for the mention. I think that this is what you meant:
http://perishablepress.com/blackhole-bad-bots/
Basic idea:
(1) Create a "blackhole" directory, and add it to robots.txt
(2) Add hidden links in your page header or footer that points to the
blackhole directory as a trap for bad bots. The trick is that a bot will
follow this link if they don't honour the robots.txt entry, whereas a
person won't see the link.
(3) Have a PHP script harvest the bot's details and put it in a data file
(4) Have mod_rewrite (via .htaccess) or something similar block
connections from sites listed in the data file.
Clever idea.
--
Regards,
Jan Henkins
3 Jan
3 Jan
6:40 p.m.
On Thu, Jan 03, 2013 at 06:24:54AM +0000, Andy Smith wrote:
Hello,
What's the list's preferred techniques for preventing spidering of a
web application (in this case Mediawiki) by misguided web robots?
robots.txt already in place, but they ignore that of course.
Ideally Apache-based.
I don't particularly care if they are still able to download the
content or not, I just don't want them taking up every single
process slot thus impacting non-abusive 'real' web clients. So a
rate-limiting solution would be acceptable.
http://dominia.org/djao/limitipconn2.html
I implemented this at DreamHost 5 or 6 years ago to great effect and as
far as I know it's still in place. It was implemented for a different
reason, but it worked quite well.
There does seem to be a race condition with regard to a high rate of
incoming connections from a given IP which will cause them to get
dropped slightly lower than the normal threshhold, but I just set our
limit to 20 and it stopped the baddies and I think we got maybe
3 complaints about it ever.
-Jeremy
4369
days inactive
4370
days old
3 comments
4 participants
participants (4)
-
Andy Smith -
Jan Henkins -
Jeremy Kitchen -
Simon