We have many different setups of hosting (some shared, some custom). What are your best suggestions for rate limiting and blocking certain aggressive crawlers for servers hosting WordPress? And yes, we do have Cloudflare running on some of the builds and have our own testing going, but interested to know what others are doing.
-
Join the MyBroadband community
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
-
Question of the Day: How was your voting experience?
Bot & Crawler Blocking & Rate Limiting Suggestions
- Thread starter MtraX
- Start date
Koos Custodiet
Expert Member
After trying to be nice and finding that a large number of the bots ignore robots.txt, I implemented a user agent blocklist. That worked for a while, but many bots are not honest there either.
So now there is a button to click, with some user agents whitelisted. So far that seems to be holding.
So now there is a button to click, with some user agents whitelisted. So far that seems to be holding.
netstrider
Executive Member
- Joined
- Nov 1, 2006
- Messages
- 5,897
- Reaction score
- 4,933
There are an exceptional number of false bots, actually.
