2.4.1.16. Bot Protection
Management of protection against bots for individual pages and sections of the site, as well as setting up access for bad bots is performed in the section "Bot protection":
Bot protection
Important points:
- Bot protection does not work for static files, CNC and addresses with GET parameters.
- Bot protection cannot be installed to the main page of the site.
- When protection from bots not passed, the visitor receives a 429 response.
- The language of the Bot Protection page can be changed in site settings.
Bot protection allows you to increase site security, protect the registration page from automatic registrations, protect the site admin panel from automatic password guessing systems, and reduce the load created by bots during attacks on the site. Examples of pages for which it is recommended to install such protection: for WordPress — /wp-login.php
and /xmlrpc.php
(wherein Not recommended install for /wp-admin/
), for Joomla! — /administrator/
, etc.
Setting protection
- In the tab "Captcha" click "Install bot protection".
- Fill out the form and click "Save":
- "Application" — part of the URL in which the specified pages will be searched:
- "URL starts with" — at the beginning of the URL.
- "URL ends with" — at the end of the URL.
- "URL contains" — in any part of the URL.
- "List of pages" — pages that will be searched for in the selected part of the URL:
- Every page a new line.
- Specify domain not necessary.
- Page URLs must start with
/
and point to real files and directories.
- "Protection method" — protection method that will be triggered when the page is opened by a site visitor:
- "Calculating the amount" — a simple mathematical problem is displayed to the visitor, the page opens after its successful solution.
- "Captcha" — the visitor is shown a standard ReCaptcha, the page opens after passing it.
- "JavaScript validation" — imperceptibly for the visitor in his browser, a simple arithmetic problem is calculated using JavaScript, the page opens in case of successful calculation. Note If JavaScript is disabled, the visitor will see an access error and a request to enable JavaScript.
- "Disable for User-Agent" — a list of User-Agent separated by commas, for which will not act protection. The field can be left blank.
- Wait approximately 10 minutes for the changes to take effect.
- Open the protected pages in the browser and check the protection.
Statistics
At the bottom of the tab "Captcha" Displays statistics for the last 3 days. On the graph, you can see how many successful visits to protected pages were and how many hits were blocked due to the fact that the protection was not passed.
Bad bots
Some bots can create unnecessary load on the site or scan it for vulnerabilities. For such bots, access to the site is blocked by default. Blocking is performed by User-Agent, blocked bots for all requests receive answer 403. If necessary, on the tab, you can unblock access to the site for such bots:
On the tab you can:
- Unblock individual bots — just select them in the list.
- Return the blocking of individual bots — you need to remove the choice from them.
- Revert blocking of all bots — button "Block all bots" at the bottom of the list.
- Unblock all bots — button "Disable bot protection" at the top of the list (not recommended).
Blockable bots
- 7Siters
- 80legs.com
- Ahrefs
- AhrefsBot
- Aibot
- Amazonbot
- ApacheBench
- AspiegelBot
- Attentio
- AwarioBot
- AwarioRssBot
- AwarioSmartBot
- Barkrowler
- BLEXBot
- BorneoBot
- BOT for JCE
- BuiltWith
- CareerBot
- CCBot
- cmscrawler
- coccoc
- DataForSeoBot
- domaincrawler.com
- Dotbot
- exabot.com
- filterdb.iss.net
- GeedoBot
- GetIntent
- heritrix
- https://gdnplus.com
- ia_archiver
- IndoXploitTools
- J-BRO
- JDatabaseDriverMysqli
- JikeSpider
- Keys.so
- KOCMOHABT
- libwww-perl
- Linkfluence
- LTX71
- magpie-crawler
- meanpathbot
- MegaIndex
- MJ12Bot
- NetcraftSurveyAgent
- netEstate NE Crawler
- NetpeakSpiderBot
- Nmap
- panscient.com
- PetalBot
- python-requests
- radian6
- Re-re Studio
- Riddler
- Screaming Frog SEO Spider
- SearchAtlas
- Seekport Crawler
- SeekportBot
- SemrushBot
- SEOkicks
- SeopultContentAnalyzer
- Serendeputy
- serpstatbot
- SISTRIX
- Sosospider
- statdom.ru
- tkl.iis.u-tokyo.ac.jp
- velen.io
- weborama
- WPScan
- www.exb.de
- xpymep.exe
- ZoominfoBot