2.4.1.16. Bot Protection

Management of protection against bots for individual pages and sections of the site, as well as setting up access for bad bots is performed in the section "Bot protection":

Important points:

  • Bot protection does not work for static files, CNC and addresses with GET parameters.
  • Bot protection cannot be installed to the main page of the site.
  • When protection from bots not passed, the visitor receives a 429 response.
  • The language of the Bot Protection page can be changed in site settings.

Bot protection allows you to increase site security, protect the registration page from automatic registrations, protect the site admin panel from automatic password guessing systems, and reduce the load created by bots during attacks on the site. Examples of pages for which it is recommended to install such protection: for WordPress — /wp-login.php and /xmlrpc.php (wherein Not recommended install for /wp-admin/), for Joomla! — /administrator/, etc.

  1. In the tab "Captcha" click "Install bot protection".
  2. Fill out the form and click "Save":
    • "Application" — part of the URL in which the specified pages will be searched:
      • "URL starts with" — at the beginning of the URL.
      • "URL ends with" — at the end of the URL.
      • "URL contains" — in any part of the URL.
    • "List of pages" — pages that will be searched for in the selected part of the URL:
      • Every page a new line.
      • Specify domain not necessary.
      • Page URLs must start with / and point to real files and directories.
    • "Protection method" — protection method that will be triggered when the page is opened by a site visitor:
      • "Calculating the amount" — a simple mathematical problem is displayed to the visitor, the page opens after its successful solution.
      • "Captcha" — the visitor is shown a standard ReCaptcha, the page opens after passing it.
      • "JavaScript validation" — imperceptibly for the visitor in his browser, a simple arithmetic problem is calculated using JavaScript, the page opens in case of successful calculation. Note If JavaScript is disabled, the visitor will see an access error and a request to enable JavaScript.
    • "Disable for User-Agent" — a list of User-Agent separated by commas, for which will not act protection. The field can be left blank.
  3. Wait approximately 10 minutes for the changes to take effect.
  4. Open the protected pages in the browser and check the protection.

At the bottom of the tab "Captcha" Displays statistics for the last 3 days. On the graph, you can see how many successful visits to protected pages were and how many hits were blocked due to the fact that the protection was not passed.

Some bots can create unnecessary load on the site or scan it for vulnerabilities. For such bots, access to the site is blocked by default. Blocking is performed by User-Agent, blocked bots for all requests receive answer 403. If necessary, on the tab, you can unblock access to the site for such bots:

On the tab you can:

  • Unblock individual bots — just select them in the list.
  • Return the blocking of individual bots — you need to remove the choice from them.
  • Revert blocking of all bots — button "Block all bots" at the bottom of the list.
  • Unblock all bots — button "Disable bot protection" at the top of the list (not recommended).
  1. 7Siters
  2. 80legs.com
  3. Ahrefs
  4. AhrefsBot
  5. Aibot
  6. Amazonbot
  7. ApacheBench
  8. AspiegelBot
  9. Attentio
  10. AwarioBot
  11. AwarioRssBot
  12. AwarioSmartBot
  13. Barkrowler
  14. BLEXBot
  15. BorneoBot
  16. BOT for JCE
  17. BuiltWith
  18. CareerBot
  19. CCBot
  20. cmscrawler
  21. coccoc
  22. DataForSeoBot
  23. domaincrawler.com
  24. Dotbot
  25. exabot.com
  26. filterdb.iss.net
  27. GeedoBot
  28. GetIntent
  29. heritrix
  30. https://gdnplus.com
  31. ia_archiver
  32. IndoXploitTools
  33. J-BRO
  34. JDatabaseDriverMysqli
  35. JikeSpider
  36. Keys.so
  37. KOCMOHABT
  38. libwww-perl
  39. Linkfluence
  40. LTX71
  41. magpie-crawler
  42. meanpathbot
  43. MegaIndex
  44. MJ12Bot
  45. NetcraftSurveyAgent
  46. netEstate NE Crawler
  47. NetpeakSpiderBot
  48. Nmap
  49. panscient.com
  50. PetalBot
  51. python-requests
  52. radian6
  53. Re-re Studio
  54. Riddler
  55. Screaming Frog SEO Spider
  56. SearchAtlas
  57. Seekport Crawler
  58. SeekportBot
  59. SemrushBot
  60. SEOkicks
  61. SeopultContentAnalyzer
  62. Serendeputy
  63. serpstatbot
  64. SISTRIX
  65. Sosospider
  66. statdom.ru
  67. tkl.iis.u-tokyo.ac.jp
  68. velen.io
  69. weborama
  70. WPScan
  71. www.exb.de
  72. xpymep.exe
  73. ZoominfoBot
Content