2.34. Blocking and unblocking bots

1 By default, the hosting blocks:

  • Bots that create a senseless load on the site or scan it for vulnerabilities.
  • Requests from IP addresses, which we consider to be bots, for which clients do not support JavaScript.

List of User-Agents who have limited access to hosting (in response to requests, they receive answer 403) and which cannot be unlocked:

  1. JDatabaseDriverMysqli
  2. 80legs.com
  3. DLE_Spider.exe
  4. xpymep.exe
  5. libwww-perl
  6. BOT for JCE
  7. coccoc
  8. heritrix
  9. Aibot
  10. exabot.com
  11. butterfly
  12. CareerBot
  13. FTRF
  14. www.exb.de
  15. magpie-crawler
  16. SISTRIX
  17. radian6
  18. BLEXBot
  19. JikeSpider
  20. Sosospider
  21. Attentio
  22. filterdb.iss.net
  23. cmscrawler
  24. tkl.iis.u-tokyo.ac.jp
  25. J-BRO
  26. ApacheBench
  27. meanpathbot
  28. IndoXploitTools

Some User-Agents used by bots are blocked due to the creation of meaningless load, but their blocking can be disabled. This feature is available for bots:

  1. SemrushBot
  2. MegaIndex
  3. Dotbot
  4. LTX71
  5. Nmap
  6. velen.io
  7. Re-re Studio
  8. netEstate NE Crawler
  9. statdom.ru
  10. SEOkicks
  11. 7Siters
  12. Seekport Crawler
  13. serpstatbot
  14. panscient.com
  15. ZoominfoBot
  16. SearchAtlas
  17. AwarioRssBot
  18. AwarioSmartBot
  19. NetpeakSpiderBot
  20. GetIntent
  21. ia_archiver
  22. AspiegelBot
  23. WPScan
  24. https://gdnplus.com
  25. ApacheBench
  26. python-requests
  27. BorneoBot
  28. Ahrefs
  29. MJ12Bot

To deactivate the lock in site settings against "Do not block bots" select the bots you don't want to block and save the changes:

Changes take effect within 10-15 minutes from the moment of saving.

Warning!

User-agent can be specified on the sending client side and can change over time. It is highly recommended to check from time to time IP load to define new User-agents.

Any other User-Agent can independently restrict access using directives in .htaccess.

Content