2.4.5. Blocking and unblocking bots

Blockable bots

Bots that create unnecessary load on the site or scan it for vulnerabilities are blocked by default. Blocking is performed by User-Agent. Some bots can be unlocked, some can't. Blocked bots for all requests receive answer 403.

Can be unlocked:

  1. SemrushBot
  2. MegaIndex
  3. Dotbot
  4. LTX71
  5. Nmap
  6. velen.io
  7. Re-re Studio
  8. netEstate NE Crawler
  9. statdom.ru
  10. SEOkicks
  11. 7Siters
  12. Seekport Crawler
  13. serpstatbot
  14. panscient.com
  15. ZoominfoBot
  16. SearchAtlas
  17. AwarioRssBot
  18. AwarioSmartBot
  19. NetpeakSpiderBot
  20. GetIntent
  21. ia_archiver
  22. AspiegelBot
  23. WPScan
  24. https://gdnplus.com
  25. ApacheBench
  26. python-requests
  27. BorneoBot
  28. Ahrefs
  29. MJ12Bot

Can't Unlock:

  1. JDatabaseDriverMysqli
  2. 80legs.com
  3. DLE_Spider.exe
  4. xpymep.exe
  5. libwww-perl
  6. BOT for JCE
  7. coccoc
  8. heritrix
  9. Aibot
  10. exabot.com
  11. butterfly
  12. CareerBot
  13. FTRF
  14. www.exb.de
  15. magpie-crawler
  16. SISTRIX
  17. radian6
  18. BLEXBot
  19. JikeSpider
  20. Sosospider
  21. Attentio
  22. filterdb.iss.net
  23. cmscrawler
  24. tkl.iis.u-tokyo.ac.jp
  25. J-BRO
  26. ApacheBench
  27. meanpathbot
  28. IndoXploitTools

Unlocking

V site settings in field «Do not block bots» select the bots that should have access to your site, save the changes and wait 10-15 minutes for them to take effect:

Self-locking

Attention!

The User-Agent is specified on the client side and is subject to change. We recommend checking periodically IP load to define new User-Agents.

Any User-Agent can independently restrict access to the site using directives in the file .htaccess.

Content