2.4.1.2.4. Bot protection

Notes:

  • Checking is not performed for IP addresses that are added to the Firewall whitelist.
  • If the bot you're looking for isn't on the list, use the "Suggest bot" button.

Principle of operation

Some bots can create senseless load on the site or scan it for vulnerabilities. For such bots, access to the site is blocked by default. Blocking is performed by User-Agent, blocked bots get response 403 for all requests.

flowchart LR request@{ shape: stadium, label: "➡️ Request" } whitelist@{ shape: rounded, label: "🧱 Restricted
access mode" } blacklist@{ shape: rounded, label: "📋 IP blacklist" } firewall@{ shape: rounded, label: "🔥 Firewall" } subgraph bots [🤖 Bots] list@{ shape: diamond, label: "📋 Bot
in list?" } E[ ]:::empty classDef empty height: 0, width: 0 subgraph rules [Rules] allow@{ shape: rounded, label: "✔️ Always allow" } ddos@{ shape: rounded, label: "🛡 Allow,
but restrict during times of high load on site" } deny@{ shape: rounded, label: "❌ Deny" } end end countries@{ shape: rounded, label: "🌎 Countries" } suspicious@{ shape: rounded, label: "🔍 Suspicious IPs" } iam@{ shape: rounded, label: "🛡️ I'm Under Attack" } site@{ shape: stadium, label: "✅ Site" } error_403@{ shape: stadium, label: "🚫 Error 403" } request-->whitelist request-->blacklist blacklist-->firewall whitelist-->firewall firewall-->list list---E-->|✅|rules allow-->site ddos-->countries deny-....->error_403 list-->|❌ or request not from bot|countries countries-->suspicious suspicious-->iam iam-->site firewall-.->site

In the chain of all site protection stages, bot protection comes after restricted access mode, IP blacklist and Firewall before country access restriction, checking suspicious IPs and I'm Under Attack.

Configure

Access to the site for bots is configured in the "Site protection" section on the "Bots" tab using the following rules:

  • Always allow — bots will be allowed to access the site. Exception: IP addresses that are restricted by the IP blacklist or restricted access mode.
  • Allow, but restrict during times of high load on site — if the bot falls under any rule that restricts access (for example, country access restrictions, I'm Under Attack), access will be denied. This is useful when you need this bot to work, but if the load exceeds and protection is enabled, access to it is denied and it gets error 429.
  • Deny — bots will get error 403.

To set rules, use the ✔️, 🛡, and ❌ buttons next to the name of the bot or group of bots:

The "Apply to all sites in hosting account" button sets the current site settings for all sites in this hosting account.

The "Restore default rules" button sets the standard bot protection settings.

Statistics

Statistics allow you to quickly determine which bots are generating the most requests and which ones may require access restrictions.

At the top of the "Bots" tab, the top 20 bots are displayed based on the number of allowed requests (total requests minus blocked requests) by day for the last 10 days:

The row color corresponds to the current rule set for the bot (green = ✔️, orange = 🛡, red = ❌).

Clicking on the bot's name scrolls the page to its access settings and highlights them.

Statistics for each bot are also displayed in the settings next to its name (if there was a minimum of 1 request):

Content
    (4)