The bot management module of Web Application Firewall (WAF) mitigates attacks that are initiated by automation tools such as emulators and malicious scripts. The attacks include data crawling, fraud, credential stuffing attacks, spam user registrations, auto-purchase bots, promotion abuse, and SMS flood attacks. We recommend that you configure bot management rules based on analytical data of bot traffic to reduce data leaks and risks during marketing activities. This helps reduce server workloads and bandwidth costs. This topic describes how to enable the bot management module and configure bot management rules.
Feature description
The bot management module provides bot traffic analysis, basic protection, and scenario-specific protection. The bot management module helps you identify bot traffic and defend against crawlers to protect your web services from being crawled.
Bot traffic analysis: You can use the feature regardless of whether you enable the bot management module.
You can view the results of bot traffic analysis without enabling the bot management module, including the bot traffic trends within a specific time range, top 20 risky clients, top 20 risky IP addresses, and analytical data of bot traffic to protected objects. This helps you identify and locate risky domain names in an efficient manner. For more information, see View analytical data of bot traffic.
You can apply for a trial or enable the bot management module to configure scenario-specific protection for risky domain names. For more information about how to apply for a trial and how to enable the bot management module, see Enable the bot management module.
Scenario-specific Protection: You must enable the bot management module before you can use the feature.
The scenario-specific protection feature provides SDKs that you can integrate to configure custom rules to protect your websites and apps from crawlers. For more information, see Create an anti-crawler rule for websites and Create an anti-crawler rule for apps.
This feature is suitable for users who are sensitive to bot traffic.
Basic Protection: You must enable the bot management module before you can use the feature.
The basic protection feature detects Layer 4 and Layer 7 bot traffic by using fingerprinting techniques. You can use the feature without the need to integrate SDKs. For more information, see Create a basic protection rule.
This feature is suitable for users who want to defend against low-level crawlers by configuring protection rules in a simplified manner.
Prerequisites
Web services are added to WAF on the Website Configuration page. For more information, see Website configuration overview.
The Anti-Bot SDK is integrated into your apps. Make sure that this prerequisite is met if you want to configure a protection template of the App type to defend against crawlers. For more information, see Integrate the Anti-Bot SDK into Android apps and Integrate the Anti-Bot SDK into iOS apps.
View analytical data of bot traffic
Log on to the WAF 3.0 console. In the top navigation bar, select the resource group and region of the WAF instance. You can select Chinese Mainland or Outside Chinese Mainland.
In the left-side navigation pane, choose .
On the Bot Traffic Analysis tab, view bot traffic trend, top 20 risky clients, top 20 risky IP addresses, and analytical data of bot traffic to protected objects.
Enable the bot management module
Log on to the WAF 3.0 console. In the top navigation bar, select the resource group and region of the WAF instance. You can select Chinese Mainland or Outside Chinese Mainland.
In the left-side navigation pane, choose .
Enable the bot management module.
Apply for a free trial
NoteYou can apply for a free trial of the bot management module only once. You can apply for a free trial only if you use a WAF Pro, Enterprise, or Ultimate instance.
You can receive a seven-day free trial after your application is approved. The analytical data that is generated during the trial period is available only during the trial period. If you want to retain the analytical data, enable the bot management module before the trial period ends.
On the Bot Traffic Analysis tab, click Apply for Trial. On the page that appears, enter the application information and click Submit.
After you submit your trial application, Alibaba Cloud engineers will contact you based on the contact information that you submit to confirm information that is related to your application. After the application is approved, the bot management module is automatically enabled for your WAF instance.
Enable the bot management module
On the Bot Traffic Analysis, Scenario-specific Protection, or Basic Protection tab, click Purchase Now.
On the buy page that appears, set the Bot Management - Web Application Protection or Bot Management - App Protection parameter to Enable and complete the payment.
NoteAfter you enable bot management for web application protection, you can configure basic protection rules and anti-crawler rules for websites.
After you enable bot management for app protection, you can configure basic protection rules and anti-crawler rules for apps.
If you want to configure basic protection rules, anti-crawler rules for websites, and anti-crawler rules for apps, enable both bot management for web application protection and bot management for app protection.
After you enable the bot management module, you can configure scenario-specific protection rules on the Bot Traffic Analysis tab. To configure a scenario-specific protection rule, go to the Bot Traffic Analysis of Protected Objects section of the page, find the domain name of the website or app that you want to protect, and then click Configure Protection in the Actions column. For more information, see Create an anti-crawler rule for websites and Create an anti-crawler rule for apps.
If you want to configure basic protection rules to defend against low-level crawlers, go to the Basic Protection tab. For more information, see Create a basic protection rule.
Create an anti-crawler rule for websites
If you want to use WAF to mitigate the security threats that are caused by bot traffic on web pages, HTML5 pages, or HTML5 apps, we recommend that you create a protection template and configure an anti-crawler rule for websites.
If a request from a client matches a protection rule in which the Action parameter is set to Run JavaScript Validation or Slider CAPTCHA, WAF performs JavaScript validation or slider CAPTCHA verification on the client. If the client passes the validation or verification, WAF adds the
acw_sc__v2
oracw_sc__v3
cookie to the header of the request to indicate that the client passed the validation or verification.If you configure a scenario-specific protection temple for the bot management module and enable the automatic integration of the Web SDK feature, WAF adds the
ssxmod_itna
,ssxmod_itna2
, andssxmod_itna3
cookies to the HTTP request header. The cookies are used to obtain fingerprint information about the browser on the client. The fingerprint information includes the host field in the HTTP request and the height and width of the browser window.
Log on to the WAF 3.0 console. In the top navigation bar, select the resource group and region of the WAF instance. You can select Chinese Mainland or Outside Chinese Mainland.
In the left-side navigation pane, choose .
On the Scenario-specific Protection tab, click Create Template.
In the Configure Scenarios step, configure the parameters and click Next. The following table describes the parameters.
Parameter
Description
Template Name
Enter a name for the template.
The name of the template must be 1 to 255 characters in length and can contain letters, digits, periods (.), underscores (_), and hyphens (-).
Template Description
Enter a description for the template.
Service Type
Select Websites. This way, WAF protects web pages, HTML5 pages, and HTML5 apps.
Web SDK Integration
For more information, see Integrate the Web SDK into web applications.
Traffic Characteristics
Add match conditions to identify traffic that is destined for the domain name that you want to protect. To add a match condition, you must configure the match field, logical operator, and match content. Make sure that the match field is a header field of HTTP requests. You can add up to five conditions. The conditions are evaluated by using a logical AND. For more information about match fields, see Match conditions.
In the Configure Protection Rules step, configure the parameters and click Next. The following table describes the parameters.
Parameter
Description
Risk Identification
Select Business Security and enter relevant information. For more information, see Risk identification.
The feature helps block requests from abnormal mobile phone numbers based on Fraud Detection. You are charged based on rule hits.
Legitimate Bot Management
Select Spider Whitelist. Then, select search engines from the drop-down list.
After you select Spider Whitelist and then select search engines from the drop-down list, requests that are sent from the crawler IP addresses of the search engines are sent to the origin server. The bot management module no longer checks these requests.
Bot Characteristic Detection
Bot Behavior Detection
Bot Threat Intelligence
In the Configure Effective Scope step, configure the parameters and click Next. The following table describes the parameters.
Parameter
Description
Apply To
Select the protected objects or protected object groups to which you want to apply the protection template in the Objects to Select section and click the icon to move the protected objects or protected object groups to the Selected Objects section.
Effective Time and Canary Release
You must specify validity periods and configure canary release settings for the protection rules. If you do not specify validity periods or configure canary release settings, canary release is disabled for the rules and the rules are permanently valid.
Find the rule whose configurations you want to modify and click Edit in the Actions column.
Configure canary release settings and specify validity periods.
Canary Release: You can turn on Canary Release to apply the rule to a specific proportion of objects.
If you turn on Canary Release, you must configure the Dimension and Canary Release Ratio parameters. Valid values of the Dimension parameter: IP Address, Custom Header, Custom Header, Custom Cookie, and Session.
Effective Mode
Permanently Effective (default): If you select Permanently Effective, the protection rule is permanently in effect.
Fixed Schedule: You can specify a time zone and a period of time during which you want the template to be in effect.
Recurring Schedule: You can specify a time zone, days of the week, and a period of time in a day. The template is in effect during the same period of time in a day on the same days of the week.
You can select multiple rules to specify validity periods and configure canary release settings for the rules at the same time.
In the Verify Protection Effect step, test the anti-crawler rule.
To prevent false positives that are caused by improper rule configurations or compatibility issues, we recommend that you verify the protection effect of the rule before you publish an anti-crawler rule If the configurations are correct, click Skip.
Test steps:
By default, a newly created template is enabled. On the Scenario-specific Protection tab, you can perform the following operations:
Click a template card to view the rule information about the template.
Copy, Edit, or Delete a template.
Turn on or turn off the switch to enable or disable a template.
View the settings of the Action and Protected Object/Group parameters of a template.
Create an anti-crawler rule for apps
You can configure anti-crawler rules for native iOS or Android apps to protect your services against crawlers. HTML5 apps are not native iOS or Android apps.
If a request from a client matches a protection rule in which the Action parameter is set to Run JavaScript Validation or Slider CAPTCHA, WAF performs JavaScript validation or slider CAPTCHA verification on the client. If the client passes the validation or verification, WAF adds the acw_sc__v2
or acw_sc__v3
cookie to the header of the request to indicate that the client passed the validation or verification.
Log on to the WAF 3.0 console. In the top navigation bar, select the resource group and region of the WAF instance. You can select Chinese Mainland or Outside Chinese Mainland.
In the left-side navigation pane, choose .
On the Scenario-specific Protection tab, click Create Template.
In the Configure Scenarios step, configure the parameters and click Next. The following table describes the parameters.
Parameter
Description
Template Name
Enter a name for the template.
The name of the template must be 1 to 255 characters in length and can contain letters, digits, periods (.), underscores (_), and hyphens (-).
Template Description
Enter a description for the template.
Service Type
Select App to configure anti-crawler rules for native iOS and Android apps. HTML5 apps are excluded.
App SDK Integration
WAF provides the Anti-Bot SDK to enhance protection capabilities for native Android and iOS apps. After the Anti-Bot SDK is integrated into apps, the Anti-Bot SDK collects the characteristics of clients and generates security signatures in requests. WAF identifies and blocks requests that are identified as unsafe based on the signatures.
You can perform the following steps to integrate the Anti-Bot SDK.
Obtain the SDK for iOS apps. To obtain the SDK, submit a ticket.
Click Obtain and Copy AppKey to send SDK initialization requests.
Integrate the Anti-Bot SDK into your apps. For more information, see Integrate the Anti-Bot SDK into iOS apps.
Traffic Characteristics
Add match conditions to identify traffic that is destined for the domain name that you want to protect. To add a match condition, you must configure the match field, logical operator, and match content. Make sure that the match field is a header field of HTTP requests. You can add up to five conditions. The conditions are evaluated by using a logical AND. For more information about match fields, see Match conditions.
In the Configure Protection Rules step, configure the parameters and click Next. The following table describes the parameters.
Parameter
Description
Risk Identification
Select Business Security and enter relevant information. For more information, see Risk identification.
The feature helps block requests from abnormal mobile phone numbers based on Fraud Detection. You are charged based on rule hits.
Bot Characteristic Detection
Detection rules
Protection action
You can select Monitor, Block, or Strict Slider CAPTCHA Verification as the action that you want WAF to perform on the requests that match the rule specified below Bot Characteristic Detection.
Advanced protection
Click Advanced Protection and configure the following parameters:
Bot Behavior Detection
If you select Intelligent Protection, you must select Monitor, Slider CAPTCHA, Strict Slider CAPTCHA Verification, or Add Tag as the action that you want WAF to perform on detected bot requests. If you select Add Tag, you must configure the Header Name and Header Content parameters.
After you select Intelligent Protection, the intelligent protection engine analyzes access traffic and performs machine learning. Then, a blacklist or a protection rule is generated based on the analysis results and learned patterns.
Throttling
You can configure custom throttling conditions to filter out crawler requests that are frequently initiated. This helps prevent HTTP flood attacks.
Bot Threat Intelligence
In the Configure Effective Scope step, configure the parameters and click Next. The following table describes the parameters.
Parameter
Description
Apply To
Select the protected objects or protected object groups to which you want to apply the protection template in the Objects to Select section and click the icon to move the protected objects or protected object groups to the Selected Objects section.
Effective Time and Canary Release
You must specify validity periods and configure canary release settings for the protection rules. If you do not specify validity periods or configure canary release settings, canary release is disabled for the rules and the rules are permanently valid.
Find the rule whose configurations you want to modify and click Edit in the Actions column.
Configure canary release settings and specify validity periods.
Canary Release: You can turn on Canary Release to apply the rule to a specific proportion of objects.
If you turn on Canary Release, you must configure the Dimension and Canary Release Ratio parameters. Valid values of the Dimension parameter: IP Address, Custom Header, Custom Header, Custom Cookie, and Session.
Effective Mode
Permanently Effective (default): If you select Permanently Effective, the protection rule is permanently in effect.
Fixed Schedule: You can specify a time zone and a period of time during which you want the template to be in effect.
Recurring Schedule: You can specify a time zone, days of the week, and a period of time in a day. The template is in effect during the same period of time in a day on the same days of the week.
You can select multiple rules to specify validity periods and configure canary release settings for the rules at the same time.
In the Verify Protection Effect step, test the anti-crawler rule.
To prevent false positives that are caused by improper rule configurations or compatibility issues, we recommend that you verify the protection effect of the rule before you publish an anti-crawler rule If the configurations are correct, click Skip.
Test steps:
By default, a newly created template is enabled. On the Scenario-specific Protection tab, you can perform the following operations:
Click a template card to view the rule information about the template.
Copy, Edit, or Delete a template.
Turn on or turn off the switch to enable or disable a template.
View the settings of the Action and Protected Object/Group parameters of a template.
Create a basic protection rule
You can configure basic protection rules to defend against medium- and low-level crawlers for your services. The bot management module does not provide a default basic protection rule template. Before you can enable the basic protection feature provided by the bot management module, you must create a basic protection rule template.
Log on to the WAF 3.0 console. In the top navigation bar, select the resource group and region of the WAF instance. You can select Chinese Mainland or Outside Chinese Mainland.
In the left-side navigation pane, choose .
On the Basic Protection tab, click Create Template.
In the Create Template - Bot Management panel, configure the parameters and click OK. The following table describes the parameters.
Parameter
Description
Template Name
Enter a name for the template.
The name of the template must be 1 to 255 characters in length and can contain letters, digits, periods (.), underscores (_), and hyphens (-).
Template Description
Enter a description for the template.
Action
Specify an action that you want WAF to perform on the requests that match the rule. Valid values: Block and Monitor.
Advanced Settings
Canary Release: You can turn on Canary Release to apply the rule to a specific proportion of objects.
If you turn on Canary Release, you must configure the Dimension and Canary Release Ratio parameters. Valid values of the Dimension parameter: IP Address, Custom Header, Custom Header, Custom Cookie, and Session.
Effective Mode
Permanently Effective (default): If you select Permanently Effective, the protection rule is permanently in effect.
Fixed Schedule: You can specify a time zone and a period of time during which you want the template to be in effect.
Recurring Schedule: You can specify a time zone, days of the week, and a period of time in a day. The template is in effect during the same period of time in a day on the same days of the week.
Apply To
Select the protected objects and protected object groups to which you want to apply the template.
You can apply only one template of a protection module to a protected object or protected object group. For more information about how to add protected objects and create protected object groups, see Configure protected objects and protected object groups.
By default, a newly created template is enabled. On the Basic Protection tab, you can perform the following operations:
View the IDs of the rules that are included in a template.
NoteA basic protection template includes two whitelist rules and one access control or HTTP flood protection rule. You can view the protection performance of the rules on the Security Reports page by using the rule IDs. For more information, see Security reports.
Copy, Edit, or Delete a template.
Turn on or turn off the switch to enable or disable a template.
View the settings of the Action and Protected Object/Group parameters of a template.
FAQ
If an error occurs in the Verify Protection Effect step, refer to the following table to fix the error.
Error | Cause | Solution |
No valid test requests are detected. See WAF documentation or contact us to identify the possible causes. | The test request failed to be sent or is not sent to WAF. | Make sure that the test request is sent to the IP address that maps the CNAME provided by WAF. |
The header fields in the test request do not match the header fields that you configured for the Traffic Characteristics parameter in the anti-crawler rule. | Modify the Traffic Characteristics parameter in the anti-crawler rule. | |
The source IP address of the test request is different from the public IP address that you specified in the anti-crawler rule. | Make sure that you use the correct public IP address. We recommend that you use Alibaba Network Diagnose Tool to obtain your public IP address. | |
The test requests failed the verification. See WAF documentation or contact us to identify the possible causes. | No real user access is simulated. For example, the debugging mode or automation tools are used. | Simulate a real user to access your website or app during the test. |
An incorrect service type is selected. For example, Websites is selected when you configure an anti-crawler rule for apps. | Modify the value of the Service Type parameter. | |
An intermediate domain name is used, but an incorrect intermediate domain name is selected in the anti-crawler rule. | Select Use Intermediate Domain Name. Then, select the correct intermediate domain name from the drop-down list. | |
Compatibility issues occur in the frontend. | Submit a ticket to contact us. | |
No verification is triggered. See WAF documentation or contact us to identify the possible causes. | No test rules are generated. | Perform the test several times until a test rule is generated. |
No valid test requests are detected or blocked. See WAF documentation or contact us to identify the possible causes. | The test request failed to be sent or is not sent to WAF. | Make sure that the test request is sent to the IP address that maps the CNAME provided by WAF. |
The header fields in the test request do not match the header fields that you configured for the Traffic Characteristics parameter in the anti-crawler rule. | Modify the Traffic Characteristics parameter in the anti-crawler rule. | |
The source IP address of the test request is different from the public IP address that you specified in the anti-crawler rule. | Make sure that you use the correct public IP address. We recommend that you use Alibaba Network Diagnose Tool to obtain your public IP address. |
What to do next
On the Security Reports page, you can query the protection details of the protection rules that you configured. For more information, see Security reports.