Problem description

After my service is added to an Anti-DDoS Pro or Anti-DDoS Premium instance, issues such as slow response, high latency, and access failures occur.

Causes

  1. Collect the affected IP addresses and test connectivity by using tools such as traceroute or MTR to identify the cause.
    Note

    The MTR tool is used in this step to describe how to check latency in network links. 

    mtr --no-dns [$IP]
    Note [$IP] indicates the affected IP address that you collect.
  2. Check whether the Host value of the node is the IP address of your origin server or the IP address of the Anti-DDoS Pro or Anti-DDoS Premium instance.

Solutions

If you want to immediately access your service, we recommend that you bypass Anti-DDoS Pro or Anti-DDoS Premium and directly access the origin server. This ensures normal access to your service. Then, troubleshoot the issue based on the following sections.

Troubleshooting for the exceptions on origin servers

Troubleshoot the exceptions based on the type of your origin server.

Type of origin server Troubleshooting
Server Load Balancer (SLB) instance
  1. Use the TCPing tool to ping the IP address and port of the SLB instance and check whether an exception occurs. For more information, see Troubleshooting for traffic scrubbing events.
  2. Check the status of the SLB instance. For example, check whether the number of connections to the instance exceeds the Max Connection specification of the instance.
  3. Check whether access control policies are configured.

    If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.

  4. Check whether the security software or IP blocking policy on the backend server of the SLB instance denies access from the back-to-origin CIDR blocks of your Anti-DDoS Pro or Anti-DDoS Premium instance by mistake. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
    Note If a backend server is associated with an SLB instance, the backend server cannot identify the originating IP address of the client because Layer 7 load balancing is not used. The backend server considers all requests as being initiated from the back-to-origin CIDR blocks of an Anti-DDoS Pro or Anti-DDoS Premium instance. As a result, the backend server considers that each back-to-origin CIDR block initiates a large number of requests. In this case, the back-to-origin CIDR blocks of an Anti-DDoS Pro or Anti-DDoS Premium instance may be blocked by the security software by mistake. You must allow these IP addresses on the origin server.
  5. Check whether the IP address of the SLB instance is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, we recommend that you use another SLB instance. Otherwise, attackers may bypass Anti-DDoS Pro or Anti-DDoS Premium to attack the origin server.
Elastic Compute Service (ECS) instance
  1. Use the TCPing tool to ping the IP address and port of the ECS instance and check whether an exception occurs based on logs. For more information, see Troubleshooting for traffic scrubbing events.
  2. Check whether exceptions occur on the ECS instance. Exceptions include high CPU utilization, slow processing of database requests, and high bandwidth of outbound traffic. You can also check blackhole filtering events and traffic scrubbing events.
  3. Check whether access control policies, such as security groups and security software, are configured. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
  4. Check whether a security group that allows requests from the originating IP addresses of your non-website service to the ECS instance is added.
  5. Check whether the IP address of the ECS instance is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, attackers may bypass Anti-DDoS Pro or Anti-DDoS Premium to attack the origin server. In this case, we recommend that you change the IP address of the ECS instance. For more information, see Change the public IP address of an ECS origin server.
Server not deployed on Alibaba Cloud
  1. Use the TCPing tool to ping the IP address and port of the server and check whether an exception occurs based on logs. For more information, see Troubleshooting for traffic scrubbing events.
  2. Check whether exceptions occur on the server. Exceptions include high CPU utilization, slow processing of database requests, and high bandwidth of outbound traffic.
  3. Check whether access control policies, such as a blacklist, a whitelist, and security software, are configured. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
  4. Check whether the IP address of the server is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, we recommend that you change the IP address of the server. Otherwise, attackers may bypass Anti-DDoS Pro or Anti-DDoS Premium to attack the origin server.

Troubleshooting for an Anti-DDoS Pro and Anti-DDoS Premium instance

View the status of the instance on the Instances page of the Anti-DDoS Pro console. Then, determine the handling methods based on the status.
  • Scrubbing

    If network traffic exceeds the traffic scrubbing threshold, the instance scrubs traffic. Traffic scrubbing events may cause slow response or latency.

  • Blackholing

    If volumetric attacks are launched to your server and blackholing filtering is triggered for your server, the server is accessible only from Alibaba Cloud and services in the same region as the server. Traffic from other sources is denied.

Troubleshooting for traffic scrubbing events
The following figure shows that traffic scrubbing events occurred and caused the exception. You can use the TCPing tool to ping the affected and unaffected ports and check whether latency or packet loss occurs.Instance status
Identify the cause and resolve the issue based on the following table.
Latency and packet loss on affected ports

Latency and packet loss on unaffected ports

Solution
Yes No The traffic scrubbing policy does not cause high latency and packet loss.
We recommend that you check the status and attack mitigation capabilities of the backend server. If the backend server does not have sufficient capabilities to mitigate attacks, adjust the mitigation policies of your Anti-DDoS Pro or Anti-DDoS Premium instance to harden the security of the server. You can analyze the attack mitigation capabilities of your server and adjust the mitigation policies based on the following details:
  • Access statistics
  • Service interaction process
  • Performance data
Yes Yes The traffic scrubbing policy causes latency and packet loss.
No No The traffic scrubbing policy does not cause latency or packet loss.
No Yes This case does not exist.
Troubleshooting for blackhole filtering events
  1. The following figure shows that blackhole filtering events occurred. In this case, check the IP address that is in the blackholing state and check whether the affected service used the IP address. Status: Blackholing
  2. We recommend that you deactivate blackhole filtering. Each Alibaba Cloud account can deactivate blackhole filtering up to five times per day. For more information, see Deactivate blackhole filtering.

Additional information

This section describes how to use TCPing to check the port status, detect TCP latency, and view connection information by using TCP connections. Click TCPing to download the TCPing tool.
  • Use TCPing in Windows
    Copy the TCPing tool to the specified directory in Windows and run the tcping [$Domain_Name] [$Port] command.
    Note
    • [$Domain_Name] indicates the domain name or IP address for which you want to test connectivity.
    • [$Port] indicates the port for which you want to test connectivity.
    The following output is returned: 
    Probing 192.168.XX.XX:80/tcp - Port is open - time=19.550ms
Probing 140.XXX.XXX.8:80/tcp - Port is open - time=8.761ms
Probing 192.168.XX.XX:80/tcp - Port is open - time=10.899ms
Probing 192.168.XX.XX:80/tcp - Port is open - time=13.013ms

Ping statistics for 192.168.XX.XX:80
     4 probes sent.
     4 successful, 0 failed.
Approximate trip times in milli-seconds:
     Minimum = 8.761ms, Maximum = 19.550ms, Average = 13.056ms
  • Use TCPing in Linux
    1. Run the following commands in sequence to install TCPing: 
      tar zxvf tcping-1.3.5.tar.gz
cd tcping-1.3.5
make tcping.linux
    2. Run the following command to test connectivity: 
      for ((i=0; i<10; ++i)) ; do ./tcping www.example.com 80;done
      The following output is returned: 
      www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.

Applicable scope

Anti-DDoS Pro and Anti-DDoS Premium