Container Service for Kubernetes:Install the NGINX Ingress controller in high-load scenarios

Deployment notes

When you deploy the NGINX Ingress controller in a high-load scenario, take note of the following items:

Step 1: Add nodes

Create a node pool in the ACK cluster and add two nodes to the node pool.

Configure the node pool based on the following description. For more information, see Create and manage a node pool.

• Set Operating System to Alibaba Cloud Linux 3.

• Set Node Labels and Taints.

  • Add a taint. Set Key to ingress-pod, set Value to yes, and set Effect to NoExecute.

  • Add a node label. Set Key to ingress-pod and set Value to yes.

• Set CPU Policy to Static.

Step 2: Update the NGINX Ingress controller

Run the kubectl edit deploy nginx-ingress-controller -n kube-system command to edit the configuration file of the NGINX Ingress controller based on the following description.

1. Delete the pod anti-affinity settings.

   podAntiAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
     - labelSelector:
         matchExpressions:
         - key: app
         operator: In
         values:
         - ingress-nginx
      topologyKey: kubernetes.io/hostname

2. Set the requests and limits parameters for the init container.

   resources:
     limits:
       cpu: 100m
       memory: 70Mi
     requests:
       cpu: 100m
       memory: 70Mi

3. Set the requests and limits parameters of the nginx-ingress-controller containers to 30 Core and 40 GiB.

   resources:
     limits:
       cpu: "30"
       memory: 40Gi
     requests:
       cpu: "30"
       memory: 40Gi

4. Set node affinity settings and tolerations.

   nodeSelector:
     ingress-pod: "yes"
   tolerations:
   - effect: NoExecute
     key: ingress-pod
     operator: Equal
     value: "yes"

5. Set the number of the replicated pods of the NGINX Ingress controller to the number of newly added nodes.

6. Disable metric collection by adding --enable-metrics=false to the startup parameters.

   Note

   If you do not need metrics, we recommend that you disable metric collection. If the version of the NGINX Ingress controller is v1.9.3 or later, you can also configure startup parameters --exclude-socket-metrics to exclude metrics. Example: nginx_ingress_controller_ingress_upstream_latency_seconds. This mitigates the impact on the system performance.

   containers:
   - args:
     - /nginx-ingress-controller
     - --configmap=$(POD_NAMESPACE)/nginx-configuration
     - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
     - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
     - --annotations-prefix=nginx.ingress.kubernetes.io
     - --publish-service=$(POD_NAMESPACE)/nginx-ingress-lb
     - --enable-metrics=false
     - --v=1

Step 3: Update the ConfigMap of the NGINX Ingress controller

1. Run the kubectl edit cm -n kube-system nginx-configuration command to edit the ConfigMap of the NGINX Ingress controller. Modify the ConfigMap based on the following template:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nginx-configuration
     namespace: kube-system
   data:
     allow-backend-server-header: "true"
     enable-underscores-in-headers: "true"
     generate-request-id: "true"
     ignore-invalid-headers: "true"
     log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]
     max-worker-connections: "65536"
     proxy-body-size: 20m
     proxy-connect-timeout: "3"
     proxy-read-timeout: "5"
     proxy-send-timeout: "5"
     reuse-port: "true"
     server-tokens: "false"
     ssl-redirect: "false"
     upstream-keepalive-timeout: "900"
     worker-processes: "28" # Specify the number of NGINX processes to start. You can set this value based on your node specifications. We recommend that you set the value to the number of CPU cores - 2. 
     worker-cpu-affinity: auto
     upstream-keepalive-connections: "300"
     upstream-keepalive-requests: "1000" # Specify the maximum number of requests through a persistent connection. 
     keep-alive: "900"
     keep-alive-requests: "10000"

2. Import logs to a file and set up log rotation.

   By default, logs are written into the /dev/stdout file. When the cluster receives a large number of requests, the CPU usage is high. In this case, we recommend that you write logs into the /dev/stdout file and set up log rotation.

   1. Use SSH to log on to the ECS instance where the ingress-controller pods are deployed. For more information, see Connect to a Linux instance by using an SSH key pair.

   2. Add the following content to the end of the /etc/crontab file:

      */15 * * * *  root /root/nginx-log-rotate.sh

      Note

      In this example, the logs are rotated every 15 minutes. You can change the interval based on your requirements.

   3. Create a file named nginx-log-rotate.sh in the /root directory.

      Docker clusters

      #!/bin/bash
      # Specify the maximum number of log files that are retained. You can change the number based on your requirements. 
      keep_log_num=5
      ingress_nginx_container_ids=$(docker ps | grep nginx-ingress-controller | grep -v pause | awk '{print $1}')
      if [[ -z "$ingress_nginx_container_ids" ]]; then
       echo "error: failed to get ingress nginx container ids"
       exit 1
      fi
      # Make the NGINX Ingress controller pods sleep for a time period of a random length between 5 and 10 seconds. 
      sleep $(( RANDOM % (10 - 5 + 1 ) + 5 ))
      for id in $ingress_nginx_container_ids; do
       docker exec $id bash -c "cd /var/log/nginx; if [[ \$(ls access.log-* | wc -l) -gt $keep_log_num ]]; then rm -f \$(ls -t access.log-* | tail -1); fi ; mv access.log access.log-\$(date +%F:%T) ; kill -USR1 \$(cat /tmp/nginx/nginx.pid)"
      done	
      	

      containerd clusters

      #!/bin/bash
      # Specify the maximum number of log files that are retained. You can change the number based on your requirements. 
      keep_log_num=5
      ingress_nginx_container_ids=$(crictl ps | grep nginx-ingress-controller | grep -v pause | awk '{print $1}')
      if [[ -z "$ingress_nginx_container_ids" ]]; then
       echo "error: failed to get ingress nginx container ids"
       exit 1
      fi
      # Make the NGINX Ingress controller pods sleep for a time period of a random length between 5 and 10 seconds. 
      sleep $(( RANDOM % (10 - 5 + 1 ) + 5 ))
      for id in $ingress_nginx_container_ids; do
       crictl exec $id bash -c "cd /var/log/nginx; if [[ \$(ls access.log-* | wc -l) -gt $keep_log_num ]]; then rm -f \$(ls -t access.log-* | tail -1); fi ; mv access.log access.log-\$(date +%F:%T) ; kill -USR1 \$(cat /tmp/nginx/nginx.pid)"
      done	
      	

   4. Run the following command to make the nginx-log-rotate.sh file executable:

      chmod 755 /root/nginx-log-rotate.sh

Step 4: Optimize configurations

This step shows how to optimize kernel parameters, application configurations, compressions settings, and HTTPS performance.

• Optimize kernel parameters

  Before you optimize kernel parameters, familiarize yourself with the parameters and exercise caution.

  • Modify the TIME_WAIT setting

    To optimize the performance of NGINX Ingresses, you can modify OS parameters to enable the reuse of TIME_WAIT ports and reduce the timeout period of connections in the FIN_WAIT2 and TIME_WAIT states.

    • net.ipv4.tcp_fin_timeout=15: reduces the timeout period of connections in the FIN_WAIT2 state. This way, connections are released within a shorter period of time.

    • net.netfilter.nf_conntrack_tcp_timeout_time_wait=30: reduces the time period in which a connection remains in the TIME-WAIT state. This way, connections are released within a shorter period of time.

  • Configuration steps

    1. Run the following command to modify the configurations of the Deployment created for the NGINX Ingress controller:

       kubectl edit deployments -n kube-system nginx-ingress-controller

    2. Add the sysctl -w net.ipv4.tcp_fin_timeout=15 and sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 settings to the init container section. Then, save the changes and exit.

       initContainers:
             - command:   # Add the sysctl -w net.ipv4.tcp_fin_timeout=15 and sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 settings. 
               - /bin/sh
               - -c
               - |
                 if [ "$POD_IP" != "$HOST_IP" ]; then
                 mount -o remount rw /proc/sys
                 sysctl -w net.core.somaxconn=65535
                 sysctl -w net.ipv4.ip_local_port_range="1024 65535"
                 sysctl -w kernel.core_uses_pid=0
                 sysctl -w net.ipv4.tcp_fin_timeout=15 
                 sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30   
                 fi

• Optimize the application configurations

  Perform the following steps to optimize the application configurations of the NGINX Ingress controller:

  1. Run the following command to modify the ConfigMap of the NGINX Ingress controller:

      kubectl edit cm -n kube-system nginx-configuration

  2. Modify the parameters in the ConfigMap based on the parameter description in the following table.

     Item	Parameter	Description
     Keep-alive settings for downstream connections	keep-alive: "60"	The timeout period of keep-alive connections. Unit: seconds.
     	keep-alive-requests: "10000"	The maximum number of requests that can be sent over a keep-alive connection.
     Keep-alive settings for upstream connections	upstream-keepalive-connections: "1000"	The maximum number of keep-alive connections.
     	upstream-keepalive-requests: "2147483647"	The maximum number of requests that can be sent over a keep-alive connection.
     	upstream-keepalive-time: 1h	The maximum period of time during which requests can be processed over a keep-alive connection.
     	upstream-keepalive-timeout: "150"	The timeout period of an idle upstream keep-alive connection. Unit: seconds.
     Connection upper limit of each work process	max-worker-connections: "65536"	The maximum number of simultaneous connections that can be opened by a worker process.
     Timeout settings Note You can modify the parameter values based on your business requirements.	proxy-connect-timeout: "3"	The timeout period for establishing a connection. Unit: seconds.
     	proxy-read-timeout: "5"	The timeout period for reading data. Unit: seconds.
     	proxy-send-timeout: "5"	The timeout period for sending data. Unit: seconds.
     Retry settings Note When errors occur on backend services, multiple retries may lead to excessive requests. This may increase the load on the backend services or even cause a service avalanche. For more information, see ingress-nginx official documentation.	proxy-next-upstream-tries: "3"	The number of retries after a request fails to be sent. Default value: 3. The default value includes the original request and two retries.
     	proxy-next-upstream: "off"	The conditions in which retries are triggered. To disable retries, set the value to off.
     	proxy-next-upstream-timeout	The timeout period of a request retry. Unit: seconds. You can modify the value based on your business requirements.

• Enable Brotli compression

  Although data compression consumes additional CPU time, compressed data packets reduce bandwidth usage, which increases network throughput. Brotli is an open source compression algorithm developed by Google. Compared with the commonly used gzip compression algorithm, the compression rate of Brotli is 20% to 30% higher. gzip is the default compression algorithm used by the NGINX Ingress controller. To enable Brotli compression for ingress-nginx, you must configure the following parameters. For more information, see ingress-nginx official documentation.

  • enable-brotli: specifies whether to enable Brotli. Valid values: true and false.

  • brotli-level: the compression level. Valid values: 1 to 11. Default value: 4. A higher compression level requires a higher amount of CPU resources.

  • brotli-types: the Multipurpose Internet Mail Extensions (MIME) types that are compressed on the fly by Brotli.

  Run the following command to enable Brotli compression:

   kubectl edit cm -n kube-system nginx-configuration

  Sample configurations:

  enable-brotli: "true"
  brotli-level: "6"
  brotli-types: "text/xml image/svg+xml application/x-font-ttf image/vnd.microsoft.icon application/x-font-opentype application/json font/eot application/vnd.ms-fontobject application/javascript font/otf application/xml application/xhtml+xml text/javascript application/x-javascript text/plain application/x-font-truetype application/xml+rss image/x-icon font/opentype text/css image/x-win-bitmap"

• Optimize HTTPS performance

  To improve HTTPS performance, run the kubectl edit cm -n kube-system nginx-configuration command to modify the nginx-configuration ConfigMap and configure the following settings in the ConfigMap: SSL session caching, Online Certificate Status Protocol (OCSP) stapling, support for TLS 1.3 early data, and cipher suite priorities.

  • SSL session caching and timeout

    Set the size of the shared SSL session cache and the time period in which a client can reuse the session parameters stored in a cache. This helps you reduce the overheads of SSL handshakes.

    • ConfigMap configurations:

      ssl-session-cache-size: "10m"
      ssl-session-timeout: "10m"

    • The nginx.conf configurations at the NGINX side. You can modify the configurations based on your business requirements.

      ssl_session_cache shared:SSL:120m;   # 1 MB of cache can store about 4,000 sessions. 
      ssl_session_timeout 1h;              # The timeout period of sessions is 1 hour.

  • Enable OCSP stapling

    OCSP stapling can reduce the time required by client certificate verification.

    enable-ocsp: "true"

  • Support for TLS 1.3 early data (0-RTT)

    The TLS 1.3 early data feature, also known as zero round-trip time (0-RTT), allows clients to send data before the handshake is completed. This helps reduce the response time.

    ssl-early-data: "true"
    ssl-protocols: "TLSv1.3"

  • Modify the cipher suite priorities (optimized)

    You can modify the cipher suite priorities to reduce network latency. ACK has optimized cipher suite priorities for the NGINX Ingress controller configurations.

    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
    ssl_prefer_server_ciphers on;    # The cipher configurations on the server side take precedence.