All Products
Search

This Product

    Alibaba Cloud Service Mesh:Pods on some nodes cannot access the ingress gateway CLB IP address

    Document Center

    Alibaba Cloud Service Mesh:Pods on some nodes cannot access the ingress gateway CLB IP address

    Last Updated:Mar 10, 2026

    When a Kubernetes cluster on the data plane of a Service Mesh (ASM) instance uses a Classic Load Balancer (CLB) with externalTrafficPolicy: Local for the ingress gateway, pods on some nodes may fail to reach the CLB IP address. This topic explains the cause and provides three solutions.

    Symptom

    A Kubernetes cluster is added to an ASM instance. A CLB instance with externalTrafficPolicy set to Local is configured for the ingress gateway. You observe the following behavior:

    • Pods on some nodes can access the CLB IP address of the ingress gateway.

    • Pods on other nodes cannot access the same CLB IP address.

    Cause

    When externalTrafficPolicy is Local, kube-proxy programs iptables or IP Virtual Server (IPVS) forwarding rules only on nodes that run backend pods of the ingress gateway Service. Nodes without ingress gateway pods have no forwarding rules, so traffic to the CLB IP address is dropped.

    The CLB IP address is treated as an external IP of the Service. Rather than routing traffic out to the load balancer and back, kube-proxy short-circuits requests through local iptables or IPVS rules. With externalTrafficPolicy: Local, only nodes with local endpoints get these rules.

    For the upstream Kubernetes discussion, see Why kube-proxy add external-lb's address to node local iptables rule?.

    Verify the issue

    Before you apply a fix, confirm that this root cause applies to your situation.

    1. Check which nodes run ingress gateway pods: Note the NODE column. Pods on these nodes can access the CLB IP. Pods on other nodes cannot.

         kubectl get pods -n istio-system -l app=istio-ingressgateway -o wide

    2. Check the endpoints of the ingress gateway Service: The listed IP addresses correspond to ingress gateway pods. Nodes without these pods lack the local forwarding rules needed to reach the CLB IP.

         kubectl get endpoints -n istio-system istio-ingressgateway

    3. (Optional) On a node where access fails, verify that no iptables rules exist for the CLB IP: Replace <CLB_IP> with the actual CLB IP address. If no output is returned, the node has no forwarding rule for that IP, which confirms the issue.

         iptables-save | grep <CLB_IP>

    Solutions

    Choose a solution based on your requirements:

    SolutionSource IP preservedRequires ENIComplexity
    Access the ingress gateway by cluster IP or Service nameYesNoLow
    Set externalTrafficPolicy to ClusterNoNoLow
    Use Cluster with ENI direct connectionYesYesMedium

    Solution 1: Access the ingress gateway by cluster IP or Service name (recommended)

    Instead of using the CLB IP address from inside the cluster, use the cluster IP address or the Kubernetes Service name to reach the ingress gateway:

    istio-ingressgateway.istio-system

    This approach works on all nodes regardless of where ingress gateway pods run, preserves source IP addresses, and requires no configuration changes.

    Solution 2: Set externalTrafficPolicy to Cluster

    Change externalTrafficPolicy from Local to Cluster. This tells kube-proxy to program forwarding rules on all nodes, not just the ones running ingress gateway pods.

    Trade-off: Source IP addresses are not preserved. All requests appear to originate from the node IP where kube-proxy forwards the traffic.

    Update the IstioGateway custom resource:

    apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  name: ingressgateway
  namespace: istio-system
  ....
spec:
  externalTrafficPolicy: Cluster
....

    For field details, see CRD fields for a gateway.

    Solution 3: Use Cluster with ENI direct connection

    If your cluster uses Terway elastic network interfaces (ENIs) or runs in inclusive ENI mode, combine externalTrafficPolicy: Cluster with the service.beta.kubernetes.io/backend-type: eni annotation. This restores source IP preservation through direct ENI connection while keeping the CLB IP accessible from all nodes.

    Prerequisites: The cluster must use Terway as the CNI plugin, with ENI or inclusive ENI mode enabled.

    Update the IstioGateway custom resource:

    apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  name: ingressgateway
  namespace: istio-system
  ....
spec:
  externalTrafficPolicy: Cluster
  maxReplicas: 5
  minReplicas: 2
  ports:
    - name: status-port
      port: 15020
      targetPort: 15020
    - name: http2
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443
    - name: tls
      port: 15443
      targetPort: 15443
  replicaCount: 2
  resources:
    limits:
      cpu: '2'
      memory: 2G
    requests:
      cpu: 200m
      memory: 256Mi
  runAsRoot: false
  serviceAnnotations:
    service.beta.kubernetes.io/backend-type: eni
  serviceType: LoadBalancer

    The serviceAnnotations: service.beta.kubernetes.io/backend-type: eni field routes traffic directly through ENIs and preserves source IP addresses.

    For field details, see CRD fields for a gateway.

    References