Configure redirection of HTTP or TCP health check requests for applications in an ASM instance - Alibaba Cloud Service Mesh

When a sidecar proxy is injected into a pod in Service Mesh (ASM), it intercepts all inbound traffic. This breaks Kubernetes health probes in two ways:

Probe type	Problem	When it occurs
HTTP	The kubelet cannot present an ASM-issued TLS certificate, so the sidecar rejects probe requests. Probes always fail.	Strict mTLS enabled
TCP	The sidecar listens on all pod ports, so TCP connection checks always succeed -- even when the application is not listening. Probes never fail.	Any configuration with sidecar injection

Add the annotation sidecar.istio.io/rewriteAppHTTPProbers: "true" to your pod template. ASM rewrites probes to route through the pilot-agent on port 15020, which bypasses the sidecar proxy. This restores accurate health reporting for HTTP and TCP probes.

Note

If mTLS is not enabled, HTTP probes work without redirection. TCP probe redirection is useful in all configurations because it restores accurate health reporting.

How probe redirection works

ASM modifies the pod spec at sidecar injection time:

Replaces the original probe port and path with port 15020 and path /app-health/<container-name>/<probe-type> (for example, /app-health/nginx/readyz).
Serializes the original probe configuration as JSON and stores it in the ISTIO_KUBE_APP_PROBERS environment variable on the sidecar container.
The pilot-agent in the sidecar listens on port 15020, receives probe requests from the kubelet, and forwards them to the application container using the configuration in ISTIO_KUBE_APP_PROBERS.

Port 15020 is reserved for ASM observability and is never intercepted by the sidecar proxy. This means probe requests bypass mTLS requirements.

TCP probe conversion: The original TCP probe is converted to an HTTP probe targeting port 15020. The pilot-agent checks whether the application listens on the original TCP port. If the port is not open, the pilot-agent returns HTTP 500, causing the probe to fail correctly.

Probe types and redirection behavior:

Probe type	Redirection behavior	Notes
HTTP (`httpGet`)	Rewritten to port 15020. Pilot-agent forwards the HTTP request to the application.	Fixes mTLS-related probe failures
TCP (`tcpSocket`)	Converted to HTTP probe on port 15020. Pilot-agent checks whether the target port is open.	Fixes false-positive probe results

Traffic statistics: By default, health check requests appear in the ASM mesh topology. Probe redirection routes requests through port 15020, which is excluded from traffic tracking. This produces more accurate traffic statistics.

Redirect HTTP health check probes

This example deploys an NGINX application with an HTTP readiness probe, enables strict mTLS to trigger probe failure, and then applies the redirection annotation to restore correct probe behavior.

Step 1: Enable strict mTLS for the ASM instance

Log on to the ASM console.
In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, find your ASM instance. Click its name or click Manage in the Actions column.
In the left-side navigation pane of the details page, choose Mesh Security Center > PeerAuthentication.
Select a namespace from the Namespace drop-down list and click Configure Global mTLS Mode.
Set mTLS Mode (Namespace-wide) to STRICT -Strictly Enforce mTLS and click Create.

Step 2: Deploy an NGINX application with an HTTP readiness probe

Before you begin, make sure that you have connected to the Kubernetes cluster using kubectl. For instructions, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Create a file named http-liveness.yaml:

Show the http-liveness.yaml file

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 1
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            readinessProbe:
              httpGet:
                path: /index.html
                port: 80
                httpHeaders:
                - name: X-Custom-Header
                  value: hello
              initialDelaySeconds: 5
              periodSeconds: 3

Apply the deployment:
```
    kubectl apply -f http-liveness.yaml
```

Verify that the probe fails. Get the pod name and check its events: Expected output includes an event similar to: The probe fails because the kubelet cannot present a valid mTLS certificate.

    kubectl get pod | grep nginx
    kubectl describe pod <pod-name>

    Warning  Unhealthy  45s  kubelet  Readiness probe failed: Get "http://172.23.64.22:80/index.html": read tcp 172.23.64.1:54130->172.23.64.22:80: read: connection reset by peer

Step 3: Add the probe redirection annotation

Edit http-liveness.yaml and add the annotation under template.metadata:

    template:
      metadata:
        labels:
          app: nginx
        annotations:
          sidecar.istio.io/rewriteAppHTTPProbers: "true"

Full http-liveness.yaml with annotation

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 1
      template:
        metadata:
          labels:
            app: nginx
          annotations:
            sidecar.istio.io/rewriteAppHTTPProbers: "true"
        spec:
          containers:
          - name: nginx
            image: nginx
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            readinessProbe:
              httpGet:
                path: /index.html
                port: 80
                httpHeaders:
                - name: X-Custom-Header
                  value: hello
              initialDelaySeconds: 5
              periodSeconds: 3

Redeploy:
```
    kubectl apply -f http-liveness.yaml
```

Step 4: Verify the redirection

Check that the pod is ready and no probe failure events exist: The pod should show Ready status with no Unhealthy events.
```
    kubectl get pod | grep nginx
    kubectl describe pod <pod-name>
```

Confirm the probe rewrite by inspecting the pod spec: Expected output: The probe now targets port 15020 with the path /app-health/nginx/readyz instead of the original port 80 and path /index.html.

    kubectl get pod <pod-name> -o json | jq '.spec.containers[] | select(.name=="nginx") | .readinessProbe.httpGet'

    {
      "httpHeaders": [
        {
          "name": "X-Custom-Header",
          "value": "hello"
        }
      ],
      "path": "/app-health/nginx/readyz",
      "port": 15020,
      "scheme": "HTTP"
    }

Verify that the ISTIO_KUBE_APP_PROBERS environment variable is set on the sidecar container: This variable stores the original probe configuration as JSON. The pilot-agent uses it to forward probe requests to the application.
```
    kubectl get pod <pod-name> -o json | jq '.spec.containers[] | select(.name=="istio-proxy") | .env[] | select(.name=="ISTIO_KUBE_APP_PROBERS")'
```

Redirect TCP health check probes

This example deploys an NGINX application with a TCP readiness probe that targets an invalid port (2940). Without redirection, the sidecar proxy causes this probe to falsely succeed. After adding the annotation, the probe correctly reports failure.

Step 1: Deploy an NGINX application with a TCP readiness probe

Before you begin, make sure that you have connected to the Kubernetes cluster using kubectl. For instructions, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Create a file named tcp-liveness.yaml: Port 2940 is intentionally invalid. NGINX does not listen on it, so this probe should fail. However, because the sidecar proxy listens on all ports, the probe incorrectly succeeds.

Show the tcp-liveness.yaml file

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 1
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            readinessProbe:
              tcpSocket:
                port: 2940
              initialDelaySeconds: 5
              periodSeconds: 3

Apply the deployment:
```
    kubectl apply -f tcp-liveness.yaml
```
Check pod events: No failure events appear and the pod shows as ready, even though the application does not listen on port 2940.
```
    kubectl get pod | grep nginx
    kubectl describe pod <pod-name>
```

Step 2: Add the probe redirection annotation

Edit tcp-liveness.yaml and add the annotation under template.metadata:

    template:
      metadata:
        labels:
          app: nginx
        annotations:
          sidecar.istio.io/rewriteAppHTTPProbers: "true"

Full tcp-liveness.yaml with annotation

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 1
      template:
        metadata:
          labels:
            app: nginx
          annotations:
            sidecar.istio.io/rewriteAppHTTPProbers: "true"
        spec:
          containers:
          - name: nginx
            image: nginx
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            readinessProbe:
              tcpSocket:
                port: 2940
              initialDelaySeconds: 5
              periodSeconds: 3

Redeploy:
```
    kubectl apply -f tcp-liveness.yaml
```

Step 3: Verify the redirection

Check pod events: Expected output includes: The probe correctly fails. The pilot-agent detected that port 2940 is not open and returned HTTP 500.

    kubectl get pod | grep nginx
    kubectl describe pod <pod-name>

    Warning  Unhealthy  45s  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500

Confirm the rewritten probe: Expected output: The original TCP probe has been converted to an HTTP probe targeting the pilot-agent on port 15020.

    kubectl get pod <pod-name> -o json | jq '.spec.containers[] | select(.name=="nginx") | .readinessProbe.httpGet'

    {
      "path": "/app-health/nginx/readyz",
      "port": 15020,
      "scheme": "HTTP"
    }