Alibaba Cloud Service Mesh:How does the server application obtain the client source IP in a service mesh environment

Method comparison for east-west traffic

Deploy the sleep application

1. Create sleep.yaml:

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: sleep
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: sleep
        labels:
          app: sleep
          service: sleep
      spec:
        ports:
        - port: 80
          name: http
        selector:
          app: sleep
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: sleep
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: sleep
        template:
          metadata:
            labels:
              app: sleep
          spec:
            terminationGracePeriodSeconds: 0
            serviceAccountName: sleep
            containers:
            - name: sleep
              image: curlimages/curl
              command: ["/bin/sleep", "3650d"]
              imagePullPolicy: IfNotPresent
              volumeMounts:
              - mountPath: /etc/sleep/tls
                name: secret-volume
            volumes:
            - name: secret-volume
              secret:
                secretName: sleep-secret
                optional: true

2. Apply the manifest:

      kubectl -n default apply -f sleep.yaml

Deploy the httpbin application

1. Create httpbin.yaml:

      apiVersion: v1
      kind: Service
      metadata:
        name: httpbin
        labels:
          app: httpbin
      spec:
        ports:
        - name: http
          port: 8000
        selector:
          app: httpbin
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: httpbin
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: httpbin
            version: v1
        template:
          metadata:
            labels:
              app: httpbin
              version: v1
          spec:
            containers:
            - image: docker.io/citizenstig/httpbin
              imagePullPolicy: IfNotPresent
              name: httpbin
              ports:
              - containerPort: 8000

2. Apply the manifest:

      kubectl -n default apply -f httpbin.yaml

Verify the default behavior

1. Check that both pods are running: Expected output: Note the pod IP of sleep (172.17.X.XXX).

      kubectl -n default get pods -o wide

      NAME                            READY   STATUS    RESTARTS   AGE     IP             NODE
      httpbin-c85bdb469-4ll2m         2/2     Running   0          3m22s   172.17.X.XXX   cn-hongkong.10.0.0.XX
      sleep-8f764df66-q7dr2           2/2     Running   0          3m9s    172.17.X.XXX   cn-hongkong.10.0.0.XX

2. Send a request from sleep to httpbin: Expected output: The source address is 127.0.0.6 (the Envoy local address), not the actual IP of the sleep pod.

      kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip

      {
        "origin": "127.0.0.6"
      }

3. Confirm through socket information: Expected output:

   If netstat is not available in the httpbin container, log on to the container and run apt update && apt install net-tools.

      kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 80

      tcp        0      0 172.17.X.XXX:80         127.0.0.6:42691         TIME_WAIT   -

4. Check the Envoy access log for the httpbin pod. A formatted log entry looks like: Key fields:

   Field	Value in this example	Meaning
   downstream_remote_address	172.17.X.XXX:56160	The address of the sleep pod (the calling client).
   upstream_local_address	127.0.0.6:42169	The local address Envoy uses to connect to httpbin. This is why the application sees 127.0.0.6 as the source.

      {
        "downstream_remote_address": "172.17.X.XXX:56160",
        "downstream_local_address": "172.17.X.XXX:80",
        "upstream_local_address": "127.0.0.6:42169",
        "upstream_host": "172.17.X.XXX:80"
      }

Method 1: TPROXY transparent interception mode

TPROXY uses the Linux kernel transparent proxy mechanism to preserve the original source IP at the network layer. Unlike the default REDIRECT mode (which uses iptables NAT and loses the source IP), TPROXY preserves the source IP address at the socket level.

1. Patch the httpbin deployment to enable TPROXY:

      kubectl patch deployment -n default httpbin -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/interceptionMode":"TPROXY"}}}}}'

2. Send a request from sleep to httpbin: Expected output: The response now shows the actual IP of the sleep pod.

      kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip

      {
        "origin": "172.17.X.XXX"
      }

3. Confirm through socket information: Expected output:

   After the pod restarts, reinstall netstat if needed.

      kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 80

      tcp        0      0 172.17.X.XXX:80         172.17.X.XXX:36728      ESTABLISHED -

4. Verify in the Envoy access log: The upstream_local_address now shows the sleep pod IP instead of 127.0.0.6.

      {
        "downstream_remote_address": "172.17.X.XXX:39058",
        "downstream_local_address": "172.17.X.XXX:80",
        "upstream_local_address": "172.17.X.XXX:46129",
        "upstream_host": "172.17.X.XXX:80"
      }

Method 2: X-Forwarded-For (XFF) request header

This method configures the sidecar proxy to inject the real client IP into the X-Forwarded-For HTTP header before forwarding the request to the application. This approach works on all node operating systems but requires your application to parse the X-Forwarded-For header.

1. Apply the following EnvoyFilter to the ASM instance. For instructions, see Create an Envoy filter by using an Envoy filter template.

      apiVersion: networking.istio.io/v1alpha3
      kind: EnvoyFilter
      metadata:
        name: enable-xff-for-sidecar-inbound
        namespace: istio-system  # Change this namespace to the namespace where the gateway is located
        labels:
          asm-system: "true"
          provider: "asm"
      spec:
        configPatches:
        - applyTo: NETWORK_FILTER
          match:
            proxy:
              proxyVersion: "^1.*"
            context: SIDECAR_INBOUND
            listener:
              name: "virtualInbound"
              filterChain:
                filter:
                  name: "envoy.filters.network.http_connection_manager"
          patch:
            operation: MERGE
            value:
              typed_config:
                "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
                use_remote_address: true

2. Send a request from sleep to httpbin: Expected output: The httpbin application reads the client IP from the X-Forwarded-For header and returns it correctly.

      kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip

      {
        "origin": "172.17.X.XXX"
      }

This method requires your application to read the X-Forwarded-For header. The httpbin sample application has this capability built in. If your application does not parse this header, use TPROXY instead.

HTTP protocol

HTTPS protocol

After you enable source IP preservation (as described in the HTTP section), apply the same approach for HTTPS.

1. Create https-demo.yaml to expose httpbin through HTTPS:

      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: httpbin-gw-https
        namespace: default
      spec:
        selector:
          istio: ingressgateway
        servers:
          - hosts:
              - '*'
            port:
              name: https
              number: 443
              protocol: HTTPS
            tls:
              credentialName: myexample-credential
              mode: SIMPLE
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: httpbin-https
        namespace: default
      spec:
        gateways:
          - httpbin-gw-https
        hosts:
          - '*'
        http:
          - route:
              - destination:
                  host: httpbin
                  port:
                    number: 8000

2. Apply the manifest:

      kubectl -n default apply -f https-demo.yaml

3. Access httpbin through the ingress gateway: Expected output: The response shows the actual client source IP address.

      export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
      curl -k https://$GATEWAY_URL:443/ip

      {
        "origin": "120.244.XXX.XXX"
      }