Alibaba Cloud Service Mesh：Manage Spring Cloud services

Demo introduction

In this example, Spring Cloud with the Nacos registry is used. You can download the demo sample code from the nacos-examples repository on GitHub.

If the consumer and provider services are not deployed in ASM instances, that is, sidecar proxy injection is not enabled for the service pods, you can access the services but you cannot manage the services by using Istio resources.

Step 1: Enable support for Spring Cloud on the ASM control plane

Method 1: Applicable to all Spring Cloud implementations and registries

1. Use kubectl to connect to the cluster on the control plane. For more information, see Use kubectl on the control plane to access Istio resources.

2. Create an Envoy filter in the ACK cluster that is added to the ASM instance.

   1. Create an any-spring-cloud-support.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: EnvoyFilter
      metadata:
        labels:
          provider: "asm"
          asm-system: "true"
        name: any-spring-cloud-support
        namespace: istio-system
      spec:
        configPatches:
        - applyTo: HTTP_FILTER
          match:
            proxy:
              proxyVersion: "^1.*"
            context: SIDECAR_OUTBOUND
            listener:
              portNumber: 8070
              filterChain:
                filter:
                  name: "envoy.filters.network.http_connection_manager"
                  subFilter:
                    name: "envoy.filters.http.router"
          patch:
            operation: INSERT_BEFORE
            value: # reverse_dns filter specification
             name: com.aliyun.reverse_dns
             typed_config:
               "@type": "type.googleapis.com/udpa.type.v1.TypedStruct"
               type_url: type.googleapis.com/envoy.config.filter.reverse_dns.v3alpha.CommonConfig
               value:
                 pod_cidrs:
                 - "10.0.128.0/18"

      Modify the parameter settings in the file based on your business requirements. Some parameters are described as follows:

      • portNumber: specifies the port number of the Spring Cloud service. If there are multiple ports, you can delete this parameter. If the ports can be converged, you can create multiple Envoy filters and configure a port number in each Envoy filter.

      • pod_cidrs: specifies the pod CIDR block of the ACK or ACK Serverless cluster. Log on to the Container Service for Kubernetes (ACK) console, go to the Clusters page and click the desired cluster. On the cluster details page that appears, click the Cluster Resources tab. Then, click the link in the VPC section to view the CIDR block of the vSwitch in the VPC.

   2. Run the following command to enable the com.aliyun.reverse_dns filter for the service:

      kubectl apply -f any-spring-cloud-support.yaml

Method 2: Applicable to only Nacos registries

1. Use kubectl to connect to the cluster on the control plane. For more information, see Use kubectl on the control plane to access Istio resources.

2. Create a service entry.

   1. Create an external-nacos-svc.yaml file that contains the following content:

      kind: ServiceEntry
      metadata:
        name: external-nacos-svc
      spec:
        hosts:
        - "NACOS_SERVER_HOST" ## Replace this variable with the endpoint of your Nacos server host. Example: mse-xxx-p.nacos-ans.mse.aliyuncs.com. 
        location: MESH_EXTERNAL
        ports:
        - number: 8848
          name: http
        resolution: DNS

      In the preceding YAML file, port 8848 is the default port used by Nacos. If you use a self-managed Nacos server and a different port, set the number parameter to the port number that you use.

   2. Run the following command to create the service entry:

      kubectl apply -f external-nacos-svc.yaml

3. Create an Envoy filter.

   1. Create an external-envoyfilter.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: EnvoyFilter
      metadata:
        labels:
          provider: "asm"
          asm-system: "true"
        name: nacos-subscribe-lua
        namespace: istio-system
      spec:
        configPatches:
          # The first patch adds the lua filter to the listener/http connection manager.
        - applyTo: HTTP_FILTER
          match:
            proxy:
              proxyVersion: "^1.*"
            context: SIDECAR_OUTBOUND
            listener:
              portNumber: 8848
              filterChain:
                filter:
                  name: "envoy.filters.network.http_connection_manager"
                  subFilter:
                    name: "envoy.filters.http.router"
          patch:
            operation: INSERT_BEFORE
            value: # lua filter specification
             name: envoy.lua
             typed_config:
                "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
                inlineCode: |
                   -- copyright: ASM (Alibaba Cloud ServiceMesh)
                   function envoy_on_request(request_handle)
                     local request_headers = request_handle:headers()
                     -- /nacos/v1/ns/instance/list?healthyOnly=false&namespaceId=public&clientIP=10.122.63.81&serviceName=DEFAULT_GROUP%40%40service-provider&udpPort=53174&encoding=UTF-8
                     local path = request_headers:get(":path")
                     if string.match(path,"^/nacos/v1/ns/instance/list") then
                       local servicename = string.gsub(path,".*&serviceName.*40([%w.\\_\\-]+)&.*","%1")
                       request_handle:streamInfo():dynamicMetadata():set("context", "request.path", path)
                       request_handle:streamInfo():dynamicMetadata():set("context", "request.servicename", servicename)
                       request_handle:logInfo("subscribe for serviceName: " .. servicename)
                     else
                       request_handle:streamInfo():dynamicMetadata():set("context", "request.path", "")
                     end
                   end
                   function envoy_on_response(response_handle)
                     local request_path = response_handle:streamInfo():dynamicMetadata():get("context")["request.path"]
                     if request_path == "" then
                        return
                     end
                     local servicename = response_handle:streamInfo():dynamicMetadata():get("context")["request.servicename"]
                     response_handle:logInfo("modified response ip to serviceName:" .. servicename)
                     local bodyObject = response_handle:body(true)
                     local body= bodyObject:getBytes(0,bodyObject:length())
                     body = string.gsub(body,"%s+","")
                     body = string.gsub(body,"(ip\":\")(%d+.%d+.%d+.%d+)","%1"..servicename)
                     response_handle:body():setBytes(body)
                   end

   2. Run the following command to create an Envoy filter:

      kubectl apply -f external-envoyfilter.yaml

Step 2: Deploy Spring Cloud services in an ACK cluster

1. Use kubectl to connect to the cluster on the data plane. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

2. Run the following command to deploy Spring Cloud services:

   export NACOS_ADDRESS=xxxx # Replace xxxx with the endpoint of the MSE Nacos registry or a self-managed Nacos registry. We recommend that you use the endpoint of a virtual private cloud (VPC). 
   wget https://alibabacloudservicemesh.oss-cn-beijing.aliyuncs.com/asm-labs/springcloud/demo.yaml -O demo.yaml
   sed -e "s/NACOS_SERVER_CLUSTERIP/$NACOS_ADDRESS/g" demo.yaml |kubectl apply -f -

3. Run the following command to check the Spring Cloud services:

    kubectl get pods

   Expected output:

   consumer-bdd464654-jn8q7       2/2     Running     0          25h
   provider-v1-66bc67fb6d-46pgl   2/2     Running     0          25h
   provider-v2-76568c45f6-85z87   2/2     Running     0          25h

Step 3: Create an Istio gateway and a virtual service

1. Create an Istio gateway.

   1. Create a test-gateway.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: test-gateway
      spec:
        selector:
          istio: ingressgateway # use istio default controller
        servers:
        - port:
            number: 80
            name: http
            protocol: HTTP
          hosts:
          - "*"

   2. Use kubectl to connect to the ASM instance based on the information in the kubeconfig file, and then run the following command to create an Istio gateway:

      kubectl apply -f test-gateway.yaml

2. Create a virtual service.

   1. Create a consumer.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: consumer
      spec:
        hosts:
        - "*"
        gateways:
        - test-gateway
        http:
        - match:
          - uri:
              prefix: /
          route:
          - destination:
              host: consumer.default.svc.cluster.local
              port:
                number: 8080

   2. Run the following command to create a virtual service:

      kubectl apply -f consumer.yaml

Step 4: Check whether ASM can manage the Spring Cloud services

1. Query the IP address of the ingress gateway.

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.

   3. On the Ingress Gateway page, view the Service address of the ingress gateway.

2. Run the following command to initiate requests from the ingress gateway to the Spring Cloud consumer service:

   curl <IP address of the ingress gateway>/echo/world

   Expected output:

   Hello Nacos Discovery From v1world
   Hello Nacos Discovery From v2world
   Hello Nacos Discovery From v1world
   Hello Nacos Discovery From v2world

   The output shows that traffic is routed to V1 and V2 of the provider service in polling mode by default.

3. Create a destination rule and a virtual service.

   1. Create a service-provider.yaml file that contains the following content:

      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: service-provider
      spec:
        host: service-provider
        subsets:
        - name: v1
          labels:
            label: v1
        - name: v2
          labels:
            label: v2
                                      

   2. Run the following command to create a destination rule:

      kubectl apply -f service-provider.yaml

   3. Create a service-provider1.yaml file that contains the following content.

      The virtual service to be created defines that /echo/hello requests are routed to the provider service of V1 and other requests are routed to the provider service of V2.

      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: service-provider
      spec:
        hosts:
        - service-provider
        http:
        - name: "hello-v1"
          match:
          - uri:
              prefix: "/echo/hello"
          route:
          - destination:
              host: service-provider
              subset: v1
        - name: "default"
          route:
          - destination:
              host: service-provider
              subset: v2

   4. Run the following command to create a virtual service:

      kubectl apply -f service-provider1.yaml

4. Run the following command to initiate requests to the Spring Cloud consumer service:

   curl <IP address of the ingress gateway>/echo/hello

   Expected output:

   Hello Nacos Discovery From v1hello
   Hello Nacos Discovery From v1hello

   The output shows that /echo/hello requests are routed to the provider service of V1 and other requests are routed to the provider service of V2. This indicates that Spring Cloud traffic is taken over by Istio and Custom Resource Definitions (CRDs) provided by Istio can be used to configure routing rules. In this case, ASM can manage the Spring Cloud services.

FAQ

What do I do if the Spring Cloud services that I deployed do not take effect?

1. Check whether traffic blocking is enabled for the port or IP address of Nacos.

   • If you use reverse DNS lookup, you must block the IP address of the pod.

   • If you use Lua scripts, you must block the Nacos server IP address and cluster IP address.

2. Make sure that an Envoy filter is created before you create deployments so as to intercept the registration process. If specific deployments are created before you create the Envoy filter, you must enable rolling updates for the deployments.

3. Check whether you have created Kubernetes service resources and provided a cluster IP address for the Spring Cloud services.

4. View the method of enabling support for Spring Cloud on the ASM control plane.

   If Method 2 is used, the client SDK of Nacos must be of a version earlier than 2.0. The client SDK of v2.0 or later uses gRPC to connect to the server. Method 2 is not applicable. Method 1 has no requirements for the Nacos version and is applicable to all Nacos versions.

5. Check the sidecar version of the services. If the sidecar image version is earlier than 1.13.4.32, you may have updated only the control plane of the ASM instance but not updated the data plane. In this case, enable rolling updates for the deployments of the services.