Implementation of Service Mesh in DataWorks Microservice Clusters

About the author

Yuan Gengtuo (Sanchen) is a technical expert at the Alibaba Cloud Intelligent-computing platform division. He is responsible for the construction of the microservice ecosystem for the Data Technology Plus platform and DataWorks. Currently, he focuses on microservices and other related technologies.

The Original Aspiration

DataWorks is a unified, intelligent big data platform that is widely used inside and outside Alibaba Group. It includes DataStudio, StreamStudio, HoloStudio, AppStudio, GraphStudio, PAIStudio, WebExcel, and many other online products. Each of these products is a rich, interactive single-page application similar to an Integrated Development Environment (IDE).

Though DataWorks serves tens of thousands of data developers in the group's major business lines and participants in the data ecology, this huge carrier cannot accurately meet the needs of every user; just like the saying goes, "no dish suits all tastes."

At this critical moment of the architecture upgrade, we are determined to implement the plug-in-based and service-based customization feature to DataWorks. By doing so, the coupling, due to a co-construction mode, between businesses in R&D can be removed, and a more lightweight and more open platform can be built. This also remains true to our original aspiration and helps our internal and external clients move their business customization into practice more quickly.

How can we open up the customization on DataWorks?

We have established a complete set of control systems for microservice development, which is the DataWorks MicroService Platform (DMSP). It meets the service customization requirements of business developers, integrates the mid-end capabilities of DataWorks, and helps redefine their own R&D platform for business data.

Pack Light

Packing light is for users, developers, and the business side. As the platform provider, we must strive for that. For users, the lightest way is to focus only on business logic. Business developers only focus on business logic. They do not have to keep an eye on authentication, routing, load balancing, mirroring, and the throttling of the ingress, or timeouts, retries, and access control of the egress. They do not need to pay attention to other factors that are irrelevant to business logic, like isolation policies for inter-service calls, circuit breaking, and distributed tracing.

In the past, a large number of components and frameworks have been developed in the microservice ecosystem to help us solve these problems. However, these components and frameworks have high thresholds, high maintenance costs, and coupled development languages. So, it is too costly for most developers to apply them in emerging businesses that require agile development.

The Service Mesh technology, which has become popular in recent years, can extricate us from this predicament. For more information, see "Get Started with Service Mesh".

Capabilities of Microservice Clusters

DMSP is a microservice running platform based on Kubernetes clusters. Based on the Kubernetes Operator, we have developed a complete lifecycle management system for microservices. As a whole, DMSP enhances microservices in the following fields.

DMSP Service Mesh Architecture

Based on the preceding requirements of cluster capabilities, we make the Service Mesh of Istio into practice. The following figure shows the overall architecture.

The lower-level SOA services include many of the existing key services of DataWorks, for example, tenants, data development, and AI platforms.

The DMS console is a microservice console. It is the only tunnel for microservice developers to control and interact with service clusters. It manages all the clusters, including the internal ones used by Alibaba, and the external ones in the 22 regions of Alibaba Public Cloud. In the console, all the aforementioned functions will be interconnected, like deployment, traffic control and grayscale, dynamic security policies, and service governance (logging, monitoring, and alerting.)

Interactions between the console and the control plane are completely based on Kubernetes API Server. They interact with each other through the Custom Resource Definition (CRD), an abstract layer. Then, a specific operator executes the control logic according to the CRD. By controlling Istio, we can extend the service capability with the help of Sidecar. With the extension, like given a pair of wings, business services seamlessly acquire the capabilities conferred by clusters. They take over the maximum capabilities from microservices at the minimum development cost.

Why the Operator and CRD?

The best practice to expand Kubernetes is to adopt the Operator + CRD mode. Based on the Kubernetes Operator, we could move the workloads of service deployment and Service Mesh control layer forward to the clusters. This reduces the complexity of deployment and control that would otherwise require multiple clusters and multiple configuration centers.

Eventual Consistency Guarantee

The control operations are reliable and ensure eventual consistency. This relies on two advantages:

The Operator framework ensures eventual consistency by means of the Kubernetes Event and Watch mechanism (Reconcile Cycle), multi-event triggers, and idempotence. Developers do not need to pay much attention to the idempotence due to the other advantage below.
The generation mechanism of CRUD operations to Kubernetes resources ensures the atomicity of CRD operations.

With these advantages, we only need to focus on what the goal is and forge ahead when developing operators.

Clear business logic boundaries

By leveraging a set of customized CRDs with the Kubernetes Operator, we can reduce the complexity of business logic configuration. By abstracting each business logic to a CRD, the business functional boundaries become clear and maintainable. Moreover, the robustness is improved.

Secure control links

Owing to CRD-based interactions, clusters do not require additional control interfaces. Therefore, all the external control operations can be managed within the native API server specification. The RBAC system of Kubernetes can be fully reused. This ensures the security and reliability of control links with subtle and controllable permissions.

For more information, click here for OpenShift best practices.

Abstract Design of Istio

The CRD system of Istio is complex. Therefore, we need to implement a simple control plane, abstract Istio, and use easy-to-understand APIs for control interactions to achieve more with less. For example: The control logic at the abstract layer only requires less than 10 lines and 1 CRD for a control that would otherwise require about 100 lines and 3 CRDs when using the Istio native control configuration.

spec:
  service: myservice1
  rules:
  - version: "1.1"
    weight: 70
  - version: "1.2"
    weight: 30
  - version: "2.0"
    weight: 100

Service Routing (Route)

Microservices go with quick iteration and agile development. Therefore, the refined control of the traffic to different Canary versions is the primary focus of cluster construction.

This is often referred to as the version grayscale. The grayscale works mainly in two ways:

Weight grayscale
Rule grayscale

Weight Grayscale

It balances the traffic to a specific version of the service in scale. Here, we designed major versions and minor versions and specify that each major version upgrade must use a different domain name (/ContextPath). We also specify for all the minor versions under each major version, the sum of their traffic weights is 100.

示例：
spec:
  service: httpbin
  rules:
  - version: "1.0"
    weight: 80
  - version: "1.1"
    weight: 20
  - version: "2.0"
    weight: 100

After adjustment, visualized service traffic monitoring works as follows:

Rule Grayscale

It directs traffic to a specified version based on the specified matching rule.

spec:
  service: myservice1
  rules:
  - version: "2.0"
    matches:
    - headers:
        gray:
          exact: "1"
    - queryParams:
        debug:
          exact: "true"

The traffic will be led to version 1.1, if the header has the Gray field as 1 or the query parameter has the Debug field as true. We can know whether the grayscale takes effect from the header carried in the response.

Service Security

According to the direction of traffic, service security is divided into two categories, north-south traffic control and east-west traffic control. North-south traffic control manages the traffic that flows into and out of the mesh, whereas east-west traffic control manages the traffic inside the mesh.

North-south traffic control mainly manages the ingress traffic with Ingress Gateway's Virtual Service (VS) capabilities and manages the egress traffic with the help of Sidecar and Egress Gateway.
The east-west traffic is controlled through mTLS and Authorization Policy.

These measures can be used to enhance the security of microservices in clusters, especially the clusters in public clouds.

Ingress Traffic

Some control requirements for ingress traffic have been mentioned in the preceding Service Routing topic. In comparison, service security provides global control capabilities, such as:

CORS cross-domain configuration
CSRF protection
Log-on authentication

CORS and CSRF are able to cover unified general rules with the help of the Sidecar Filter. Therefore, it is undemanding for further refinement and dynamic configuration.

Log-on authentication is mainly used to verify whether the identity of the frontend access traffic is legal (no permission system is involved.) It will interconnect with various frequently used log-on systems. Currently, it mainly supports the built-in log-on system of DataWorks.

We have designed a flexible authentication framework based on the filter and the normative authentication service to meet the requirements of log-on authentication.

Egress Traffic

Egress traffic refers to the traffic that accesses websites and services outside the mesh, for example, when microservices access Alibaba Cloud OSS.

To prevent services from accessing unknown malicious websites after being infected, especially in high security-level cluster environments, we trust no IP addresses as destinations by default and add specific trusted addresses in the whitelist.

Currently, the Flannel network does not support network policies. As the cluster provided by Alibaba Cloud Container Service for Kubernetes (ACK) uses the Flannel network by default (Terway can be selected as another option), pod-level network policies do not take effect, and therefore pods not controlled by Istio still need protection. For pods under Istio control, there is only one layer of protection from Istio.

Default access denial

We could configure that all outbound traffic from Sidecar to services outside the mesh must be registered in ServiceEntry to get through. In other words, all connections are denied by default.

kind: IstioControlPlane
spec:
 gateways:
   components:
     egressGateway:
       enabled: true
 values:
   global:
     outboundTrafficPolicy:
       mode: REGISTRY_ONLY

After the configuration takes effect, the traffic that flows through SideCar for mutual access between pods will be checked in ServiceEntry for registration. Unregistered services will be blocked as follows:

$ kubectl exec -it sleep-788f7c9bcb-m67rh -c sleep -- curl -I www.aliyun.com
HTTP/1.1 502 Bad Gateway

Code 502 means that both the DNS and IP address can be reached, but the access attempt is blocked by Sidecar. You can see that all traffic has reached BlackHoleCluster, a built-in cluster dedicated to receiving blocked requests.

The case for PassthroughCluster is similar.

Whitelists

Based on the default denial policy, we add a whitelist.

Global whitelists

The Service Entry API does not have the selector that specifies certain workloads as the targets to take effect. Once the configuration is made, it is globally applied. For this reason, it only supports namespace-level isolation with the ExportTo parameter.

Therefore, we abstract a dynamic configuration layer to accommodate more configuration rules in the future.

spec:
  service: sleep
  hosts:

  - "www.aliyun.com"

We can see that all traffic goes through the endpoint defined by Service Entry, and the access attempt is permitted.

Service Access Control

Service access control is the access authorization mechanism between microservices. In the zero-trust network, access to all services is denied by default.

In the mesh, the mTLS mechanism is used for identity verification between services. Each service has its own Kubernetes service account, which verifies the identity of the certificate that uses the mTLS mechanism provided by Istio. Therefore, identities of the accessing services can be distinguished. This way, we can control traffic with authorization policies. Configure the default denial:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
spec:
  {}

After the configuration takes effect, the blocked access attempt returns a message stating "403 RBAC: access denied."

$ kubectl exec -it sleep-788f7c9bcb-m67rh -c sleep -- curl -i httpbin:8000/get
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Mon, 13 Jan 2020 16:09:16 GMT
server: envoy
x-envoy-upstream-service-time: 0
RBAC: access denied

Looking Ahead

With the Service Mesh technology, we have built a DataWorks microservice DevOps system that can customize open APIs.

We hope that our efforts can substantially lower the development threshold, make business access more agile, and accelerate development iteration.

In the future, the Alibaba will do more with DataWorks to empower control and governance capabilities oriented to the microservice ecology.

Community