Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Implement a New Service Mesh Integrating Sidecarless and Sidecar Modes

Implement a New Service Mesh Integrating Sidecarless and Sidecar Modes

This article introduces how ASM implements this new form of service mesh that integrates Sidecarless and Sidecars, achieving a serverless service mesh.

By Zehuan Shi and Hang Yin

Preface

In September 2022, the Istio community introduced a new architecture called Ambient, also known as the Sidecarless mode. This mode transforms the Sidecar architecture into Layer 4 and Layer 7 proxies. Alibaba Cloud service mesh (ASM) is the industry's first managed service mesh that supports the Ambient mode.

This article is based on the latest progress of ASM product technologies shared at the 2023 Apsara Conference. Zehuan Shi and Hang Yin from the service mesh team at Alibaba Cloud's cloud-native product line will present four parts to introduce how ASM implements this new form of service mesh that integrates Sidecarless and Sidecars, achieving a serverless service mesh.

Evolution of Service Mesh Architecture

As service mesh becomes increasingly popular in the cloud-native technology ecosystem, more and more enterprise technical teams are adopting service mesh in their production environments. We assume that many of you are already familiar with service mesh. To help beginners understand the upcoming content more smoothly, let's first introduce the classic Sidecar mode of service mesh.

The classic service mesh architecture is divided into the control plane and the data plane based on the components' responsibilities. The control plane acts as the manager of the service mesh, providing different configurations for the data plane components as required. The data plane, as the executor, manipulates traffic based on the configurations issued by the control plane, serving as the real executor of service mesh capabilities.

To implement service mesh capabilities such as traffic routing, load balancing, fault injection, request response manipulation, authentication, and zero-trust networking, the service mesh's Injector injects a dedicated container for executing service mesh capabilities into the application pod. This container resides in the same pod as the application and shares the network namespace. This container is known as the Sidecar of the service mesh.

Since the Sidecar and the application container share the same network namespace, iptables rules can be used to easily intercept application traffic to the mesh proxy process inside the container. This represents the classic Sidecar architecture.

_1

ASM decouples the control plane from the data plane and enables hosted deployment. Compared to Istio, ASM offers several significant advantages. Firstly, ASM provides comprehensive lifecycle management capabilities. In service mesh operation and maintenance, challenges often arise with installation and upgrades due to the complex configuration and rapid iteration of the community. With ASM's lifecycle management, users can effortlessly create, delete, and upgrade service mesh instances without concerns about configuration issues, compatibility, or adaptation.

Secondly, ASM offers the ability to block and diagnose configuration errors. It is common for customers to face issues with expected results due to configuration errors when using service mesh. In such cases, ASM converts these errors into check and diagnosis items to intercept and promptly send alerts, helping service mesh operations and maintenance personnel identify and resolve problems in a timely manner.

As a network infrastructure for cloud-native applications, service mesh facilitates the connection between applications and observable platforms. To this end, ASM provides fast connectivity to various cloud services, enabling users to quickly integrate with the cloud-native ecosystem.

Lastly, ASM supports enterprise-level multi-cluster mode, allowing users to swiftly implement multi-data-plane clusters in the service mesh.

_2

ASM currently leverages serverless technology for some control plane components. Serverless components can automatically scale out and be utilized on-demand. By utilizing the capabilities of the serverless framework, ASM achieves higher scheduling efficiency and optimized startup speeds. It significantly reduces the time required for readiness and startup by utilizing image caching..

_3

New Data Plane Mode

While the sidecar mode is intuitive and effective, it does have some drawbacks:

  1. Sidecar injection has a intrusive impact on workloads. Injecting or removing the sidecar requires restarting the workloads, and adjusting sidecar configurations (such as resources) may also necessitate workload restarts. Sidecars and workloads are strongly coupled.
  2. Resource utilization is suboptimal. To handle worst-case scenarios, each sidecar needs to reserve some resources. The larger the cluster scale, the more idle resources there are.
  3. The computational cost of Layer 7 processing, such as traffic capture and protocol identification, is high. However, not all requests are HTTP protocol or require processing by sidecars.

For these reasons, we need a less invasive and more cost-effective way to apply service mesh to a wider range of scenarios. As a response, in September 2022, the Istio community introduced the Sidecarless Ambient mode. This mode separates the functions of the sidecar into Layer 4 and Layer 7 proxies. The deployment of these proxies is decoupled from the workloads, addressing the limitations of the sidecar mode in certain scenarios. ASM is the industry's first managed service mesh that supports the Ambient mode.

In Ambient mode, the Layer 4 proxy focuses on observability, routing, and communication encryption at the transport layer. The Layer 7 proxy performs more complex behavior processing in traffic management, security, and observability based on Layer 7 protocols. Users can gradually select whether to enable the proxy for an application and which layer of proxies to enable based on their business requirements.

_4

We refer to the Layer 4 proxy on the data plane as Ztunnel, which serves as the processing layer. The processing layer is responsible for forwarding all Layer 4 communications of the application. With the capabilities of Ztunnel, the application immediately gains zero-trust security capabilities upon joining the mesh, including MTLS, authentication, and authorization policies. After deploying Ztunnel in DaemonSet mode, CNI is used to configure traffic interception rules on nodes, intercepting traffic from pods in the mesh to the Ztunnel instance. Ztunnel encrypts the traffic using MTLS and transmits it, while the peer Ztunnel decrypts the traffic and forwards it to the application. Along these paths, Ztunnel can also collect TCP monitoring metrics and access logs.

_5

We refer to the Layer 7 proxy as the Waypoint proxy. Similar to the Sidecar in the classic architecture, the Waypoint proxy is an Envoy-based proxy that implements higher-level capabilities based on Layer 7 protocols in Ambient Mesh mode. For example, the Waypoint proxy can apply advanced service mesh policies based on request headers and credentials, such as fusing, traffic shaping, traffic splitting, retries, fault injection, and role-based access control authorization policies. As opposed to the Ztunnel proxy deployed at the node level, the Waypoint proxy is deployed at the service level. Users can enable or disable the Layer 7 proxy for a service, or adjust the deployment scale as needed to improve resource utilization in the cluster.

_6

After understanding the specific capabilities of Layer 4 and Layer 7 proxies, let's take a look at the network topology in Ambient mode decoupled by Layer 4 and Layer 7:

_7

Let's take a look at the traffic path in Ambient mode, starting with the Layer 4 proxy:

  1. In Ambient mode, when an application pod starts, the CNI plugin writes its IP address to the ipset in the node's network namespace.
  2. When a request is made, traffic packets reach the node's network namespace through the pod's veth pair interface. Packets from addresses in the ipset are captured and processed by the iptables rules on the node.
  3. The packets are marked as 0x100 by the iptables rules.
  4. The policy-based routing rule on the node specifies that any packet marked as 0x100 should be directed to the destination IP 192.168.127.2 through the istio outbound network interface.
  5. The transparent proxy iptables rule in the Ztunnel proxy pod sends packets from pistioout to the Ztunnel outbound port 15001.
  6. Ztunnel processes the packets and forwards them to the IP address of the destination service (httpbin). Since the address is in Node B, where httpbin's veth equipment is located, the packets are routed to Node B.
  7. After the packets reach Node B, inbound traffic rules ensure that the packets are routed to the istioin interface.
  8. The packets enter the Ztunnel pod through the tunnel formed by istioin and pistioin.
  9. The iptables rule in the Ztunnel pod captures the packets from pistioin and directs them to port 15008 based on the tag.
  10. The Ztunnel pod processes the packets and sends them to the destination pod.

_8

When the Layer 4 proxy forwards outbound data, if the destination application has enabled the Layer 7 proxy, the Layer 4 proxy forwards the traffic to the Layer 7 proxy of the destination application through the HBONE tunnel. The traffic enters the Layer 7 proxy through the connect terminate listener, listening on port 15008. This listener uses a specific filter to unpack the HBONE traffic, complete identity authentication, and then sends the traffic to the main internal listener for further processing. On the main listener, the traffic is matched based on the service IP and port, and the Layer 7 traffic policy determines the destination cluster. Finally, the traffic enters an internal listener named connect_originate, which continues to forward the traffic to the upstream destination through the HBONE tunnel.

_9

Those who are interested in the Ambient Mesh traffic path can also refer to another article, in which the traffic path in Ambient mode is analyzed in more depth and detail.

The Integration of Sidecarless and Sidecar Modes

In the new architecture, the Sidecar mode does not conflict with the Sidecarless mode. Users can use the two modes together to deploy in a hybrid manner. With this feature, it is possible to gradually complete the switchover as needed. The Ambient mode managed by ASM can reduce resource overhead by 60%, O&M by 50%, and request latency by 40% in specific scenarios.

_10

ASM uses a unified control plane API to provide a configuration management platform for the data plane. In addition, in the new architecture, ASM delivers different configurations as needed for Sidecars in Sidecar mode, and for Layer 4 and Layer 7 proxies and even managed proxies in Sidecarless mode.

_11

The Layer 7 proxy is more capable of handling service mesh functionalities. Therefore, in practical use, the Layer 7 proxy is more likely to scale out and scale in as business applications also scale out and scale in. To provide a flexible deployment of the Layer 7 proxy under the new architecture, ASM offers a managed Layer 7 proxy. This Layer 7 proxy is deployed in a serverless mode to simplify the operation and maintenance complexity. There is no need to plan the capacity of the Layer 7 proxy in advance or perform any operation and maintenance tasks. You can deploy, reclaim, and scale the Layer 7 proxy at any time based on your business needs.

_12

Let's take a look at the technical architecture of the managed Layer 7 proxy. Users can declare the Layer 7 proxy through Kubernetes standard Gateway APIs. The Waypoint Proxy Controller on the managed ASM side will monitor the Gateway APIs. When a Gateway CR is created or modified, the Waypoint Proxy Controller will manage the lifecycle of the Waypoint proxy workload based on the specifications of the Gateway APIs. You can deploy the Layer 7 proxy in the Waypoint proxy pool managed by ASM or on Serverless nodes of an ECI in your clusters using the Gateway APIs.

_13

Conclusion

Finally, let's summarize the enterprise-level service mesh capabilities provided by ASM. In terms of service mesh capabilities, ASM supports unified governance of heterogeneous services, network management of multi-cluster and cross-cluster applications, grayscale release and deployment of applications integrated with CI/CD tools, smooth evolution of application cloud architecture, and AI-based elastic service management using Kserve. In terms of integration and compatibility, ASM supports a Web user interface, provides a complete OpenAPI, and offers powerful and flexible integration capabilities. ASM is fully compatible with the usage specifications of Istio and allows configuration changes for service mesh instances through standard Kubernetes APIs.

The core components of the ASM control plane are fully managed. The standard edition and enterprise edition use a unified and flexible architecture to provide comprehensive support for multiple versions. Additionally, ASM provides several powerful customization capabilities, including traffic management enhancement, protocol enhancement, adaptive XDS optimization, software and hardware integration optimization, mesh diagnosis and analysis, extension center, and heterogeneous service registration and integration.

ASM serves as a network platform for cloud-native applications and provides unified mesh governance capabilities for application services running on different computing infrastructures. Leveraging the powerful capabilities of heterogeneous computing infrastructures, ASM enables users to manage and maintain workloads running on Kubernetes nodes, Serverless workloads running on ECIs, workloads of managed Serverless components, and even workloads and heterogeneous computing infrastructures in other public clouds or IDCs. For more information about ASM capabilities, please visit the ASM homepage or the ASM documentation.

_14

Developers Kubernetes cloud native Service Mesh Sidecar ASM Sidecarless Ambient
8 1 1
Share on

Read previous post:

Analysis of Istio Ambient Mesh Ztunnel Implementation - Configuration Analysis

Read next post:

Best Practices of OpenSergo and Dubbo Microservices Governance

Alibaba Cloud Native Community

506 posts | 48 followers

You may also like

Comments

5965002955462120 December 19, 2023 at 4:34 am

'">

5965002955462120 December 19, 2023 at 4:35 am

lmao

5965002955462120 December 19, 2023 at 4:35 am

lmaoo2

5965002955462120 December 19, 2023 at 4:36 am

5965002955462120 December 19, 2023 at 4:36 am

5965002955462120 December 19, 2023 at 4:36 am

5965002955462120 December 19, 2023 at 4:37 am

<

5965002955462120 December 19, 2023 at 4:38 am

<<<weg>>>wegweg

Alibaba Cloud Native Community

506 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All