By Zehuan Shi
In the Istio Ambient Mesh, a new component called Ztunnel has been introduced by the community. The name Ztunnel is derived from Zero-Trust Tunnel, which represents the Zero Trust Pipeline. Ztunnel focuses on providing Layer 4 security capabilities for workloads in Ambient Mesh, such as mTLS, authentication, and L4 authorization. Unlike Layer 7 traffic resolution, Ztunnel ensures efficient and secure transmission of traffic to the waypoint proxy responsible for Layer 7 processing or to the peer Ztunnel in the absence of a waypoint proxy at the peer end. Each node in Ambient Mesh is equipped with a Ztunnel pod, which hosts the Ztunnel workload. To enable the flow of traffic from pods belonging to Ambient Mesh and running on nodes, Ambient Mesh employs a set of iptables and routing rules to direct the traffic to Ztunnel. Subsequently, the traffic is forwarded through Ztunnel to the Ztunnel of the destination workload's node or to the waypoint proxy of the destination service. Understanding the Ztunnel component is crucial in comprehending the complete principle of Ambient Mesh. This article delves into the configuration of Ztunnel, providing insights into its capabilities and configuration mode.
Similar to the Istio Sidecar, Ztunnel exposes the Admin service on port 15000. You can use the following command to obtain the configuration of Ztunnel:
$ curl 127.0.0.1:15000/config_dump
{
"certificates": {...},
"config": {...},
"local_node": {...},
"policies": {...},
"policies_by_namespace": {...},
"proxy_mode": {...},
"static_config": {...},
"version": {...},
"vips": {...},
"workload_to_vip": {...},
"workloads": {...}
}If the configuration is fully expanded, even in a cluster with only 1 or 2 services, the content of the configuration file will still be relatively lengthy. Therefore, we can roughly understand the general structure of the configuration content from its outermost layer. Under the root nodes of the exported json configurations, there are 11 child nodes respectively. The contents and values of some configurations are clear, and it is not meaningful to describe them in detail. Therefore, we simply list these fields and descriptions here and will not describe them in detail:
• proxy_mode: indicates the working mode of Ztunnel.
• local_node: indicates the name of the node where Ztunnel is currently located.
• static_config: similar to static configurations of Envoy, which describes the configurations that are included in Ztunnel at startup.
• version: indicates the version information of the current Ztunnel, including the git version of Ztunnel, the git tag used for compilation, and the version of rust.
Since the other parts of the configurations are relatively complex, this article describes these configurations separately:
• certificats: information about the certificates of the workloads on all current nodes
• workloads and workd_to_vip and vips describe information about workloads and services
• policies and policies_by_namespace describe security-related rules
• config: the meta-configuration of Ztunnel, including the listening address, network configuration, and xds configuration.
Next, we will describe the above four parts in detail one by one.
The content of the certificates node is the certificate of the service. In the outbound implementation of Ztunnel, when the connection of the workload is accepted, it will initiate an HTTP CONNECT request to the waypoint or Ztunnel of the destination workload to establish the HBONE tunnel connection. This link is based on MTNLS, so when initiating the connection, the Ztunnel of the client needs to pretend to be the client workload. This requires that Ztunnel must have the certificate of the client workload it may proxy in order to be able to provide it to the peer end during connection, so that the other side can complete authentication and execute security policies. At the same time, the server Ztunnel also provides its own certificate for the client side to verify whether it is valid.
Next, let's look at the data structure of the certificates node, which is an array type. Each element of the array corresponds to certificate-related information of a ServiceAccount, mainly including CA certificate, CA certificate issuance and expiration time, ServiceAccount certificate chain, and ServiceAccount certificate issuance and expiration time.
{
...
"certificates": [
...
{
"identity": "spiffe://cluster.local/ns/default/sa/httpbin",
"state": "Available",
"ca_cert": [
{
"pem": "-----BEGIN CERTIFICATE-----\nMIICdTCCAV2gAwIBAgIQcctOHe3HvRaImFu1QUsgGDANBgkqhkiG9w0BAQsFADAY\nMRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIzMDgwNDIzNTEyMFoXDTIzMDgw\nNTIzNTMyMFowADBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABHRudKVC++2nUYrj\nx/TQ2U2l/wvmX6CvuDsvcLELf7+b2c3WVcQITnio2apFAWOIZU9qVt+CurnbY466\nMAWUrH6jgZ0wgZowDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMB\nBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFOnPx5l9UHAOSUvh\nuc0MnSgS2TKVMDoGA1UdEQEB/wQwMC6GLHNwaWZmZTovL2NsdXN0ZXIubG9jYWwv\nbnMvZGVmYXVsdC9zYS9odHRwYmluMA0GCSqGSIb3DQEBCwUAA4IBAQApr7RKte4F\n+2Qh6u5om3otFhM1esmC9ikMc0zjqHEc7+7lep4lqtUwM8mBcrA5CdwCLiiqDoNL\n3EL1AZ2mZ4cIbiotymoSxQrSx2SjrRZWaRyTsjA92d0iWj1Lx0pwCBIYTBjE21an\nTrZiz0WeZxF2J0gtlw621+xLI/cLhIWBHOfWT2qZOg6LbyWuS3qYCHKFyQ5X/go0\nwlsHWcKGUzc6+xvNKneDbiL/oEPpmhXRGlqnL2O2z+jaLwyEGMRKDtylReztQSvV\nmxvNaRx/HVMQHAdJNSfO15g0JzcQZ6Yqwx2vqkPQ/zDMecZj/Dg7xoYPPf8x9SyS\nF4QVET10523T\n-----END CERTIFICATE-----\n",
"serial_number": "151258384185133277899379188962641518616",
"valid_from": "2023-08-04T23:51:20Z",
"expiration_time": "2023-08-05T23:53:20Z"
}
],
"cert_chain": [
{ "pem": "-----BEGIN CERTIFICATE-----\nMIIC/TCCAeWgAwIBAgIRAORkFfz5r9oYtNAgPGO6V8wwDQYJKoZIhvcNAQELBQAw\nGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMzA3MjQxMjAzMDVaFw0zMzA3\nMjExMjAzMDVaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3\nDQEBAQUAA4IBDwAwggEKAoIBAQDCernaYAUasEywyIFUApAudaCd7ACUmWfn2dQ4\n0j+uDOJ8aCk6H5BX9fKjc76ytnus40gXhXGCCcJ9GAJk/EGFiD9cfRvfsSGrA3Gq\ngjYuzkgl7ugc1adABuXKiP+Syd8WRI6wBAfHp2NGP9eO8irQ9js+9McpbtMgkIJ1\nFgVjMRwiYN984zFQCOhOPs5iYtbDoT/8XVkxTOqO4U1pzexMewxgRXCao3+GPht4\nL424lYjyCHH2gvjm0U7eQnV9OcmOB4irqlhErXnFPxVW9ydQnimp8NauSJ4h7WqS\nFgbbqw9632ha0UJUakw9no13UpF2cZvO/ZViAVXBm2Kn3kSTAgMBAAGjQjBAMA4G\nA1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTpz8eZfVBw\nDklL4bnNDJ0oEtkylTANBgkqhkiG9w0BAQsFAAOCAQEAUvJM8BD+5yeY0Jt6yD4i\nSWBoACisot54zoXGxSMY5bMhiv1lLEMF0Lt0e7yLmLy92u51gxmBFZVNg6Qp6ek8\n4zgoksRi8roBWnw6s0VIL31MargRIbtQf7D/P4sojqeoTPmT9fE8nzYi49eTT785\nNQUdRzkRkmymbXOd5+8OVWUT+ouP/fN3FfBk5sWLgHbFX6ZG9XJ7bkJbH6HCiyFC\naAsCb738LDGIFxetN3GqivXB34TeO1ZPADXnOlvVZ9vEpYaXMXpSIJc54B58nHM+\nO5cr2vT4+aOjLNRzD4jwrx95mOmy95uGsJq6hbAG7TH8u6RNQ0+eBqvLsG6TIV2I\n/g==\n-----END CERTIFICATE-----\n",
"serial_number": "303583658698187291460204590577649080268",
"valid_from": "2023-07-24T12:03:05Z",
"expiration_time": "2033-07-21T12:03:05Z"
}
]
}
]
...
}If we decode this certificate chain, we can see that the subject of the certificate is spiffe://cluster.local/ns/default/sa/httpbin
Common Name:
Subject Alternative Names: URI:spiffe://cluster.local/ns/default/sa/httpbin
Organization:
Organization Unit:
Locality:
State:
Country:
Valid From: August 4, 2023
Valid To: August 5, 2023
Issuer: cluster.local
Serial Number: 71cb4e1dedc7bd1688985bb5414b2018Next, let's decode the CA certificate. We can observe that the certificate belongs to the organization cluster.local, which is the cluster's domain name. This CA certificate is issued by the Istio CA and is provided by Istiod. The certificates for the workloads are issued by this CA certificate.
Common Name:
Subject Alternative Names:
Organization: cluster.local
Organization Unit:
Locality:
State:
Country:
Valid From: July 24, 2023
Valid To: July 21, 2033
Issuer: cluster.local
Key Size: 2048 bit
Serial Number: e46415fcf9afda18b4d0203c63ba57ccThe second important part is the workload information, which consists of workloads, workloads_to_vip, and vips. This data is primarily used for Ztunnel routing. When Ztunnel performs Layer 4 forwarding, it needs to determine how to forward by obtaining the corresponding workload based on the source and destination addresses.
Starting with workloads, its data structure is a mapping from workload address to workload information. The key is the IP address of the workload, and the value is the detailed information of the workload. The design of this data structure is related to the implementation of Ztunnel. Since Ztunnel is a Layer 4 proxy, it can only obtain Layer 4 information, which includes the source and destination IP addresses. Therefore, Ztunnel needs this information to find the details of the source and destination workloads. This information includes the workload's name, waypoint information, protocol, service account, and node.
{
"workloads": {
...
"10.244.1.5": {
"workloadIp": "10.244.1.5",
"waypointAddresses": [],
"gatewayAddress": null,
"protocol": "HBONE",
"name": "httpbin-5c5944c58c-cvc84",
"namespace": "default",
"trustDomain": "cluster.local",
"serviceAccount": "httpbin",
"workloadName": "httpbin",
"workloadType": "deployment",
"canonicalName": "httpbin",
"canonicalRevision": "v1",
"node": "ambient-worker",
"nativeHbone": false,
"authorizationPolicies": [],
"status": "Healthy",
"clusterId": "Kubernetes"
},
}
...
}workload_tp_vip maintains the information from the workload to its corresponding services, and uses the address of the workload as the key to map the service address to which it belongs. This mapping can help Ztunnel find the corresponding service quickly through the workload information to maintain its work node information when the health status of the workload changes.
{
...
workloads_to_vip: {
"10.244.1.7": [ # Workload address
[
"10.96.130.105:8000", # The corresponding service address and port number
80, # The workload port corresponding to this service port
]
],
...
}
...
}The vips information is the information from the service to the workload. Key is the service address and port two-tuples, and the value is the array of the backend workload address and port two-tuples corresponding to this service and port.
{
...
"vips": {
"10.96.130.105:8000": [
...
[
"10.244.1.7",
80
],
...
],
}
}Ztunnel needs to implement a Layer 4 security policy, that is, the part of Layer 4 security of the AuthorizationPolicy is implemented by Ztunnel. For example, we define an AuthorizationPolicy that prohibits workloads from the foo namespace from accessing the httpbin service in the default namespace:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: default
spec:
action: DENY
rules:
- from:
- source:
namespaces: ["foo"]Then, use curl to initiate a request from the sleep pod in the foo namespace to the httpbin service in the default namespace. You can see that the request is rejected:
root@iZj6c2jc85jb656w3fto5uZ:~/temp# kubectl -n foo exec -it sleep-bc9998558-q5xfm -- curl httpbin.default.svc.cluster.local:8000
curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56Observe the Ztunnel log of the node where the httpbin service backend workload is located. The log shows the details of the rejected request:
RBAC rejected conn=10.244.2.12(spiffe://cluster.local/ns/foo/sa/sleep)->10.244.1.5:80。
2023-08-05T18:06:57.276108Z INFO inbound{midm=380b5fe19b2cf10214d2752ded3fdef3 mpeer_ipm=10.244.2.12 mpeer_idm=spiffe://cluster.local/ns/foo/sa/sleep}: ztunnel::proxy::inbound: got CONNECT request to 10.244.1.5:80
2023-08-05T18:06:57.276135Z INFO inbound{midm=380b5fe19b2cf10214d2752ded3fdef3 mpeer_ipm=10.244.2.12 mpeer_idm=spiffe://cluster.local/ns/foo/sa/sleep}: ztunnel::proxy::inbound: RBAC rejected conn=10.244.2.12(spiffe://cluster.local/ns/foo/sa/sleep)->10.244.1.5:80It should be noted that in the AuthorizationPolicy, we can also restrict the request's method, path, and other Layer 7 rules, which will not be executed by Ztunnel.
Next, let's take a closer look at the details of the security policy configuration. The policies_by_namespace indicates all the policy names in each namespace. The namespace serves as the key, and the value is an array. Each element in the array represents the name of a security policy, which corresponds to the key in policies. The content of policies contains the specific security policies. Referring to the previous AuthorizationPolicy, there is a security policy named default/httpbin (httpbin is the name of the AuthorizationPolicy). Its action is deny, and the matching rule is under groups. It can be seen that the namespace foo is specified under namespaces. This means that the request matching the conditions under groups will execute the action specified by deny.
{
...
"policies_by_namespace": {
"default": [
"default/httpbin"
]
},
"policies": {
"default/httpbin": {
"name": "httpbin",
"namespace": "default",
"scope": "Namespace",
"action": "Deny",
"groups": [
[
[
{}
],
[
{
"namespaces": [
{
"Exact": "foo"
}
]
}
]
]
]
}
},
...
}config describes some meta information of Ztunnel, which is roughly divided into the following categories:
• HTTP/2 transmission-related configurations: window_size, connection_window_size, and frame_size. These parameters can be used to optimize HTT/2 performance.
• Listener configurations: socks5_addr, admin_addr, stats_addr, readiness_addr, inbound_addr, inbound_plaintext_addr, and outbound_addr.
• Ztunnel attributes: local_node, proxy_mode, local_ip, cluster_id, and proxy_metadata.
• Certificate-related configurations: ca_address, ca_root_cert, and fake_ca.
• XDS related configurations: xds_address, xds_root_cert, and xds_on_demand.
• Ztunnel running-related configurations: termination_grace_period, num_worker_threads, enable_original_source, and proxy_args.
{
...
"config": {
"window_size": 4194304, # http2 stream的window_size
"connection_window_size": 4194304, # http2连接的window_size
"frame_size": 1048576, # http2的帧大小
"socks5_addr": "127.0.0.1:15080", # socks5
"admin_addr": "127.0.0.1:15000",
"stats_addr": "[::]:15020",
"readiness_addr": "[::]:15021",
"inbound_addr": "[::]:15008",
"inbound_plaintext_addr": "[::]:15006",
"outbound_addr": "[::]:15001",
"local_node": "ambient-worker2",
"proxy_mode": "Shared",
"local_ip": "10.244.2.3",
"cluster_id": "Kubernetes",
"ca_address": "https://istiod.istio-system.svc:15012",
"ca_root_cert": {
"File": "./var/run/secrets/istio/root-cert.pem"
},
"xds_address": "https://istiod.istio-system.svc:15012",
"xds_root_cert": {
"File": "./var/run/secrets/istio/root-cert.pem"
},
"xds_on_demand": false,
"fake_ca": false,
"termination_grace_period": {
"secs": 5,
"nanos": 0
},
"proxy_metadata": {
"ISTIO_VERSION": "1.18.1"
},
"num_worker_threads": 2,
"enable_original_source": null,
"proxy_args": "proxy ztunnel"
},
...
}After analyzing, we can see that the configurations of Ztunnel components are relatively easy to understand, which aligns with the community's expectation of Ztunnel being a single-function and simple component. Therefore, in the configuration of Ztunnel, apart from its own meta properties, it mainly includes workload and service information, security-related certificates, and policy configurations.
Configure Sidecar Parameters to Balance gRPC/HTTP2 Streaming Performance and Resource Usage
Implement a New Service Mesh Integrating Sidecarless and Sidecar Modes
506 posts | 48 followers
FollowAlibaba Cloud Native Community - December 11, 2023
Alibaba Cloud Native Community - December 11, 2023
Alibaba Cloud Native Community - December 18, 2023
Alibaba Cloud Native - November 16, 2023
Xi Ning Wang(王夕宁) - July 21, 2023
Alibaba Container Service - August 16, 2024
506 posts | 48 followers
Follow
Alibaba Cloud Service Mesh
Alibaba Cloud Service Mesh (ASM) is a fully managed service mesh platform that is compatible with Istio.
Learn More
Cloud-Native Applications Management Solution
Accelerate and secure the development, deployment, and management of containerized applications cost-effectively.
Learn More
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn More
PolarDB for Xscale
Alibaba Cloud PolarDB for Xscale (PolarDB-X) is a cloud-native high-performance distributed database service independently developed by Alibaba Cloud.
Learn MoreMore Posts by Alibaba Cloud Native Community