Enable Mesh Topology to improve observability - Alibaba Cloud Service Mesh

You can enable the Mesh Topology feature to monitor interactions between services in a visualized manner. The Mesh Topology feature allows you to monitor the health status of services in real time, analyze latency of service-to-service communication and other communication issues, and obtain information about the distribution and flow of traffic among services. This allows you to quickly identify and resolve issues and guarantee reliability and high performance of services.

Feature introduction

As an observability tool of Service Mesh (ASM), Mesh Topology provides powerful visualization of ASM traffic. Mesh Topology combines real-time request traffic with ASM configuration information to provide instant insights into service calls and service health status and help you quickly pinpoint issues. Mesh Topology provides the following capabilities:

Traffic call topology: Mesh Topology provides powerful visualization of mesh traffic and displays the real-time request traffic in a topology.
Mesh configuration visualization: Mesh Topology displays related configurations on the nodes in the topology based on the configurations of ASM. This helps you quickly know the configurations that take effect such as peer authentication, virtual services, destination rules, throttling, and circuit breaking.
Health status visualization: Mesh Topology colors nodes or edges to indicate the health status of services or requests. This helps you quickly find the abnormal nodes or requests.
Traffic replay: Traffic replay allows you to replay the traffic conditions in a selected time period in the past. This allows you to trace application traffic in a specific period.

Step 1: Enable Mesh Topology

Scenario 1: Enable Mesh Topology for a new ASM instance

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click Create ASM Instance.

In the Observability section, select Enable Managed Service for Prometheus, and then select operations based on your Prometheus instance type.

Prometheus instance type	Operation
Managed Service for Prometheus instance	Select Use Managed Service for Prometheus to Collect Metrics, and then select Enable ASM Mesh Topology. For more information about how to integrate Managed Service for Prometheus to monitor service meshes and the billing of Managed Service for Prometheus, see Integrate Managed Service for Prometheus to monitor service meshes and Comparison between the previous and current prices.
Self-managed Prometheus instance	Select Integrate the self-built Prometheus to achieve metrics monitoring and Enable ASM Mesh Topology, and then enter the HTTP API address of the self-managed Prometheus instance that is used by Mesh Topology. For more information about how to integrate a self-managed Prometheus instance, see Monitor ASM instances by using a self-managed Prometheus instance.

Note

After you select Enable Managed Service for Prometheus, only the metrics of the Service Mesh instance are collected. No Application Real-Time Monitoring Service (ARMS) instances or self-managed Prometheus instances are automatically created.
If you clear both Enable Managed Service for Prometheus and Enable ASM Mesh Topology, Mesh Topology is disabled.

Configure related parameters, read and agree to Service Agreement, and then click Create Service Mesh at the bottom of the page.
For the descriptions of the configuration items, see Create an ASM instance.
Add a Container Service for Kubernetes (ACK) cluster to the ASM instance. For more information, see Add a cluster to an ASM instance.

Scenario 2: Enable Mesh Topology for an existing ASM instance

Make sure that an ACK cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.

For ASM instances whose versions are earlier than 1.12.4.50

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance that you want to manage. In the left-side navigation pane, choose ASM Instance > Base Information. In the upper right part of the page, click Settings.

In the Settings Update panel, select Enable Managed Service for Prometheus, and select operations based on your Prometheus instance type.

Prometheus instance type	Operation
Managed Service for Prometheus instance	Select Use Managed Service for Prometheus to Collect Metrics, and then select Enable ASM Mesh Topology. For more information about how to integrate Managed Service for Prometheus to monitor service meshes and the billing of Managed Service for Prometheus, see Integrate Managed Service for Prometheus to monitor service meshes and Comparison between the previous and current prices.
Self-managed Prometheus instance	Select Integrate the self-built Prometheus to achieve metrics monitoring and Enable ASM Mesh Topology, and then enter the HTTP API address of the self-managed Prometheus instance that is used by Mesh Topology. For more information about how to integrate a self-managed Prometheus instance, see Monitor ASM instances by using a self-managed Prometheus instance.

After the configuration is complete, in the lower part of the Settings Update panel, click OK.

For ASM instances whose versions are 1.12.4.50 or later

Note

Mesh Topology requires ASM to integrate Managed Service for Prometheus to monitor metrics. If ASM has not been integrated with a Prometheus instance to monitor service meshes, integrate ASM with a Prometheus instance first. For more information, see Integrate Managed Service for Prometheus to monitor service meshes and Monitor ASM instances by using a self-managed Prometheus instance. For more information about the billing of Managed Service for Prometheus, see Comparison between the previous and current prices.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance that you want to manage. In the left-side navigation pane, choose Observability Management Center > Mesh Topology. In the Enable ASM Mesh Topology section, perform operations based on the version of your ASM instance.
- If the version of your ASM instance is earlier than 1.18.2.112, click To enable.
- If the version of your ASM instance is 1.18.2.112 or later, select either of the following deployment modes. For more information about the differences between the two modes and how to configure the managed mode, see Enable Mesh Topology in managed mode.
  - Click In-Kubernetes-cluster Mode, enter the HTTP API URL of the Prometheus instance used by ASM Mesh Topology, and then click To enable.
  - Click Managed Mode and then click To enable. In the dialog box that appears, configure the required parameters and click OK.

Step 2: Open the logon page of Mesh Topology

You can access Mesh Topology by using the following methods. To support direct access to Mesh Topology, the ASM version must be 1.12.4.50 or later. To support custom access to Mesh Topology, the ASM version must be 1.16.4.5 or later.

Method 1: Directly access Mesh Topology

If the version of your ASM instance is 1.12.4.50 or later, you can create a Classic Load Balancer (CLB) instance to access ASM Mesh Topology.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance that you want to manage. In the left-side navigation pane, choose Observability Management Center > Mesh Topology.
In the Access section, perform one of the following operations based on the version of your ASM instance:
- If the version of your ASM instance is earlier than 1.15.3.120, select Automatically Create an Internet-facing CLB Instance to Access ASM Mesh Topology.
- If the version of your ASM instance is 1.15.3.120 or later but is earlier than 1.17.2.19, turn on Automatically Create an Internet-facing CLB Instance to Access ASM Mesh Topology. In the dialog box that appears, click OK.
- If the version of your ASM instance is 1.17.2.19 or later, turn on Create a CLB Instance to Access ASM Mesh Topology. In the dialog box that appears, configure the required parameters and click OK.
After Mesh Topology access is enabled, the ASM instance will enter the update state for a short period.
Check whether the ASM instance update is complete.
1. In the left-side navigation pane, choose ASM Instance > Base Information.
2. In the Basic Information section, check the Status of the ASM instance.
  If the Status is Updating, the ASM instance update is in progress. If the Status is Running, the update is complete.
In the Access section of the Mesh Topology page, click Click here to access ASM Mesh Topology or Copy token and open in new window to open the logon page of Mesh Topology.

Method 2: Use an ASM gateway to access Mesh Topology

Create an ingress gateway and configure a port for the ingress gateway to access Mesh Topology. For more information, see Create an ingress gateway or Create and manage an ingress gateway by using the Kubernetes API.
YAML example of port configurations:
```
- name: http-kiali
  port: 20001
  protocol: TCP
  targetPort: 20001
```

Use the following YAML template to create an Istio gateway. For more information, see Manage Istio gateways.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kiali-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 20001
        protocol: HTTP

Use the following YAML template to create a virtual service. For more information, see Manage virtual services.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kiali-vs
  namespace: istio-system
spec:
  gateways:
    - kiali-gateway
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: kiali
            port:
              number: 20001

Open the logon page of Mesh Topology.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance that you want to manage. In the left-side navigation pane, choose ASM Instance > Base Information.
3. In the Config Info section, click Access from Ingress Gateway on the right of Enable ASM Mesh Topology to open the logon page of Mesh Topology.

Method 3: Customize access to Mesh Topology

After you enable Mesh Topology, Mesh Topology creates a service named Kiali in the istio-system namespace of the Kubernetes cluster. You can create a proxy service for this service and then use the proxy service to access Mesh Topology. For example, you can create an NGINX reverse proxy for the Kiali service and use NGINX to access Mesh Topology. You can specify the method of accessing the Kiali service by configuring the parameters in the Customize Access Modes section. To use this method, the ASM version must be 1.16.4.5 or later. For more information about how to update an ASM instance, see Update an ASM instance.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Mesh Topology.

In the lower part of the Authentication section, click Show Advanced Settings. In the Customize Access Modes section, configure relevant information and click Save configuration of Mesh Topology.

Parameter	Description
Customize a Domain Name	The custom domain name of Mesh Topology. If you use a custom domain name to access Mesh Topology, we recommend that you configure this parameter. Otherwise, the OpenID Connect (OIDC)-based logon process of Mesh Topology may be affected. When you log on to Mesh Topology by using OIDC, Mesh Topology replaces the domain name in the Redirect URI with the custom domain name. This ensures a successful connection between Mesh Topology and the OIDC application.
Customize a Port	The port on which Mesh Topology listens. Default value: 20001. After you specify this parameter, Mesh Topology listens on the specified port.
Customize a Service Root Path	The root path of Mesh Topology. Default value: /. After you specify this parameter, Mesh Topology will provide services with the specified root path. To access Mesh topology, you must add the custom root path to the original address.
Select a Protocol (HTTPS/HTTP)	The protocol used to access Mesh Topology. Valid values: http and https. If you use HTTPS to access Mesh Topology, we recommend that you configure this parameter. Otherwise, the OIDC-based logon process of Mesh Topology may be affected. When you log on to Mesh Topology by using OIDC, Mesh Topology replaces the protocol in the Redirect URI with the specified protocol. This ensures a successful connection between Mesh Topology and the OIDC application.

Step 3: Log on to Mesh Topology

You can log on to Mesh Topology by using the following methods. If you use OIDC for logon, the version of the ASM instance must be 1.15.3.120 or later. If you use an Alibaba Cloud account for logon, the version of the ASM instance must be 1.16.4.5 or later.

Method 1: Log on to Mesh Topology by using a token

For ASM instances whose versions are earlier than 1.12.4.50

You can log on to the ACK console to obtain a token and then log on to Mesh Topology by using the token.

Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click the name of the cluster that you want to manage and choose Configurations > Secrets in the left-side navigation pane.
On the Secrets page, select istio-system from the Namespace drop-down list. Click kiali-service-account-token-**** in the Name column. On the page that appears, click the icon in the token row to copy the token.
On the logon page of Mesh Topology, paste the token and click Login to open the Mesh Topology console.

For ASM instances whose versions are 1.12.4.50 or later

You can log on to the ASM console to obtain a token and then log on to Mesh Topology by using the token.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance that you want to manage. In the left-side navigation pane, choose Observability Management Center > Mesh Topology.
In the Access section of the Mesh Topology page, click Copy token and open in new window.
On the logon page of Mesh Topology, paste the token and click Login to open the Mesh Topology console.

Method 2: Log on to Mesh Topology by using OIDC

OIDC is an identity authentication and authorization protocol, which is commonly used to implement Single Sign On (SSO). You can connect to an identity provider (IdP) over the OIDC protocol and then log on to Mesh Topology. If you want to use OIDC, the version of the ASM instance must be 1.15.3.120 or later. For more information about how to update an ASM instance, see Update an ASM instance.

Configure the IdP. For more information, see Step 1 and Step 2 in Integrate Alibaba Cloud IDaaS with ASM to implement single sign-on.
Set Redirect URIs to the IP address of the CLB instance if you want to access Mesh Topology by using the CLB instance. Alternatively, set this parameter to the IP address of the ASM gateway if you want to access Mesh Topology by using the ASM gateway. Configuration examples:
- If you use a CLB instance to access Mesh Topology and the IP address of the CLB instance is xxx.xxx.xxx.xxx, set the Redirect URIs parameter to http://xxx.xxx.xxx.xxx:20001 without forward slashes (/) following 20001. For example, 20001/xxx, 20001/, 20001/xxx/, and 20001/xxx/yyy are not allowed.
- If you use an ASM gateway to access Mesh Topology and the IP address of the ASM gateway is yyy.yyy.yyy.yyy, set the Redirect URIs parameter to http://yyy.yyy.yyy.yyy:20001 without forward slashes (/) following 20001. For example, 20001/xxx, 20001/, 20001/xxx/, and 20001/xxx/yyy are not allowed.
Log on to Mesh Topology by using OIDC.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance that you want to manage. In the left-side navigation pane, choose Observability Management Center > Mesh Topology.
3. In the Authentication section, select Login with OIDC and configure the related parameters. Click Save configuration of Mesh Topology. Then, click Click here to access ASM Mesh Topology.
4. On the logon page of Mesh Topology, click Log In With OpenID to redirect to the logon page of the OIDC application.

Method 3: Log on to Mesh topology by using an Alibaba Cloud account

You can use the OIDC protocol to connect to the Alibaba Cloud RAM OAuth application. This way, you can log on to Mesh Topology by using your Alibaba Cloud account. For more information, see Access Alibaba Cloud APIs from a web application.

If you use this method, ASM automatically creates an Alibaba Cloud RAM OAuth application whose name starts with asm-kiali and integrates the application with Mesh Topology. To use this method, your ASM version must be 1.16.4.5 or later. For more information about how to update an ASM instance, see Update an ASM instance.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Mesh Topology.
In the Authentication section, select Logon by Using an Alibaba Cloud Account. In the dialog box that appears, click Add a Redirect URI, enter a callback URL, and then click OK.
The ASM instance will be updated for a short period. Wait until the update is complete before you proceed to the next step.
Note
- For more information about the format of callback URLs, see Redirect URI in Method 2: Log on to Mesh Topology by using OIDC.
- If you use the method of directly accessing Mesh Topology, the callback URL is automatically specified.
On the logon page of Mesh Topology, click Log In With OpenID to log on as an Alibaba Cloud account.

FAQ

Why did Mesh Topology fail to load when Managed Service for Prometheus was used to collect metrics?

If you have selected Use ARMS Prometheus to collect monitoring indicators for your ASM instance and enabled token-based authentication for the Managed Service for Prometheus instance, Mesh Topology may not be able to access your Managed Service for Prometheus instance. As a result, Mesh Topology could not be loaded. You can solve this issue by using the following methods. For more information, see Import data from Managed Service for Prometheus to a local Grafana system.

Method 1: Disable token for the Managed Service for Prometheus instance on the Settings page of the ARMS console. For more information, see HTTP API URLs.
Method 2: Configure a Managed Service for Prometheus authentication token for Mesh Topology to access the Managed Service for Prometheus instance.
Note
This operation requires the version of the ASM instance to be 1.15.3.120 or later.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Mesh Topology.
3. In the text field under Connection with ARMS Prometheus, enter the authentication token configured for Managed Service for Prometheus and click Save configuration of Mesh Topology to configure an authentication token for Mesh Topology to access the Managed Service for Prometheus instance.

Why can I view only part of the traffic in Mesh Topology?

You can check the following possible causes to troubleshoot this issue.

Configurations
Check whether you have selected the correct types of traffic, such as gRPC, HTTP, and TCP, in Mesh Topology. If you do not select these options, you cannot view all traffic information.
Namespace
In Mesh Topology, check whether the correct namespace is selected. Some traffic may exist in a namespace that is not selected.
Data collection and update
- Check whether metric collection of the Prometheus instance is normal. ASM may display only the data that has been successfully collected. Some traffic data may not be displayed on the topology due to various reasons, such as latency and packet loss.
- Check whether all workloads in the cluster are injected with sidecar proxies. Only the workloads that are injected with sidecar proxies report metric data of traffic.
- Check whether the Mesh Topology page is refreshed in real time or manually refresh the Mesh Topology page to obtain the latest traffic data.
Sidecar proxy configurations and sidecar traffic configurations
Due to sidecar proxy configurations or sidecar traffic configurations in ASM, some traffic does not pass through sidecar proxies and cannot be observed. Check whether traffic is configured to bypass sidecar proxies by port or IP address in sidecar proxy configurations, or whether the desired inbound and outbound traffic is correctly configured in sidecar traffic configurations.

References

If you find that some requests experience high response latency, you can use access logs to locate the cause of high latency. For more information, see Locate the cause of high response latency by using access logs in ASM.
If you want to obtain the lowest service call latency, you can use the zone aware routing feature to allow a client to call a destination service deployed in the same zone first. For more information, see Verify the zone aware routing feature on the topology of an ASM instance.
If you want to monitor the traffic topologies of multiple clusters, you can use the Mesh Topology feature in managed mode. In managed mode, you need to deploy only one Mesh Topology service for the entire ASM instance. This reduces the configuration workload. For more information, see Enable Mesh Topology in managed mode.
The request interception mechanism of sidecar proxies may interfere with health checks of applications in an ASM instance and cause the checks to fail. You can enable redirection of health check requests to ensure that the checks are performed correctly. For more information, see Configure redirection of HTTP or TCP health check requests for applications in an ASM instance.