All Products
Search
Document Center

Alibaba Cloud Service Mesh:Enable control-plane log collection and log-based alerting in an ASM instance of a version earlier than 1.17.2.35

Last Updated:Feb 22, 2024

Service Mesh (ASM) allows you to collect control-plane logs and sends you alert notifications based on the log data. For example, you can collect logs related to configuration pushes from the control plane of an ASM instance to sidecar proxies on the data plane. This topic describes how to enable control-plane log collection and log-based alerting.

Background information

One of the main features of the components on the control plane of an ASM instance is to push configurations to the sidecar proxies or gateways on the data plane. If configuration conflicts occur, the sidecar proxies or gateways cannot receive the configurations. In such cases, the sidecar proxies or gateways may continue to run based on the configurations they have previously received. However, the sidecar proxies or gateways are likely to fail if the pods where they reside are restarted. In many practical situations, sidecar proxies or gateways become unavailable due to improper configurations. Therefore, we recommend that you enable log-based alerting to detect and resolve issues in a timely manner.

Prerequisites

An ASM instance is created and the version of the instance is earlier than 1.17.2.35. For more information, see Create an ASM instance.

Enable control-plane log collection

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.

  3. In the Config Info section of the page that appears, click Enable next to Control-plane log collection.

    Important

    The control-plane Logstore stores the logs of the last 30 days and automatically discards earlier logs.

    • If this is the first time you enable control-plane log collection, the Enable Control-plane log collection dialog box appears. Create a project or select an existing project.

      When you create a project, you can choose to use either the default project name or a custom one. Then, click Submit.

      启用控制面日志

    • If you have previously enabled and disabled control-plane log collection, a Note message appears. Click OK. The project that was used last time is automatically selected.

    After you enable control-plane log collection, you can view detailed control-plane logs by clicking View log next to Control-plane log collection in the Basic Information section.

Enable log-based alerting

Important

Before you enable log-based alerting, you must enable control-plane log collection.

If a discovery services (xDS) request sent from the control plane to the data plane is rejected by the data plane, an alert that indicates failed synchronization to the data plane is triggered. In this case, sidecar proxies or gateways on the data plane cannot obtain the latest configurations and run differently depending on the situation:

  • If the sidecar proxies or gateways have received configurations before, they run based on the last received configurations.

  • If the sidecar proxies or gateways have not received any configurations before, they have no listeners configured and are not able to process or forward requests based on routing rules.

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.

  3. In the Config Info section of the page that appears, click Alert Setting next to Control-plane log collection.

  4. In the Control-Plane Alert Setting dialog box, find the alert policy that you want to enable, select ASM Built-in Action Strategy (recommended) or select a custom action policy, and then click Enable Alert.

    An action policy specifies the action to be performed when an alert is triggered. You can create and modify action policies in Simple Log Service projects. For more information, see Create an action policy.

  5. In the Note message that appears, click OK.

Configure alert contacts

You can configure built-in action policies for gateways, alert contacts, and notification templates in Simple Log Service.

  1. Log on to the Simple Log Service console.

  2. In the Projects section, click the name of the desired project. In the left-side navigation pane, click Alerts.

  3. On the Alert Center page, choose Notification Objects > User Group Management.

  4. On the User Group Management tab, find sls.app.asm.builtin and click Edit in the Actions column.

  5. In the Edit User Group dialog box, select the members that you want to add, click the 添加 icon to add the members to the user group, and then click OK. 修改用户组

Verify the alert notification settings

Note

This topic does not cover the alert notification settings for all alerts. In the following example, incorrect configurations are set to trigger an alert. For more information about error messages, see Alerts triggered when configuration pushes from the control plane to the data plane fail due to configuration errors.

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Gateway. On the page that appears, click Create from YAML.

  3. On the Create page, select a namespace and a template, configure a YAML file, and then click Create.

    In this example, the default namespace is used to configure an Istio gateway. Sample YAML code:

    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name:  gateway-test
      namespace: default
    spec:
      selector:
        istio: ingressgateway
      servers:
        - hosts:
            - '*console.aliyun.com'
          port:
            name: https
            number: 443
            protocol: HTTPS
          tls:
            credentialName: not-existing-credential
            mode: SIMPLE
  4. View alert notifications.

    1. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.

    2. In the Config Info section of the page that appears, click View log next to Control-plane log collection.

    3. In the Simple Log Service console, search for 'ACK ERROR' to view the alert information.

      If you have configured email notifications for alerts, you can view alert information by checking your emails.

Handle alerts

Alerts triggered by potential configuration risks

If ASM detects configurations in your cluster that may cause unexpected results, alerts are reported. You can view the alerts on the Mesh Diagnosis page and follow the instructions on the page to correct the configurations.

Alerts triggered by incorrect configurations

If ASM detects incorrect configurations in your cluster that are likely to cause unexpected behaviors, alerts are reported. We recommend that you view such alerts on the Mesh Diagnosis page and follow the instructions on the page to correct the configurations as soon as possible.

Alerts triggered when configuration pushes from the control plane to the data plane fail due to configuration errors

The following table describes common error messages that may appear when configuration pushes from the control plane to the data plane fail and provides suggestions for troubleshooting the errors. If your error message is not included in the table, submit a ticket.

Error message

Suggestions

Internal:Error adding/updating listener(s) 0.0.0.0_443: Failed to load certificate chain from <inline>, only P-256 ECDSA certificates are supported

Indicates that clusters on the data plane do not support the certificate that you configured for the data plane. To resolve this issue, configure the P-256 ECDSA certificate for the data plane. For more information about how to reconfigure a certificate, see Use an ingress gateway to enable HTTPS.

Internal:Error adding/updating listener(s) 0.0.0.0_443: Invalid path: ****

Indicates that the path of the certificate that you configured for the data plane is invalid or that the specified certificate does not exist. Check whether the mount path of the certificate is the same as the path specified in the configurations of the gateway. For more information, see Use an ingress gateway to enable HTTPS.

Internal:Error adding/updating listener(s) 0.0.0.0_xx: duplicate listener 0.0.0.0_xx found

Indicates that duplicate listening ports are configured for your gateway. To resolve this issue, check your gateway and delete any duplicate ports.

Internal:Error adding/updating listener(s) 192.168.33.189_15021: Didn't find a registered implementation for name: '***'

Indicates that the EnvoyFilter-based reference *** for the 15021 listener patch cannot be found in sidecar proxies or ingress gateway services. To resolve this issue, delete the reference.

Internal:Error adding/updating listener(s) 0.0.0.0_80: V2 (and AUTO) xDS transport protocol versions are deprecated in grpc_service ***

Indicates that the xDS v2 protocol on the data plane will be deprecated soon. This is usually because the version of sidecar proxies on the data plane does not match that on the control plane. To resolve this issue, update sidecar proxies on the data plane. To do so, you must delete existing pods. Sidecar proxies of the latest version are automatically injected into the recreated pods.

Related operations

Modify the settings of a control-plane log project

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.

  3. In the Config Info section of the Base Information page, click Change Log Project on the right of Control-plane log collection. In the Change Log Project dialog box, modify the settings as needed and click Submit.