All Products
Search

This Product

    Simple Log Service:Use Logtail to collect container logs across Alibaba Cloud accounts

    Document Center

    Simple Log Service:Use Logtail to collect container logs across Alibaba Cloud accounts

    Last Updated:Sep 04, 2024

    This topic describes how to collect container logs from Container Service for Kubernetes (ACK) across Alibaba Cloud accounts.

    Background information

    For example, an e-commerce enterprise has two e-commerce applications that are deployed on ACK clusters in the China (Hangzhou) region. The enterprise uses two Simple Log Service projects that reside in the China (Hangzhou) region to manage logs.

    • Application A is deployed on an ACK cluster that belongs to Alibaba Cloud Account A (12****456) and Simple Log Service is activated for the account to manage logs.

    • Application B is deployed on an ACK cluster that belongs to Alibaba Cloud Account B (17****397) and Simple Log Service is activated for the account to manage logs.

    The enterprise wants to use Simple Log Service that is activated for Alibaba Cloud Account A (12****456) to collect the logs of the two applications and store the logs in two Logstores of the same project. In this case, you must create a Logtail configuration, a machine group, and a Logstore to collect and store the logs of Application B. The Logtail configuration, machine group, and Logstore that are configured for Application A remain unchanged. Architecture

    Step 1: Configure the ID of an Alibaba Cloud account as a user identifier

    1. Log on to the ACK console with Alibaba Cloud Account B.

    2. Configure the ID of Alibaba Cloud Account A as a user identifier.

      1. In the left-side navigation pane, click Clusters.

      2. On the Clusters page, click the cluster that you want to manage.

      3. In the left-side navigation pane, choose Configurations > ConfigMaps.

      4. Set the Namespace parameter to kube-system. In the ConfigMap list, find alibaba-log-configuration and click Edit in the Actions column.

      5. In the Edit panel, configure the following configuration and click OK.

        Add the ID of Alibaba Cloud Account A to the log-ali-uid file, and then obtain the value of the log-machine-group parameter, for example, k8s-group-cc47****54428. When you create a machine group, specify the value for the Custom Identifier parameter.

        Separate multiple account IDs with commas (,). Example: 17****397,12****456.

        Parameter

    3. Restart logtail-ds for the settings to take effect.

      1. In the left-side navigation pane, choose Workloads > DaemonSets.

      2. In the DaemonSets list, find logtail-ds and click Edit in the Actions column.

      3. In the Environment Variable section, click Add.

      4. Add a custom variable and specify an arbitrary key-value pair, for example, random_id: 439157431651471905349.

        logtail-ds

      5. Click Update.

      On the details page of logtail-ds, check whether each container pod is in the Running state and whether the time when each pod is created is the same as the time when you update the settings.

      Restart logtail-ds

    Step 2: Create a machine group

    1. Log on to the Log Service console with Alibaba Cloud Account A.

    2. In the Projects section, click the project that you want to manage.

    3. In the left-side navigation pane, choose Resources > Machine Groups.

    4. On the Machine Groups tab, choose Machine groups > Create Machine Group.

    5. In the Create Machine Group panel, configure the parameters and click OK, as shown in the following figure.

      In the Custom Identifier field, enter the machine group identifier that you obtained in Step 1: Configure the ID of an Alibaba Cloud account as a user identifier, for example, k8s-group-cc47****54428. For information about other parameters, see Create a custom identifier-based machine group. Machine groups

    6. Check whether the heartbeat status of each server in the machine group is OK.

      1. In the Machine Groups list, click the machine group that you created.

      2. On the Machine Group Settings page, view the status of each Elastic Compute Service (ECS) instance.

        If the Heartbeat status is OK, the ECS instance is connected to Simple Log Service. If the status is FAIL, see What do I do if a Logtail machine group has no heartbeats? Heartbeat status of a machine group

    Step 3: Create a Logtail configuration

    1. Log on to the Log Service console with Alibaba Cloud Account A.

    2. In the Import Data section, click Kubernetes - Object.

    3. Select a project and a Logstore. Then, click Next.

    4. Click Use Existing Machine Groups.

    5. Select the machine group that you created in Step 2: Create a machine group, move the machine group from the Source Server Groups section to the Applied Server Groups, and then click Next.

    6. Configure the parameters for the Logtail configuration and click Next.

      For information about the parameters, see Use the Simple Log Service console to collect container text logs in DaemonSet mode.

      Important

      • By default, you can use only one Logtail configuration to collect logs from a log file. The collection process of Logtail in Alibaba Cloud Account B is not stopped. In this case, the Logtail configuration of Alibaba Cloud Account A cannot take effect. To ensure that the Logtail configuration of Alibaba Cloud Account A takes effect, you can use one of the following methods:

      • After you create the Logtail configuration, delete the existing Logtail configuration of Alibaba Cloud Account B to prevent duplicate log collection. For more information, see Delete Logtail configurations.

    7. Preview data, configure indexes, and then click Next.

      By default, Simple Log Service enables full-text indexing. You can configure field indexes based on the logs that are collected in manual mode or automatic mode. For more information, see Create indexes

    Related operations

    If you want to migrate historical data from Alibaba Cloud Account B to the current Logstore, you can create a data transformation job in the original Logstore, and then replicate the data to the current Logstore. For more information, see Replicate data from a Logstore.

    Important

    If you create a data transformation job to transform data across Alibaba Cloud accounts, you must use a custom role or an AccessKey pair to grant the required permissions for the job. In this example, a custom role is used.