All Products
Search

This Product

    AnalyticDB：Use Log Service to transfer log data to AnalyticDB for MySQL

    Document Center

    AnalyticDB：Use Log Service to transfer log data to AnalyticDB for MySQL

    更新時間：Oct 10, 2024

    Logs are a special type of data that plays an important role in processing historical data and diagnosing problems. Logs are essential data sources for data analysts, developers, and O&M personnel. Alibaba Cloud allows you to use Log Service to synchronize logs such as NGINX access logs, Log4j logs, Apache access logs, and structured text to AnalyticDB for MySQL in real time. This allows you to use AnalyticDB for MySQL to perform real-time log analytics.

    Prerequisites

    • When you use Log Service for the first time, you must use your Alibaba Cloud account to log on to the Log Service buy page and click Get it Free. The system automatically redirects you to the buy page. Read and select I agree with Log Service Agreement of Service and click Enable Now. The Log Service console appears.

      Note

      If you have activated Log Service, you can start to create a project.

    • Before you use AnalyticDB for MySQL, make sure that the following requirements are met:

      1. An AnalyticDB for MySQL cluster is created. For more information, see Create a cluster.

      2. A database account is created. For more information, see Create a database account.

      3. A database is created. For more information, see Create a database.

      4. A public endpoint is applied for if you must use a public endpoint to connect to AnalyticDB for MySQL clusters. For more information, see Apply for or release a public endpoint.

      5. A table is created by using the CREATE TABLE statement to store the log data transferred to AnalyticDB for MySQL. For more information, see CREATE TABLE.

    Create a LogShipper task

    You can use the Export function in the Data Transformation module of Log Service to transfer the collected logs in Logstores to AnalyticDB for MySQL.

    1. Log on to the Log Service console.

    2. Create a project and a Logstore.

      In this example, a project named adb-test and a Logstore named adb-source are created. For more information, see Create a project and Create a Logstore.

      Note

      Log Service projects and AnalyticDB for MySQL clusters must be located in the same region because logs can be transferred only within one region.

    3. On the Logstores tab, find the Logstore and choose > > Data Transformation > Export > AnalyticDB.

      If you use Log Service to transfer log data to AnalyticDB for MySQL for the first time, you must grant access permissions on Log Service to AnalyticDB for MySQL.

      1. Click the + icon to the right of AnalyticDB. The system prompts you to complete authorization. Click permissions.

      2. On the Cloud Resource Access Authorization page, click Confirm Authorization Policy and assign the RAM role AliyunAnalyticDBAccessingLogRole to AnalyticDB for MySQL.

    4. After you complete authorization, click + to the right of AnalyticDB. You can click Ship to directly transfer log data. You can also click Go to Data Transformation to process log data before the data is transferred.

      Note

      The log data is transferred directly in this example.

    5. On the LogHub - Data Shipper page, configure the parameters.

      Table 1. LogHub - Data Shipper parameters

      Parameter

      Description

      Shipper Name

      The name of the LogShipper task. We recommend that you specify an identifiable name to facilitate subsequent management.

      Shipper Description

      The description of the LogShipper task. We recommend that you specify a meaningful description to facilitate subsequent management.

      Cluster Version

      The version of the AnalyticDB for MySQL cluster. Select 3.0.

      Cluster Name

      The name of the destination AnalyticDB for MySQL cluster.

      Database Name

      The name of the database in the destination AnalyticDB for MySQL cluster.

      Table Name

      The name of the table in the destination AnalyticDB for MySQL cluster.

      Account Name

      The name of the Create a database account used to connect to the destination AnalyticDB for MySQL cluster. You can use one of the following accounts:

      • Privileged accounts

      • Standard account

      Account Password

      The password of the database account used to connect to the destination AnalyticDB for MySQL cluster.

      Field Mapping

      Log Service automatically extracts all log fields of the last 10 minutes and maps these fields to the destination fields in AnalyticDB for MySQL.

      Delivery Start Time

      The start time of the LogShipper task.

      After log data is written into Log Service, Log Service can transfer the data to AnalyticDB for MySQL in real time.

      Filter Dirty Data

      Specifies whether to filter dirty data.

      • Disable: automatically interrupts the LogShipper task if dirty data is found.

      • Enable: filters dirty data.

        Dirty data refers to data whose data type fails to be converted or whose required fields are empty.

    6. After you configure the preceding parameters, click OK. Log Service will transfer log data to AnalyticDB for MySQL tables in real time at the specified delivery time.

    View log data

    After logs are transferred to AnalyticDB for MySQL, you can execute the SELECT statement to analyze log data. For more information, see Syntax.

    Manage a LogShipper task

    • You can click the task name to view the execution status of the LogShipper task. For example, you can view the amount of data that is transferred and fails to be transferred and delivery latency.

    • To modify a LogShipper task, click Modify LogShipper.

    • To restart a LogShipper task, first click Stop. After the LogShipper task is stopped, click Start to restart the LogShipper task.