All Products
Search
Document Center

Simple Log Service:Build a service to collect logs from mini programs to Simple Log Service

更新時間:Aug 28, 2024

To gain insights into user behavior, enterprises collect logs from mini programs and analyze key data such as user devices and browsing history. This way, enterprises can improve features and optimize user experience. After the web tracking feature of Alibaba Cloud Simple Log Service is enabled, enterprises can automatically collect user logs from mini programs to Simple Log Service. This improves data processing efficiency and reduces the burden on application servers. After you enable the web tracking feature for a Logstore, the write permissions on the Logstore are granted to anonymous users. This can result in dirty data from configuration data leaks of mini programs. To reduce this risk and enhance data upload security, we recommend that you use Security Token Service (STS) to provide mini programs with tokens that are valid within a specified period of time and allow only specific operations. This topic describes how to use Simple Log Service and STS to build a secure and efficient direct data transfer service for mini programs to collect browsing behavior logs from mini programs to Simple Log Service.

Background information

Enterprise A develops a mini program that provides users with an online, feature-rich, and user-friendly platform. This platform is designed to attract users to browse and purchase products and services. Enterprise A acknowledges the importance of in-depth analysis on user behavior for better understanding user requirements, optimizing user experience, and further improving the conversion rate of the mini program. The user behavior includes clicks, swipes, dwelling, search habits, and purchases. Therefore, the technical team of Enterprise A collects logs from the mini program to Simple Log Service for efficient data analysis and processing.

Security risks

To reduce the burden on application servers and improve the efficiency of data processing, the mini program is designed to directly upload user logs to Simple Log Service without the need for data transfer and data collection to application servers.

The following figure shows how Enterprise A plans to build a direct data transfer service to collect mini program logs to Simple Log Service.

image

After the web tracking feature is enabled for a Simple Log Service Logstore, the write permissions on the Logstore are granted to anonymous users. In this case, the following risks may occur:

  • Dirty data generation: Malicious users may upload false or sabotage data, which compromises data quality and the accuracy of analysis results.

  • Service abuse: Unlimited write operations may be abused to initiate Denial of Service (DoS) attacks. Attackers may initiate a large number of write requests to exhaust resources.

Solution

To handle the risks, Enterprise A adds temporary access authorization. This way, Enterprise A can ensure the efficiency of direct data transfer and achieve the following benefits:

  • Enhanced authentication and authorization: STS generates tokens that are valid within a specified period of time. This significantly reduces security risks even if the tokens are leaked because the tokens quickly become invalid.

  • Fine-grained access control: STS allows you to configure permissions based on the principle of least privilege. You can grant only the required permissions to the mini program. This helps narrow the impact scope of token leaks and avoids granting excessive permissions to the mini program.

The following figure shows how Enterprise A builds a direct data transfer service to collect mini program logs to Simple Log Service.

image

Solution deployment

This section describes how to use Simple Log Service and STS to build a direct data transfer service for the mini program to collect browsing behavior logs from the mini program to Simple Log Service.

Manual deployment

Preparations

Before you start the deployment, you must create cloud resources.

  • Create a Simple Log Service project and a Logstore to store and analyze logs that are collected from the mini program.

    Parameter

    Example value

    Region

    China (Shanghai)

    Project Name

    sls-webtracking-mini

    Logstore Name

    web-tracking-logstore

    For more information, see Step 1: Create a project and a Logstore.

  • Create an Elastic Compute Service (ECS) instance as your application server to generate STS tokens.

    Note

    In your actual deployment, you do not need to create the ECS instance. You can directly integrate STS API into your application server.

    Parameter

    Example value

    Billing Method

    Pay-as-you-go

    Region

    China (Shanghai)

    Public IP Address

    Assign Public IPv4 Address

    Security Group

    HTTP-TCP:80-open

    For more information, see Create and manage an ECS instance in the console (express version).

Procedure

To manually deploy the project, perform the following steps:

Step 1: Create a RAM user in the RAM console

Create a Resource Access Management (RAM) user and obtain the AccessKey pair of the RAM user. When you create the RAM user, set Access Mode to OpenAPI Access. The AccessKey pair is used as a long-term access credential for accessing and managing your application server.

  1. Log on to the RAM console by using an Alibaba Cloud account or an account administrator.

  2. In the left-side navigation pane, choose Identities > Users.

  3. On the Users page, click Create User.

  4. Configure the Logon Name and Display Name parameters.

  5. In the Access Mode section, select OpenAPI Access. Then, click OK.

  6. Click Copy in the Actions column and save the AccessKey pair of the RAM user.

Step 2: Grant the RAM user the permissions to call the AssumeRole operation in the RAM console

After the RAM user is created, grant the RAM user the permissions to call the AssumeRole operation of STS. This way, the RAM user can assume a RAM role to obtain STS tokens.

  1. In the left-side navigation pane, choose Identities > Users.

  2. On the Users page, find the RAM user and click Add Permissions in the Actions column.

  3. In the Grant Permission panel, select the AliyunSTSAssumeRoleAccess system policy.

    Note

    To grant a RAM user the permissions to call the AssumeRole operation, you must attach the AliyunSTSAssumeRoleAccess system policy to the RAM user. This policy does not grant the permissions to obtain STS tokens or initiate Simple Log Service requests by using the STS tokens.

  4. Click Grant permissions.

Step 3: Create a RAM role in the RAM console

Create a RAM role that the RAM user can assume to obtain STS tokens and obtain the Alibaba Cloud Resource Name (ARN) of the RAM role. For more information, see ARN.

  1. In the left-side navigation pane, choose Identities > Roles.

  2. On the Roles page, click Create Role. In the Select Role Type step of the Create Role wizard, set Select Trusted Entity to Alibaba Cloud Account and click Next.

  3. In the Configure Role step of the Create Role wizard, enter a name in the RAM Role Name field and set Select Trusted Alibaba Cloud Account to Current Alibaba Cloud Account.

  4. Click OK. After the RAM role is created, click Close.

  5. On the Roles page, enter the role name in the search box to search for the RAM role. Example: sls-web-tracking.

  6. On the details page of the RAM role, click Copy to save the ARN of the RAM role.

    2024-05-09_17-45-50.png

Step 4: Create a custom policy in the RAM console

Create a custom policy based on the principle of least privilege. The custom policy grants the RAM role the permissions to upload logs only to a specified Simple Log Service Logstore.

  1. In the left-side navigation pane, choose Permissions > Policies.

  2. On the Policies page, click Create Policy.

  3. On the Create Policy page, click the JSON tab. Replace <Project name> and <Logstore name> in the following policy document with the names of the project and Logstore that are created in the "Preparations" section of this topic.

    Important

    The following policy document is provided only for reference. You must configure fine-grained RAM policies based on your business requirements to avoid granting excessive permissions to the RAM role. For more information about how to configure fine-grained RAM policies, see Examples of using custom policies to grant permissions to a RAM user.

    {
      "Version":"1",
      "Statement":[
        {
          "Effect":"Allow",
          "Action":[
            "log:PostLogStoreLogs"
          ],
          "Resource":[
            "acs:log:*:*:project/<Project name>/logstore/<Logstore name>"
          ]
        }
      ]
    }
  4. After you configure the policy, click Next to edit policy information.

  5. In the Basic information section, specify a policy name. Then, click OK.

Step 5: Attach the custom policy to the RAM role in the RAM console

Attach the custom policy to the RAM role. This way, the RAM user can assume the RAM role to perform the operations that are allowed by the policy.

  1. In the left-side navigation pane, choose Identities > Roles.

  2. On the Roles page, find the RAM role and click Grant Permission in the Actions column.

  3. In the Policy section of the Grant Permission panel, select Custom Policy from the policy type drop-down list and select the custom policy.

  4. Click Grant permissions.

Step 6: Obtain STS tokens from your application server

Access your mini program and integrate STS SDK into your application server to implement the /get_sts_token operation. For more information, see STS SDK overview. When you call the /get_sts_token operation by using the HTTP GET method, the system generates and returns an STS token.

The following procedure describes how to build a mini program by using the Flask framework on your ECS instance and obtain STS tokens from the application server:

  1. Connect to the ECS instance.

    For more information, see Create and manage an ECS instance in the console (express version).

  2. Install Python 3. For more information, see Python 3.

  3. Configure the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables for the RAM user. For more information, see Configure environment variables in Linux, macOS, and Windows.

  4. Create a project folder and switch to the project directory.

    mkdir my_web_sample
    cd my_web_sample
  5. Install dependencies.

    pip3 install Flask
    pip3 install attr
    pip3 install yarl
    pip3 install async_timeout
    pip3 install idna_ssl
    pip3 install attrs
    pip3 install aiosignal
    pip3 install charset_normalizer
    pip3 install alibabacloud_tea_openapi
    pip3 install alibabacloud_sts20150401
    pip3 install alibabacloud_credentials
  6. Write backend code.

    1. Create a file named main.py.

    2. Add the following Python code to the file:

      import json
      from flask import Flask, render_template
      from alibabacloud_tea_openapi.models import Config
      from alibabacloud_sts20150401.client import Client as Sts20150401Client
      from alibabacloud_sts20150401 import models as sts_20150401_models
      from alibabacloud_credentials.client import Client as CredentialClient
      
      app = Flask(__name__)
      
      # Replace <YOUR_ROLE_ARN> with the ARN of the RAM role. 
      role_arn_for_oss_upload = '<YOUR_ROLE_ARN>'
      # Specify the region of STS. Example: cn-shanghai. 
      region_id = 'cn-shanghai'
      
      @app.route('/get_sts_token', methods=['GET'])
      def get_sts_token():
          # If you do not specify parameters when you initialize CredentialClient, the default credential chain is used. 
          # If you run the file on your computer, you can specify the AccessKey pair by configuring the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables.
          # If you run the file on ECS, Elastic Container Instance (ECI), or Container Service for Kubernetes (ACK), you can specify the role of the bound instance by configuring the ALIBABA_CLOUD_ECS_METADATA environment variable. The SDK automatically obtains STS tokens. 
          config = Config(region_id=region_id, credential=CredentialClient())
          sts_client = Sts20150401Client(config=config)
          assume_role_request = sts_20150401_models.AssumeRoleRequest(
              role_arn=role_arn_for_oss_upload,
              # Replace <YOUR_ROLE_SESSION_NAME> with the custom name of the session. 
              role_session_name='<YOUR_ROLE_SESSION_NAME>'
          )
          response = sts_client.assume_role(assume_role_request)
          token = json.dumps(response.body.credentials.to_map())
          return token
      
      
      app.run(host="0.0.0.0", port=80)
    3. Replace <YOUR_ROLE_ARN> with the ARN that is obtained in Step 3.

    4. Replace <YOUR_ROLE_SESSION_NAME> with the custom name of the session. Example: role_session_test.

  7. Run the main.py file.

    python3 main.py
  8. Visit http://<Public IP address of the ECS instance>/get_sts_token in your browser.

    The following figure shows a success response.

    sts token.png

Step 7: Use an STS token in your mini program to upload logs to Simple Log Service

Important

For more information about the development process and code structure of mini programs, see Create a mini program in the documentation center of Alipay. In this example, you can write the client-side code of the mini program based on Todo App and Use the STS plug-in of web tracking SDK for JavaScript to upload logs.

After you implement the /get_sts_token operation on your application server, integrate the @aliyun-sls/web-track-mini instrumentation SDK and the @aliyun-sls/web-sts-plugin STS plug-in of Simple Log Service into the app.js file of the mini program. This helps monitor user behavior in real time and upload related logs.

When you create a tracker, the tracker automatically calls the /get_sts_token operation to obtain an STS token. This way, the user behavior data generated during mini program interactions can be securely transferred to Simple Log Service. The user behavior data includes the data of logons, product browsing, and order submission.

  1. Download a developer tool of mini programs. For more information, see Developer tools of mini programs.

  2. Create and use the Todo App mini program. For more information, see Todo App.

  3. Install npm.

  4. Install project dependencies.

    npm install --save @aliyun-sls/web-track-mini
    npm install --save @aliyun-sls/web-sts-plugin
  5. Modify the sample code of the Todo App mini program to upload user logs to the specified Logstore by using the web tracking feature. The following table describes the files that you need to modify.

    File

    Code modification description

    app.js

    • Call the initSlsTracker method in the onLaunch method. This way, when you start the mini program, the SlsTracker object is initialized.

    • Configure parameters such as project and logstore in the initSlsTracker method. For more information about the parameters, see Web tracking parameters.

    • Configure the stsOpt parameter and the refreshSTSToken function to dynamically obtain and update STS tokens. For more information about STS token-related parameters, see STS parameters.

    mini.project.json

    The @aliyun-sls/web-sts-plugin STS plug-in supports ECMAScript 6 (ES6). When you modify the mini.project.json file, set compileOptions.transpile to true. This way, the developer tool of the mini program translates ES6 code to ECMAScript 5 (ES5) code to ensure compatibility.

    todo.xml

    <view class="todo-footer">
        <add-button text="Add Todo" onClickMe="handleAddTodo"></add-button>
      </view>

    Define a button named Add Todo at the bottom of the user interface. You can use add-button to define the button. onClickMe="handleAddTodo" specifies that the system calls the handleAddTodo method defined in the todo.js file when you click the button.

    todo.js

    • Use the handleAddTodo() function to handle events. When you add a to-do item on the user interface, this function is triggered. The function performs the following operations:

      1. Call the this.addTodo() method to add a to-do item.

      2. Call the this.onAddTodoButtonClick() method to send a log message indicating that a user clicks the Add Todo button.

    • Use the addTodo() function to call the my.navigateTo function of the mini program. This way, you are navigated from the current page to the required page in the /pages/add-todo/add-todo directory of the mini program.

    • Use the onAddTodoButtonClick() method to call the send method of the SlsTracker object to send a log. The log content is eventType: 'addtodo_click'. The SlsTracker object is defined in the app.js file.

    add-todo.js

    Use the add method to perform the following operations:

    • Create an object named newTodo. The newTodo object specifies a to-do item, which contains the following attributes:

      • text: the content of the to-do item. You can obtain the content by calling the this.data.inputValue method. This call indicates that the content may come from user input.

      • completed: specifies whether the to-do item is completed. Initial value: false. This value specifies that the to-do item is not completed.

    • Call the send method of the SlsTracker object defined in the app.js file to send a log. The following code shows the content of the log:

      eventType: 'add_todo'
      todoText: newTodo.text

    add-button.js

    Define custom buttons. The onClickMe method first checks whether this.props.onClickMe is a function. If this.props.onClickMe is a function, the method calls the function.

  6. Configure parameters in the app.js file.

    // app.js
    import SlsTracker from '@aliyun-sls/web-track-mini';
    import createStsPlugin from '@aliyun-sls/web-sts-plugin';
    
    App({
      todos: [
        { text: 'Learning Javascript', completed: true },
        { text: 'Learning ES2016', completed: true },
        { text: 'Learning the Alipay mini program', completed: false },
      ],
      userInfo: null,
      tracker: null,
    
      onLaunch: function() {
        // When you start the mini program, initialize the SlsTracker object.
        this.initSlsTracker();
      },
    
      initSlsTracker: function() {
        const opts = {
          host: 'cn-shanghai.log.aliyuncs.com', // Replace the value with your endpoint.
          project: '${project}', // Replace the value with your project name.
          logstore: '${Logstore}', // Replace the value with your Logstore name.
          time: 10,
          count: 10,
          topic: 'topic',
          source: 'source',
          tags: {
            tags: 'tags',
          },
        };
    
        const stsOpt = {
          accessKeyId: '',
          accessKeySecret: '',
          securityToken: '',
          refreshSTSToken: () => new Promise((resolve, reject) => {
            my.request({
              url: 'http://${Public IP address of the ECS instance}}/get_sts_token', // Replace the value with the actual address of your application server that generates the STS token.
              method: 'GET',
              success: (res) => {
                if (res.statusCode === 200) {
                  let credential;
    
                  // Check whether the data type of the returned data is string. If the data type is not string, the data may be an object.
                  if (typeof res.data === "string") {
                    // Parse the returned data into a JSON object.
                    credential = JSON.parse(res.data);
                  } else {
                    // Use the object.
                    credential = res.data;
                  }
                  stsOpt.accessKeyId = credential.AccessKeyId;
                  stsOpt.accessKeySecret = credential.AccessKeySecret;
                  stsOpt.securityToken = credential.SecurityToken;
                  resolve(credential);
                } else {
                  reject(new Error('Failed to refresh STS token with status code: ' + res.statusCode));
                }
              },
              fail: (err) => {
                reject(new Error('Failed to refresh STS token', err));
              },
            });
          }),
        };
    
        const tracker = new SlsTracker(opts);
        const stsPlugin = createStsPlugin(stsOpt);
        tracker.useStsPlugin(stsPlugin);
    
        // When you start the mini program, a log is uploaded to Simple Log Service. 
        // tracker.send({
        //   eventType:'view_product',
        //   productName: 'Tablet',
        //   price: 500  
        // });
    
    
        this.tracker = tracker;
      },
    
      
    });
    
    1. For more information about the opts and stsOpt parameters, see Use the STS plug-in of web tracking SDK for JavaScript to upload logs.

    2. Replace ${project} and ${logstore} with the names of the project and Logstore that are created in the "Preparations" section of this topic.

    3. Replace <Public IP address of the ECS instance> with the public IP address of the ECS instance that is created in the "Preparations" section of this topic. For more information about how to view the public IP address of an ECS instance, see View IP addresses.

  7. Run the client-side code of the mini program. Click the image icon in the upper-right corner of the developer tool page. On the preview page of the mini program, click the Add Todo button on the Todo App page. When you click the Add Todo button or enter a to-do item after you click the Add Todo button, the mini program directly uploads a log to the Simple Log Service Logstore. This precludes the need for data transfer or data collection to your application server.

Verification and cleaning

Solution verification

After you complete the preceding steps, you can preview logs to check whether the logs are collected to Simple Log Service.

  1. Log on to the Simple Log Service console.

  2. In the Projects section, click the project that you want to manage.

  3. In the Logstores list, find the Logstore whose logs you want to view and choose 修改日志库 > Consumption Preview to the right of the Logstore.

  4. In the Consumption Preview panel, view the collected logs.

    image

Resource cleaning

In this example, you created an ECS instance, a Simple Log Service project, a Logstore, a RAM user, and a RAM role. After you verify the solution, you can perform the following operations to release the resources. This way, you can avoid unnecessary fees or security risks.

What to do next

  • The mini program used in this topic is only debugged by using the developer tool on your computer. To officially launch the mini program, you must perform the following operations:

    1. Create a developer account and a mini program. Then, submit the mini program for review. For more information, see Register a developer account.

    2. In your developer tool, select the required options to skip validity checks of domain names. This facilitates debugging. For more information, see Access preparations.

    3. In the Alipay console, configure a domain name whitelist.

      1. Add the domain name of your Simple Log Service server or valid domain names for requests to the whitelist. The domain name is in the https://${project}.${host} format. project specifies the name of the project that is used. host specifies the endpoint of the project. sls-webtracking-mini.cn-shanghai.log.aliyuncs.com is used in this topic. For more information, see Domain name whitelist of servers.

      2. Add the domain name of the application server of the mini program. This topic provides only the client-side code of the mini program. To officially launch the mini program, you must develop the server-side code of the mini program. Only HTTPS is supported for domain names in the whitelist. To use HTTPS, you must install SSL certificates on the related servers. For more information, see Installation overview. In this example, the options to skip the validity checks of domain names are selected in the developer tool. Before you officially launch the mini program, you must configure the checks.

  • After you collect user logs from mini programs to Simple Log Service, you can perform the following operations:

    • Query and analyze the logs to gain insights into user behavior. For more information, see Query and analyze logs.

    • Display query and analysis results in dashboards, charts, and third-party visualization tools. For more information, see Overview of visualization.

    • Analyze user behavior logs to detect potential security threats, including failed logon attempts, abnormal access modes, and suspected data leaks. If the system detects the preceding threats, alerts are triggered. Then, the security team can respond to the threats at the earliest opportunity. For more information, see Introduction to the alerting feature.