All Products
Search
Document Center

Function Compute:Simple Log Service triggers

Last Updated:Feb 28, 2024

You can create a Simple Log Service trigger to connect Simple Log Service to Function Compute. The Simple Log Service trigger automatically triggers a function to process incremental logs in a Logstore based on your business requirements.

Scenarios

  • Data cleansing and processing

    Simple Log Service allows you to quickly collect, process, query, and analyze logs.

    image
  • Data shipping

    Simple Log Service allows you to ship data to a specific destination. In this scenario, Simple Log Service serves as a data channel for big data services in the cloud.

    image

Functions for data processing

Function types

Trigger mechanism

An extract, transform, and load (ETL) job corresponds to a Simple Log Service trigger and is used to invoke a function. After you create an ETL job for a Logstore in Simple Log Service, a timer is started to poll data from the shards of the Logstore based on the job configurations. If data is written to the Logstore, a triple data record in the <shard_id,begin_cursor,end_cursor > format is generated as a function event. Then, the associated ETL function is invoked. Simple Log Service pushes function events to Function Compute.

Note

If no data is written to the Logstore and the storage system is updated, the cursor information may change. As a result, the ETL function is invoked for each shard but no data is transformed. In this case, you can use the cursor information to obtain data from the shards. If no data is obtained, the ETL function is invoked but no data is transformed. You can ignore the function invocation. For more information, see Create a custom function.

An ETL job invokes a function based on time. For example, an ETL job triggers a function every 60 seconds and data is constantly written to Shard0 of a Logstore. In this case, the shard triggers the function every 60 seconds. If data is no longer written to the shard, the function cannot be triggered. The inputs to execute the function are the start and end offsets of the cursor in the most recent 60 seconds. Data can be processed based on the cursor range of the previous 60 seconds.

image

Limits

You can associate a maximum of five times the number of Logstores in a Simple Log Service project with the project.

Note

We recommend that you configure no more than five Simple Log Service triggers for each Logstore. Otherwise, data may not be efficiently shipped to Function Compute.

Sample scenario

You can configure a Simple Log Service trigger to periodically obtain updated data and invoke a function. Simple Log Service triggers are suitable for scenarios in which you want to incrementally consume data from a Logstore. You can invoke functions to perform custom processing tasks, such as a data cleansing task and a data processing task, and ship data to a third-party service. This example only shows how to obtain and display log data.

Note

The function that is used to process data can be a template function that is provided by Simple Log Service or a custom function.

Prerequisites

  • Function Compute

    • A service is created. For more information, see Create a service.

      Note

      When you create a service, configure a service role to allow functions in the service to obtain the permissions of the role. If you do not configure a service role, an error is reported when you test the function code. In this example, the service role is AliyunFCDefaultRole and the AliyunLogReadOnlyAccess policy is attached to the role. For more information about service roles, see Grant Function Compute permissions to access other Alibaba Cloud services.

  • Simple Log Service

    • A Simple Log Service project and two Logstores are created. For more information, see Resource management overview.

      One Logstore is used to process logs and data sources, and the other Logstore is used to store logs that are generated by Function Compute.

      Note

      The log project must reside in the same region as the Function Compute service.

Step 1: Create a Simple Log Service trigger

  1. Log on to the Function Compute console.

  2. In the left-side navigation pane, click Services & Functions.
  3. In the top navigation bar, select a region.

  4. On the Services page, find the desired service and click Functions in the Actions column.
  5. On the Functions page, click the function that you want to manage.
  6. On the function details page, click the Triggers tab, select the version or alias from the Version or Alias drop-down list, and then click Create Trigger.
  7. In the Create Trigger panel, configure the parameters and click OK.

    Parameter

    Description

    Example

    Trigger Type

    Select Log Service.

    Log Service

    Name

    Enter the trigger name.

    log_trigger

    Version or Alias

    The version or alias of the trigger. Default value: LATEST. If you want to create a trigger for another version or alias, select a version or alias in the upper-right corner of the function details page. For more information about versions and aliases, see Manage versions and Manage aliases.

    LATEST

    Log Service Project

    The name of the existing Simple Log Service project.

    aliyun-fc-cn-hangzhou-2238f0df-a742-524f-9f90-976ba457****

    Logstore

    The name of the existing Logstore. The trigger created in this example subscribes to data in the Logstore and sends the data to Function Compute at regular intervals for custom processing.

    function-log

    Trigger Interval

    The interval at which Simple Log Service invokes the function.

    Valid values: 3 to 600. Unit: seconds. Default value: 60.

    60

    Retries

    The maximum number of retries that is allowed for a single trigger if an error occurs when Simple Log Service triggers a function invocation.

    Valid values: 0 to 100. Default value: 3.

    Note
    • If the function is triggered, status=200 is returned, and the value of the X-Fc-Error-Type parameter in the response header is not UnhandledInvocationError or HandledInvocationError. In other cases, the function fails to be triggered. For more information about X-Fc-Error-Type, see Response parameters.

    • If the function fails to be triggered, the system retries to invoke the function until the function is invoked. The number of retries follows the value of this parameter. If the function still fails after the value of this parameter is reached, the system retries the request in exponential backoff mode with increased intervals.

    3

    Trigger Log

    The Logstore to which you want to store the logs that are generated when Simple Log Service invokes the function.

    function-log2

    Invocation Parameters

    Configure a custom parameter in this editor. This parameter is passed to the function as the parameter field of the event parameter. The value of this parameter must be a string in JSON format.

    By default, this parameter is empty.

    None

    Role Name

    Select AliyunLogETLRole.

    Note

    After you configure the preceding parameters, click OK. The first time you create a trigger of this type, click Authorize Now in the dialog box that appears.

    AliyunLogETLRole

    After the trigger is created, it is displayed on the Triggers tab. To modify or delete an existing trigger, see Manage triggers.

Step 2: Configure the input parameter of the function

  1. On the function details page, click the Code tab, click the xialatubiao icon to the right of Test Function, and then select Configure Test Parameters from the drop-down list.

  2. In the Configure Test Parameters panel, click the Create New Test Event or Modify Existing Test Event tab, configure the Event Name parameter, specify the event content, and then click OK.

    In Function Compute, event is an input parameter. The following code shows the format of the event parameter:

    {
        "parameter": {},
        "source": {
            "endpoint": "http://cn-hangzhou-intranet.log.aliyuncs.com",
            "projectName": "aliyun-fc-cn-hangzhou-2238f0df-a742-524f-9f90-976ba457****",
            "logstoreName": "function-log",
            "shardId": 0,
            "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==",
            "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA=="
        },
        "jobName": "1f7043ced683de1a4e3d8d70b5a412843d81****",
        "taskId": "c2691505-38da-4d1b-998a-f1d4bb8c****",
        "cursorTime": 1529486425
    }                       

    Parameter

    Description

    Example

    parameter

    The value of the Invocation Parameters parameter that you configure when you create the trigger.

    None

    source

    The log block information that you want the function to read from Simple Log Service.

    • endpoint: the endpoint of the Alibaba Cloud region in which the Simple Log Service project resides.

    • projectName: the name of the Simple Log Service project.

    • logstoreName: the name of the Logstore.

    • shardId: the ID of a specific shard in the Logstore.

    • beginCursor: the offset from which data consumption starts.

    • endCursor: the offset at which data consumption ends.

    {
        "endpoint": "http://cn-hangzhou-intranet.log.aliyuncs.com",
        "projectName": "aliyun-fc-cn-hangzhou-2238f0df-a742-524f-9f90-976ba457****",
        "logstoreName": "function-log",
        "shardId": 0,
        "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==",
        "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA=="
    }

    jobName

    The name of an ETL job in Simple Log Service. Simple Log Service triggers must correspond to ETL jobs in Simple Log Service.

    1f7043ced683de1a4e3d8d70b5a412843d81****

    taskId

    For an ETL job, taskId is the identifier for a deterministic function invocation.

    c2691505-38da-4d1b-998a-f1d4bb8c****

    cursorTime

    The UNIX timestamp of the time when the last log arrives at Simple Log Service.

    1529486425

Step 3: Write and test function code

After you create the Simple Log Service trigger, you can write function code and test the function code to verify whether the code is valid. The function is invoked when Simple Log Service collects incremental logs. Function Compute obtains the corresponding logs, and then displays the collected logs.

  1. On the function details page, click the Code tab, enter function code in the code editor, and then click Deploy.

    This section uses function code for Python as an example. In the sample code, the values of the access_key_id, access_key_secret, and security_token parameters are queried from context.credentials.

    """
    The sample code is used to implement the following features:
    * Parse Simple Log Service events from the event parameter.
    * Initialize the Simple Log Service client based on the preceding information.
    * Obtain real-time log data from the source Logstore.
    
    
    This sample code is mainly doing the following things:
    * Get SLS processing related information from event
    * Initiate SLS client
    * Pull logs from source log store
    
    """
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    
    import logging
    import json
    import os
    from aliyun.log import LogClient
    
    
    logger = logging.getLogger()
    
    
    def handler(event, context):
    
        # Query the key information from context.credentials.
        # Access keys can be fetched through context.credentials
        print("The content in context entity is: \n")
        print(context)
        creds = context.credentials
        access_key_id = creds.access_key_id
        access_key_secret = creds.access_key_secret
        security_token = creds.security_token
    
        # Parse the event parameter to the OBJECT data type.
        # parse event in object
        event_obj = json.loads(event.decode())
        print("The content in event entity is: \n")
        print(event_obj)
    
        # Query the following information from event.source: log project name, Logstore name, the endpoint to access the Simple Log Service project, start cursor, end cursor, and shard ID.
        # Get the name of log project, the name of log store, the endpoint of sls, begin cursor, end cursor and shardId from event.source
        source = event_obj['source']
        log_project = source['projectName']
        log_store = source['logstoreName']
        endpoint = source['endpoint']
        begin_cursor = source['beginCursor']
        end_cursor = source['endCursor']
        shard_id = source['shardId']
    
        # Initialize the Simple Log Service client.
        # Initialize client of sls
        client = LogClient(endpoint=endpoint, accessKeyId=access_key_id, accessKey=access_key_secret, securityToken=security_token)
    
        # Read logs based on the start and end cursors in the source Logstore. In this example, the specified cursors include all logs of the function invocation.
        # Read data from source logstore within cursor: [begin_cursor, end_cursor) in the example, which contains all the logs trigger the invocation
        while True:
          response = client.pull_logs(project_name=log_project, logstore_name=log_store,
                                    shard_id=shard_id, cursor=begin_cursor, count=100,
                                    end_cursor=end_cursor, compress=False)
          log_group_cnt = response.get_loggroup_count()
          if log_group_cnt == 0:
            break
          logger.info("get %d log group from %s" % (log_group_cnt, log_store))
          logger.info(response.get_loggroup_list())
    
          begin_cursor = response.get_next_cursor()
    
        return 'success'
  2. On the Code tab, click Test Function.

    After the function is executed, you can view the results on the Code tab.

Troubleshooting