Function Compute:Simple Log Service triggers

Scenarios

• Data cleansing and transformation

  Simple Log Service allows you to quickly collect, process, query, and analyze logs.

  

• Data shipping

  Simple Log Service allows you to ship data to its destination and build data pipelines between big data services on the cloud.

  

Functions for data processing

Category

• Template functions

  For more information, visit the aliyun-log-fc-functions page on GitHub.

• Custom functions

  Function formats are related to function implementations. For more information, see Create a custom function.

Trigger mechanism

An ETL job invokes a function based on time. For example, an ETL job triggers a function every 60 seconds and data is constantly written to Shard0 of a Logstore. In this case, the shard triggers the function every 60 seconds. If data is no longer written to the shard, the function cannot be triggered. The inputs to execute the function are the start and end offsets of the cursor in the most recent 60 seconds. Data can be processed based on the cursor range of the previous 60 seconds.

Limits

You can associate a maximum of five times the number of Logstores in a Simple Log Service project with the project.

Sample scenario

You can configure a Simple Log Service trigger to periodically obtain updated data and invoke a function. Simple Log Service triggers are suitable for scenarios in which you want to incrementally consume data from a Logstore. You can invoke functions to perform custom processing tasks, such as a data cleansing task and a data processing task, and ship data to a third-party service. This example only shows how to obtain and display log data.

Prerequisites

• Function Compute

  • A function is created. For more information, see the Create a function section of the "Manage functions" topic.

• Simple Log Service

  • A Simple Log Service project and two Logstores are created. For more information, see Resource management overview.

    One Logstore is used to process logs from the data source, and the other Logstore is used to store the logs that are generated by Function Compute.

    Note

    The log project must reside in the same region as the Function Compute service.

Step 1: Create a Simple Log Service trigger

1. Log on to the Function Compute console. In the left-side navigation pane, click Functions.

2. In the top navigation bar, select a region. On the Functions page, click the function that you want to manage.

3. On the function details page, click the Configurations tab. In the left-side navigation pane, click Triggers. Then, click Create Trigger.

4. In the Create Trigger panel, configure parameters and click OK.

   Parameter	Description	Example
   Trigger Type	The type of the trigger. Select Log Service.	Log Service
   Name	The name of the trigger.	log_trigger
   Version or Alias	The version or alias of the trigger. Default value: LATEST. If you want to create a trigger for another version or alias, select a version or alias in the upper-right corner of the function details page. For more information about versions and aliases, see Manage versions and Manage aliases.	LATEST
   Log Service Project	The name of the existing Simple Log Service project.	aliyun-fc-cn-hangzhou-2238f0df-a742-524f-9f90-976ba457****
   Logstore	The name of the existing Logstore. The trigger created in this example subscribes to data in the Logstore and sends the data to Function Compute at regular intervals for custom processing.	function-log
   Trigger Interval	The interval at which Simple Log Service invokes the function. Valid values: 3 to 600. Unit: seconds. Default value: 60.	60
   Retries	The maximum number of retries that are allowed for each invocation. Valid values: 0 to 100. Default value: 3. Note If the function is triggered, status=200 is returned, and the value of the X-Fc-Error-Type parameter in the response header is not UnhandledInvocationError or HandledInvocationError. In other cases, the function fails to be triggered. For more information about X-Fc-Error-Type, see Response parameters. If the function fails to be triggered, the system retries to invoke the function until the function is invoked. The number of retries follows the value of this parameter. If the function still fails after the value of this parameter is reached, the system retries the request in exponential backoff mode with increased intervals.	3
   Trigger Log	The Logstore to which you want to store the logs that are generated when Simple Log Service invokes the function.	function-log2
   Invocation Parameters	The invocation parameters. You can configure custom parameters in this editor. The custom parameters are passed to the function as the value of the parameter parameter of the event that is used to invoke the function. The value of the Invocation Parameters parameter must be a string in the JSON format. By default, this parameter is empty.	N/A
   Role Name	Select AliyunLogETLRole. Note After you configure the preceding parameters, click OK. If this is the first time that you create a trigger of this type, click Authorize Now in the dialog box that appears.	AliyunLogETLRole

   After the trigger is created, it is displayed on the Triggers tab. To modify or delete a trigger, see Manage triggers.

Step 2: Configure the input parameter of the function

1. On the Code tab of the function details page, click the icon next Test Function and select Configure Test Parameters from the drop-down list.

2. In the Configure Test Parameters panel, click the Create New Test Event or Modify Existing Test Event tab, enter the event name and event content, and then click OK.

   An event is used to invoke a function in Function Compute. The parameters of the event are used as the input parameters of the function. The following sample code provides an example on the format of the event content:

   {
       "parameter": {},
       "source": {
           "endpoint": "http://cn-hangzhou-intranet.log.aliyuncs.com",
           "projectName": "aliyun-fc-cn-hangzhou-2238f0df-a742-524f-9f90-976ba457****",
           "logstoreName": "function-log",
           "shardId": 0,
           "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==",
           "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA=="
       },
       "jobName": "1f7043ced683de1a4e3d8d70b5a412843d81****",
       "taskId": "c2691505-38da-4d1b-998a-f1d4bb8c****",
       "cursorTime": 1529486425
   }                       

   Parameter	Description	Example
   parameter	The value of the Invocation Parameters parameter that you configure when you create the trigger.	N/A
   source	The log block information that you want the function to read from Simple Log Service. endpoint: the endpoint of the Alibaba Cloud region in which the Simple Log Service project resides. projectName: the name of the Simple Log Service project. logstoreName: the name of the Logstore. shardId: the ID of a specific shard in the Logstore. beginCursor: the offset from which data consumption starts. endCursor: the offset at which data consumption ends.	{ "endpoint": "http://cn-hangzhou-intranet.log.aliyuncs.com", "projectName": "aliyun-fc-cn-hangzhou-2238f0df-a742-524f-9f90-976ba457****", "logstoreName": "function-log", "shardId": 0, "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==", "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA==" }
   jobName	The name of an ETL job in Simple Log Service. Simple Log Service triggers must correspond to ETL jobs in Simple Log Service.	1f7043ced683de1a4e3d8d70b5a412843d81****
   taskId	For an ETL job, the taskId parameter specifies the identifier for a function invocation.	c2691505-38da-4d1b-998a-f1d4bb8c****
   cursorTime	The UNIX timestamp of the time when the last log arrives at Simple Log Service.	1529486425

Step 3: Write and test function code

After you create the Simple Log Service trigger, you can write function code and test the function code to verify whether the code is valid. The function is invoked when Simple Log Service collects incremental logs. Function Compute obtains the corresponding logs, and then displays the collected logs.

1. On the function details page, click the Code tab, enter function code in the code editor, and then click Deploy.

   In this example, the function code is written in Python. The following sample code provides an example on how to extract most logical logs. You can obtain the AccessKey ID and AccessKey secret from context and creds in the code.

   """
   The sample code is used to implement the following features:
   * Parse Simple Log Service events from the event parameter.
   * Initialize the Simple Log Service client based on the preceding information.
   * Obtain real-time log data from the source Logstore.
   
   
   This sample code is mainly doing the following things:
   * Get SLS processing related information from event
   * Initiate SLS client
   * Pull logs from source log store
   
   """
   #!/usr/bin/env python
   # -*- coding: utf-8 -*-
   
   import logging
   import json
   import os
   from aliyun.log import LogClient
   
   
   logger = logging.getLogger()
   
   
   def handler(event, context):
   
       # Query the key information from context.credentials.
       # Access keys can be fetched through context.credentials
       print("The content in context entity is: \n")
       print(context)
       creds = context.credentials
       access_key_id = creds.access_key_id
       access_key_secret = creds.access_key_secret
       security_token = creds.security_token
   
       # Parse the event parameter to the OBJECT data type.
       # parse event in object
       event_obj = json.loads(event.decode())
       print("The content in event entity is: \n")
       print(event_obj)
   
       # Query the following information from event.source: log project name, Logstore name, the endpoint to access the Simple Log Service project, start cursor, end cursor, and shard ID.
       # Get the name of log project, the name of log store, the endpoint of sls, begin cursor, end cursor and shardId from event.source
       source = event_obj['source']
       log_project = source['projectName']
       log_store = source['logstoreName']
       endpoint = source['endpoint']
       begin_cursor = source['beginCursor']
       end_cursor = source['endCursor']
       shard_id = source['shardId']
   
       # Initialize the Simple Log Service client.
       # Initialize client of sls
       client = LogClient(endpoint=endpoint, accessKeyId=access_key_id, accessKey=access_key_secret, securityToken=security_token)
   
       # Read logs based on the start and end cursors in the source Logstore. In this example, the specified cursors include all logs of the function invocation.
       # Read data from source logstore within cursor: [begin_cursor, end_cursor) in the example, which contains all the logs trigger the invocation
       while True:
         response = client.pull_logs(project_name=log_project, logstore_name=log_store,
                                   shard_id=shard_id, cursor=begin_cursor, count=100,
                                   end_cursor=end_cursor, compress=False)
         log_group_cnt = response.get_loggroup_count()
         if log_group_cnt == 0:
           break
         logger.info("get %d log group from %s" % (log_group_cnt, log_store))
         logger.info(response.get_loggroup_list())
   
         begin_cursor = response.get_next_cursor()
   
       return 'success'

2. Click Test Function.

   After the function is executed, you can view the results on the Code tab.

FAQ

• If new logs are generated but your Simple Log Service trigger does not trigger function execution, see What do I do if a trigger cannot trigger function execution?

• Every shard of Simple Log Service triggers the function when new data is written. Therefore, the trigger frequency includes the number of times that a Logstore is triggered as a whole. In addition, when the trigger is delayed, data catchup occurs, which may shorten the trigger interval of triggers. For more information, see Why is the execution frequency of a Simple Log Service trigger higher than expected?