Simple Log Service:Import OSS data

Create a data import configuration

1. Log on to the Simple Log Service console.

2. In the Data Ingestion area, on the Data Import tab, select OSS-Data Import.

3. Select the destination project and Logstore, and then click Next.

4. Set the import configuration.

   1. In the Import Configuration step, set the following parameters.

      Parameter settings

      Parameter	Description
      Task Name	The unique name of the SLS task.
      Display Name	The display name of the task.
      Task Description	The description of the import task.
      OSS Region	The region where the bucket that stores the OSS files to be imported is located. If the OSS bucket and the Simple Log Service project are in the same region, you can save Internet traffic and benefit from faster data transfer.
      Bucket	The bucket that stores the OSS files to be imported.
      Filter by File Path Prefix	Filter OSS files by file path prefix to locate the files to be imported. For example, if all files to be imported are in the csv/ directory, you can specify the prefix as csv/. If you do not set this parameter, the entire OSS bucket is traversed. Note Set this parameter. If a bucket contains many files, traversing the entire bucket significantly reduces data import efficiency.
      Filter by File Path Regular Expression	Filter OSS files by a regular expression for the file path to locate the files to be imported. Only files whose names (including file paths) match the regular expression are imported. By default, this parameter is empty, which means no filtering is performed. For example, if the OSS file is testdata/csv/bill.csv, you can set the regular expression to (testdata/csv/)(.*). For more information about how to debug a regular expression, see How do I test a regular expression?.
      Filter by File Modification Time	Filter OSS files by file modification time to locate the files to be imported. All: Select this option if you want to import all eligible files. From a Specific Time: Select this option if you want to import files modified after a specific point in time. Specific Time Range: Select this option if you want to import files modified within a specific time range.
      Data Format	The file is parsed in the following format. CSV: A text file that uses a separator. You can specify the first line of the file as field names or manually specify field names. Each line except for the field names is parsed as the values of log fields. Single-line JSON: Reads the OSS file line by line and parses each line as a JSON object. After parsing, the fields in the JSON object correspond to the fields in the log. JSON array: Reads the entire OSS file at once. The content is an array that contains one or more JSON objects. Cloudtrail: Reads the entire OSS file at once. The content is in the standard CloudTrail data structure format. Single-line Text Log: Parses each line in the OSS file as a log entry. Multi-line Text Log: Multi-line mode. You can specify a regular expression for the first or last line to parse logs. ORC: ORC file format. No configuration is required. The file is automatically parsed into a log format. Parquet: Parquet format. No configuration is required. The file is automatically parsed into a log format. Alibaba Cloud OSS Access Log: Alibaba Cloud OSS access log format. For more information, see Log storage. Alibaba Cloud CDN Download Log: Alibaba Cloud CDN download log format. For more information, see Quick Start.
      Compression Format	The compression format of the OSS files to be imported. Simple Log Service decompresses and reads the data based on the corresponding format.
      Encoding Format	The encoding format of the OSS files to be imported. Only UTF-8 and GBK are supported.
      Check for New Files Interval	If new files are continuously generated in the destination OSS file path, you can set the Check for New Files Interval as needed. After you set the interval, the import task runs in the background and periodically discovers and reads new files. The background process ensures that data from the same OSS file is not repeatedly written to Simple Log Service. For example, if you create a task at 12:00 with a new file check interval of 30 minutes, the import task runs for the first time at 12:00. If new files are generated later, the task runs for the second time at 12:30. If no new files are generated in the destination OSS file path, change the interval to Never. The import task will automatically exit after reading all eligible files.
      Import Archived Objects	If the OSS files to be imported are of the standard Archive Storage or Cold Archive storage class, they must be restored before they can be read. If you enable this feature, archived objects are automatically restored. Deep Cold Archive is not supported. Note Restoring a standard archived object takes about 1 minute. This may cause the first preview to time out. If a timeout occurs, wait for a period of time and try again. Restoring a Cold Archive object takes about 1 hour. If the preview times out, you can skip the preview or wait 1 hour and try again. When a Cold Archive object is restored, the restoration is valid for 7 days by default to ensure that you have enough time to import the Cold Archive object.
      Log Time Configuration
      Time Field	When you set Data Format to CSV, Single-line JSON, JSON array, Cloudtrail, ORC, Parquet, Alibaba Cloud OSS Access Log, or Alibaba Cloud CDN Download Log, set a time field. This is the column name in the OSS file that represents time, used to specify the time when logs are imported to Simple Log Service.
      Time Extraction Regex	When you set Data Format to Single-line Text Log or Multi-line Text Log, you need to use a regular expression to extract the time from the logs. For example, if the log sample is 127.0.0.1 - - [10/Sep/2018:12:36:49 +0800] "GET /index.html HTTP/1.1", you can set Time Extraction Regex to [0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9\: +]+. Note For other data formats, you can also use a regular expression to extract only part of the content from the time field.
      Time Field Format	Specify the time format to parse the value of the time field. Time formats that follow the Java SimpleDateFormat syntax are supported, such as yyyy-MM-dd HH:mm:ss. For more information about the time format syntax, see Class SimpleDateFormat. For common time formats, see Time formats. Epoch formats are supported, including epoch, epochMillis, epochMicro, and epochNano.
      Time Field Time Zone	Select the time zone corresponding to the time field. When the time field format is an epoch type, you do not need to set the time zone. To consider daylight saving time when parsing the log time, select a UTC format. Otherwise, select a GMT format.
      Advanced Configuration
      Use OSS Metadata Index	If the number of OSS files exceeds one million, enable this feature. Otherwise, the efficiency of discovering new files is very low. After you use the OSS metadata index, new files in the OSS bucket can be discovered within seconds, enabling near-real-time import of data from newly written files in the bucket. Before you use the OSS metadata index, you must first enable the OSS metadata index management feature in OSS. For more information, see Scalar retrieval.

      You must set additional parameters when you set Data Format to CSV or Multi-line Text Log. The following table describes these parameters.

      CSV

      Parameter	Description
      Separator	Set the separator for the logs. The default value is a comma (,).
      Quote Character	The quote character used for CSV strings.
      Escape Character	Configure the escape character for the logs. The default value is a backslash (\).
      Maximum Lines per Log	When a log entry spans multiple lines, you need to specify the maximum number of lines. The default value is 1.
      Use First Line as Field Names	After you turn on the Use First Line as Field Names switch, the first line of the CSV file is used as the field names. For example, the first line in the following figure is extracted as the names of the log fields.
      Custom Field List	After you turn off the Use First Line as Field Names switch, customize the log field names as needed. Separate multiple field names with commas (,).
      Lines to Skip	Specify the number of log lines to skip. For example, if you set this to 1, log collection starts from the second line of the CSV file.

      Cross-text logs

      Parameter	Description
      Regex Match Position	Set the position for the regular expression match. The following options are available: First Line Regex: Use a regular expression to match the first line of a log entry. The unmatched part is considered part of that log entry, up to the maximum number of lines. Last Line Regex: Use a regular expression to match the last line of a log entry. The unmatched part is considered part of the next log entry, up to the maximum number of lines.
      Regular Expression	Set the correct regular expression based on the log content. For more information about how to debug a regular expression, see How do I test a regular expression?.
      Maximum Lines	The maximum number of lines for a single log entry.

   2. Click Preview to view the import results.

   3. After confirming the settings, click Next.

5. Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also manually create field indexes for the collected logs or click Automatic Index Generation. Then, Simple Log Service generates field indexes. For more information, see Create an index.

   Important

   If you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.

6. Click Query Logs to open the query and analysis page and confirm that the OSS data was successfully imported.

   Wait for about 1 minute. The import is successful if the data is available in the target OSS location.

Related operations

After you create an import configuration, you can view the configuration and its statistical reports in the console.

1. In the Projects section, click the destination project.

2. In the navigation pane, choose Log Storage > Logstores. For the destination Logstore, choose Data Access > Data Import, and then click the configuration name.

3. View a task

   On the Import Configuration Overview page, you can view basic information and statistical reports.

   

   Modify a task

   Click Modify Configuration to modify the import configuration. For more information, see Set the import configuration.

   Delete a task

   To delete the import configuration, click Delete Configuration.

   Warning

   This operation cannot be undone. Proceed with caution.

   Stop a task

   To stop the import task, click Stop.

   Start a task

   Click Start to begin the import task.

FAQ

Error handling mechanism

OSS delivery API operations