All Products
Search

This Product

    Simple Log Service:CSV format

    Document Center

    Simple Log Service:CSV format

    Last Updated:Aug 29, 2023

    After logs are shipped from Simple Log Service to Object Storage Service (OSS), the logs can be stored in different formats. This topic describes the CSV format.

    Parameters

    The following figure shows the parameters that you must configure if you specify csv for Storage Format in a shipping rule. For more information, see Create an OSS data shipping job (new version).

    CSV Fields

    The following table describes the parameters. For more information, see Common Format and MIME Type for Comma-Separated Values (CSV) Files and PostgreSQL 9.4.26 Documentation.

    Parameter

    Description

    CSV Fields

    The log field that you want to ship to OSS. You can view log fields on the Raw Logs tab of a Logstore. We recommend that you add log fields one by one.

    The log fields that you can ship to OSS include the reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields.

    Note

    • A log field can be added to CSV Fields only once.

    • If a log contains two fields that have the same name, such as request_time, Log Service displays one of the fields as request_time_0. The two fields are still stored as request_time in Log Service. When you configure a shipping rule, you can use only the original field name request_time.

      If a log contains fields that have the same name, Log Service randomly ships the value of one of the fields. We recommend that you do not include fields that have the same name in your logs.

    Delimiter

    The delimiter that is used for log fields. You can use commas (,), vertical bars (|), spaces, or tab characters as delimiters of log fields.

    Escape Character

    If a log field contains a delimiter, you must use an escape character to enclose the field. This way, the delimiter does not take effect on the field when the field is read.

    Invalid Fields

    If a log field that you specify in CSV Fields does not exist in logs, the value that you specify in the Invalid Fields field is shipped.

    Shipped Fields

    Specifies whether to write the names of log fields to the CSV file. If you turn on Shipped Fields, the names of log fields are written to the CSV file.

    URLs of objects in OSS

    After logs are shipped to OSS, the logs are stored in OSS objects. The following table provides examples on the URLs of the OSS objects.

    Note

    • If you specify an object suffix when you create an OSS shipping job, the specified suffix takes effect.

    • If you do not specify an object suffix when you create an OSS shipping job, the suffix is determined based on the compression type.

    Compression type

    Object suffix

    URL example

    Description

    Not compressed

    If you specify an object suffix, the specified suffix takes effect. Example: .suffix.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix

    You can download the CSV file to your computer and open the file as a text file.

    If you do not specify an object suffix, the suffix .csv is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.csv

    Snappy

    If you specify an object suffix, the specified suffix takes effect. Example: .suffix.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix

    For more information about how to open Snappy-compressed files, see Decompression tools for Snappy-compressed files.

    If you do not specify an object suffix, the suffix .snappy.csv is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.snappy.csv

    Gzip

    If you specify an object suffix, the specified suffix takes effect. Example: .suffix.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix

    If objects are compressed by using Gzip, you can install Gzip and decompress the objects by using the Gzip command.

    If you do not specify an object suffix, the suffix .gz.csv is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.gz.csv

    Zstandard

    If you specify an object suffix, the specified suffix takes effect. Example: .suffix.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.suffix

    If objects are compressed by using Zstandard, you can install Zstandard and decompress the objects by using the Zstandard command.

    If you do not specify an object suffix, the suffix .zst.csv is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.zst.csv