All Products
Search

This Product

    Simple Log Service:CSV format

    Document Center

    Simple Log Service:CSV format

    Last Updated:Sep 03, 2024

    After logs are shipped from Log Service to Object Storage Service (OSS), the logs can be stored in different formats. This topic describes the CSV format.

    Parameters

    The following figure shows the parameters that you must configure if you specify csv for Storage Format in a shipping rule. For more information, see Configure a data shipping rule.

    CSV Fields
    The following table describes the parameters. For more information, see Common Format and MIME Type for Comma-Separated Values (CSV) Files and PostgreSQL 9.4.26 Documentation.
    ParameterDescription
    CSV FieldsThe log field that you want to ship to OSS. You can view log fields on the Raw Logs tab of a Logstore. We recommend that you add log fields one by one. The log fields that you can ship to OSS include the fields in the log content and the reserved fields such as __time__, __topic__, and __source__. For more information about reserved fields, see Reserved fields.
    Note
    • A log field can be added to CSV Fields only once.
    • If a log contains two fields that have the same name, such as request_time, Log Service displays one of the fields as request_time_0. The two fields are still stored as request_time in Log Service. When you configure a shipping rule, you can use only the original field name request_time.

      If a log contains fields that have the same name, Log Service randomly ships the value of one of the fields. We recommend that you do not include fields that have the same name in your logs.

    DelimiterThe delimiter that is used for log fields. You can use commas (,), vertical bars (|), spaces, or tab characters as delimiters of log fields.
    Escape CharacterIf a log field contains a delimiter, you must use an escape character to enclose the field. This way, the delimiter does not take effect on the field when the field is read.
    Invalid FieldsIf a log field that you specify in CSV Fields does not exist in logs, the value that you specify in the Invalid Fields field is shipped.
    Shipped FieldsSpecifies whether to write the names of log fields to the CSV file. If you turn on Shipped Fields, the names of log fields are written to the CSV file.

    URLs of files in OSS

    After logs are shipped to OSS, the logs are stored in OSS buckets. The following table provides examples of the URLs of the files that store the logs.

    Compression typeFile extensionURL exampleDescription
    Not compressed.csvoss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.csvYou can download the CSV file to your computer and open the file as a text file.
    Snappy.snappy.csvoss://oss-shipper-shenzhen/ecs_test/2016/01/26/20/54_1453812893059571256_937.snappy.csvFor more information about how to open Snappy-compressed files, see Decompression tools for Snappy-compressed files.