All Products
Search

This Product

    Simple Log Service:JSON format

    Document Center

    Simple Log Service:JSON format

    Last Updated:Sep 10, 2024

    After logs are shipped from Simple Log Service to Object Storage Service (OSS), the logs can be stored in different formats. This topic describes the JSON format.

    Parameters

    The following figure shows the parameters that you must configure if you set the Storage Format parameter to json in the Data Shipping to OSS panel. For more information, see Create an OSS data shipping job (new version).

    json

    Parameter

    Description

    Ship Tag

    Specifies whether to ship the tag field of logs. Example: __tag__:abc.

    URLs of OSS objects

    After logs are shipped to OSS, the logs are stored in OSS objects. The following table provides sample URLs of OSS objects.

    Note

    • If you specify an object suffix when you create a data shipping job, the OSS objects use the suffix.

    • If you do not specify an object suffix when you create a data shipping job, the OSS objects use the suffix that is generated based on the compression type.

    Compression type

    Object suffix

    Sample URL

    Description

    Not compressed

    If you specify an object suffix, the specified suffix is used. Example: .json.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.json

    You can download the JSON file to your computer and open the file as a text file. Sample file: 

    {"__time__":"1453809242","__topic__":"","__source__":"10.170.***.***","ip":"10.200.**.***","time":"26/Jan/2016:19:54:02 +0800","url":"POST
              /PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>
              HTTP/1.1","status":"200","user-agent":"aliyun-sdk-java"}

    If you do not specify an object suffix, the object does not have a suffix.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937

    Snappy

    If you specify an object suffix, the specified suffix is used. Example: .json.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.json

    For more information about how to use Snappy to compress and decompress data, see Decompression tools for Snappy-compressed files.

    Problem:

    If you used Snappy to compress objects and use Hive deployed in an E-MapReduce (EMR) cluster to read the objects, the following error may occur.

    image

    Solution:

    Execute the io.compression.codec.snappy.native=true; statement and then perform query operations. If you encounter other Hive-related errors, submit a ticket.

    If you do not specify an object suffix, the .snappy suffix is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.snappy

    gzip

    If you specify an object suffix, the specified suffix is used. Example: .json.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.json

    If objects are compressed by using gzip, you can install gzip and decompress the objects by using gzip commands.

    If you do not specify an object suffix, the .gz suffix is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.gz

    Zstandard

    If you specify an object suffix, the specified suffix is used. Example: .json.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.json

    If objects are compressed by using Zstandard, you can install Zstandard and decompress the objects by using Zstandard commands.

    If you do not specify an object suffix, the .zst suffix is used.

    oss://oss-shipper-chengdu/ecs_test/2022/01/26/20/54_1453812893059571256_937.zst