ALIYUN::SLS::OssExport - 资源编排

ALIYUN::SLS::OssExport类型用于日志服务数据传输管理，该服务提供了将日志库中的数据传输至OSS产品存储的功能。

语法

{
  "Type": "ALIYUN::SLS::OssExport",
  "Properties": {
    "Configuration": Map,
    "DisplayName": String,
    "ExportName": String,
    "ProjectName": String,
    "Description": String
  }
}

属性

属性名称	类型	必须	允许更新	描述	约束
Configuration	Map	是	是	导出任务的配置。	详情参考Configuration属性。
DisplayName	String	是	是	OSS导出的显示名称。	无。
ExportName	String	是	否	投递配置名称。	它只能包含小写字母、数字、破折号 - 和下划线 _。它必须以小写字母或数字开头和结尾，名称的长度必须为2到64个字符。
ProjectName	String	是	否	日志项目名称。	无
Description	String	否	是	导出任务的描述。	其长度可以是0到256个字符。

Configuration语法

"Configuration": {
  "ToTime": Integer,
  "Sink": Map,
  "Logstore": String,
  "FromTime": Integer,
  "RoleArn": String
}

Configuration属性

属性名称	类型	必须	允许更新	描述	约束
Logstore	String	是	是	项目的日志存储名称。	无
RoleArn	String	是	是	用于控制写入数据到OSS和读取日志存储中数据的权限。	示例：acs:ram::13234:role/aliyunlogdefaultrole。
Sink	Map	是	是	目标名称。	无
FromTime	Integer	否	是	时间范围的开始时间。	无
ToTime	Integer	否	是	时间范围的结束时间。	无

Sink语法

"Sink": {
  "BufferInterval": Integer,
  "ContentType": String,
  "ContentDetail": Map,
  "PathFormat": String,
  "Prefix": String,
  "PathFormatType": String,
  "RoleArn": String,
  "BufferSize": Integer,
  "TimeZone": String,
  "Suffix": String,
  "Endpoint": String,
  "DelaySeconds": Integer,
  "Bucket": String,
  "CompressionType": String
}

Sink属性

属性名称	类型	必须	允许更新	描述	约束
BufferInterval	Integer	是	是	投递时间。	数据投递的最大时间间隔。有效值：300至900。单位：秒。
BufferSize	Integer	是	是	投递大小。	该参数的值决定了投递并存储在一个对象中的原始日志数据的最大大小。单位：MB。如果您想要投递的日志数据的大小达到指定值，将自动创建一个投递任务。
Bucket	String	是	是	OSS 存储桶的名称。	无
ContentType	String	是	是	存储格式。	取值： json parquet csv orc 您可以选择将数据以 Parquet 格式存储以便进行数据分析。与存储在 CSV 或 JSON 格式的数据相比，以 Parquet 格式存储的数据可以以更高的效率进行扫描。当您以 Parquet 格式投递数据时，相比于 CSV 或 JSON 格式，会产生更多的费用。根据您的业务需求选择数据投递的存储格式。
RoleArn	String	是	是	用于控制写入数据到OSS和读取日志存储中数据的权限。	示例：acs:ram::13234:role/aliyunlogdefaultrole。
ContentDetail	Map	否	是	内容详情。	如果 ContentType=json，默认值为 {"EnableTag": false}。
CompressionType	String	否	是	OSS数据的压缩方法。	有效取值： none snappy gzip zstd none 表示原始数据不进行压缩。 snappy、gzip 或 zstd 表示使用指定的算法压缩数据，这可以减少OSS存储桶的存储空间使用。不同的压缩方法基于不同的存储格式。
DelaySeconds	Integer	否	是	投递延迟。	数据投递之后的时间周期。有效值：900至63244800。单位：秒。注意：请勿指定超过 Logstore 数据保留期限的延迟时间。我们建议您为延迟预留几天的缓冲期。否则，可能会发生数据丢失。
Endpoint	String	否	是	OSS 存储桶的Endpoint。	无
PathFormat	String	否	是	分区格式用于生成子目录。	子目录是基于投递时间动态生成的。默认的分区格式是 %Y/%m/%d/%H/%M。例如：2017/01/23/12/00。注意，分区格式不能以正斜杠（/）开头。有关如何与 E-MapReduce 的计算引擎（如 Hive 和 Impala）集成以查询和分析数据的更多信息。
Prefix	String	否	是	OSS 对象名称的前缀。	无
PathFormatType	String	否	是	分区格式类型。	取值： time
Suffix	String	否	是	OSS 对象名称的后缀。	无
TimeZone	String	否	是	时区。	分区路径会根据时区的不同而有所变化。该值应在 -1200 到 +1400 之间。例如，+0800。

返回值

Fn::GetAtt

ProjectName：日志项目名称。
ExportName：投递配置名称。

示例

YAML格式

ROSTemplateFormatVersion: '2015-09-01'
Parameters:
  Configuration:
    AssociationPropertyMetadata:
      Parameters:
        ToTime:
          Type: Number
          Description:
            en: The end time of the time range.
          Required: false
          Default: 0
        Sink:
          AssociationPropertyMetadata:
            Parameters:
              BufferInterval:
                Type: Number
                Description:
                  en: 'The shipping time. The maximum interval at which data is shipped. Valid values: 300 to 900. Unit: seconds.'
                Required: true
                MinValue: 300
                MaxValue: 900
              ContentType:
                Type: String
                Description:
                  en: The storage format. You can store data in the Parquet format for data analysis. Compared with data stored in the CSV or JSON format, data stored in the Parquet format can be scanned at a higher efficiency.You are charged more fees when you ship data in the Parquet format than in the CSV or JSON format. Select a storage format for data shipping based on your business requirements.
                AllowedValues:
                  - json
                  - parquet
                  - csv
                  - orc
                Required: true
              ContentDetail:
                Type: Json
                Description:
                  en: 'The content detail. If ContentType=json, the default value is {"EnableTag": false}.'
                Required: false
              PathFormat:
                Type: String
                Description:
                  en: 'The partition format is used to generate subdirectories. A subdirectory is dynamically generated based on the shipping time. The default partition format is %Y/%m/%d/%H/%M. Example: 2017/01/23/12/00. Note that the partition format cannot start with a forward slash (/). For more information about how to integrate with the compute engines of E-MapReduce such as Hive and Impala to query and analyze data'
                Required: false
                Default: '%Y/%m/%d/%H/%M'
              Prefix:
                Type: String
                Description:
                  en: The prefix of the OSS object name.
                Required: false
              PathFormatType:
                Type: String
                Description:
                  en: The partition format type. Only support time.
                AllowedValues:
                  - time
                Required: false
                Default: time
              RoleArn:
                Type: String
                Description:
                  en: 'Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole.'
                Required: true
              BufferSize:
                Type: Number
                Description:
                  en: 'The shipping size. The value of this parameter determines the maximum size of raw log data that is shipped and stored in an object. Unit: MB. If the size of log data that you want to ship reaches the specified value, a shipping job is automatically created.'
                Required: true
                MinValue: 5
                MaxValue: 256
              TimeZone:
                Type: String
                Description:
                  en: The time zone. Partition paths vary based on time zones. The value should be -1200 to +1400. For example, +0800.
                Required: false
              Suffix:
                Type: String
                Description:
                  en: The suffix of the OSS object name.
                Required: false
              Endpoint:
                Type: String
                Description:
                  en: The endpoint of the OSS bucket.
                Required: false
              DelaySeconds:
                Type: Number
                Description:
                  en: |-
                    The shipping latency. The period of time after which data is shipped. Valid values: 900 to 63244800. Unit: seconds.
                    Note: Do not specify a latency that exceeds the data retention period of the Logstore. We recommend that you reserve a buffer period of a few days for the latency. Otherwise, data loss may occur.
                Required: false
                MinValue: 900
                MaxValue: 63244800
              Bucket:
                Type: String
                Description:
                  en: The name of the OSS bucket.
                Required: true
              CompressionType:
                Type: String
                Description:
                  en: 'The compression method of OSS data. Valid values: none, snappy, gzip, and zstd. The value none indicates that raw data is not compressed. The value snappy, gzip, or zstd indicates that a specified algorithm is used to compress data, which can reduce the storage space usage of OSS buckets. The compression methods vary based on storage formats.'
                AllowedValues:
                  - none
                  - snappy
                  - gzip
                  - zstd
                Required: false
          Type: Json
          Description:
            en: The sink name.
          Required: true
        Logstore:
          Type: String
          Description:
            en: The logstore name of the project.
          Required: true
        FromTime:
          Type: Number
          Description:
            en: The start time of the time range.
          Required: false
          Default: 0
        RoleArn:
          Type: String
          Description:
            en: 'Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole.'
          Required: true
    Type: Json
    Description:
      en: The configuration of the export job.
    Required: true
  ProjectName:
    Type: String
    Description:
      en: The project name of SLS.
    Required: true
    AllowedPattern: ^[a-zA-Z0-9_-]+$
    MinLength: 3
    MaxLength: 63
  DisplayName:
    Type: String
    Description:
      en: The display name of the export job. It must be 4 to 100 characters in length.
    Required: true
    MinLength: 4
    MaxLength: 100
  ExportName:
    Type: String
    Description:
      en: The name of the export job. This value should be unique. It must be 2 to 64 characters in length and can contain lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit.
    Required: true
    AllowedPattern: ^[a-zA-Z0-9_-]+$
    MinLength: 2
    MaxLength: 64
Resources:
  OssExport:
    Type: ALIYUN::SLS::OssExport
    Properties:
      Configuration:
        Ref: Configuration
      ProjectName:
        Ref: ProjectName
      DisplayName:
        Ref: DisplayName
      ExportName:
        Ref: ExportName
Outputs:
  ProjectName:
    Description: The project name of SLS.
    Value:
      Fn::GetAtt:
        - OssExport
        - ProjectName
  ExportName:
    Description: The name of the export job.
    Value:
      Fn::GetAtt:
        - OssExport
        - ExportName

JSON格式

{
  "ROSTemplateFormatVersion": "2015-09-01",
  "Parameters": {
    "Configuration": {
      "AssociationPropertyMetadata": {
        "Parameters": {
          "ToTime": {
            "Type": "Number",
            "Description": {
              "en": "The end time of the time range."
            },
            "Required": false,
            "Default": 0
          },
          "Sink": {
            "AssociationPropertyMetadata": {
              "Parameters": {
                "BufferInterval": {
                  "Type": "Number",
                  "Description": {
                    "en": "The shipping time. The maximum interval at which data is shipped. Valid values: 300 to 900. Unit: seconds."
                  },
                  "Required": true,
                  "MinValue": 300,
                  "MaxValue": 900
                },
                "ContentType": {
                  "Type": "String",
                  "Description": {
                    "en": "The storage format. You can store data in the Parquet format for data analysis. Compared with data stored in the CSV or JSON format, data stored in the Parquet format can be scanned at a higher efficiency.You are charged more fees when you ship data in the Parquet format than in the CSV or JSON format. Select a storage format for data shipping based on your business requirements."
                  },
                  "AllowedValues": [
                    "json",
                    "parquet",
                    "csv",
                    "orc"
                  ],
                  "Required": true
                },
                "ContentDetail": {
                  "Type": "Json",
                  "Description": {
                    "en": "The content detail. If ContentType=json, the default value is {\"EnableTag\": false}."
                  },
                  "Required": false
                },
                "PathFormat": {
                  "Type": "String",
                  "Description": {
                    "en": "The partition format is used to generate subdirectories. A subdirectory is dynamically generated based on the shipping time. The default partition format is %Y/%m/%d/%H/%M. Example: 2017/01/23/12/00. Note that the partition format cannot start with a forward slash (/). For more information about how to integrate with the compute engines of E-MapReduce such as Hive and Impala to query and analyze data"
                  },
                  "Required": false,
                  "Default": "%Y/%m/%d/%H/%M"
                },
                "Prefix": {
                  "Type": "String",
                  "Description": {
                    "en": "The prefix of the OSS object name."
                  },
                  "Required": false
                },
                "PathFormatType": {
                  "Type": "String",
                  "Description": {
                    "en": "The partition format type. Only support time."
                  },
                  "AllowedValues": [
                    "time"
                  ],
                  "Required": false,
                  "Default": "time"
                },
                "RoleArn": {
                  "Type": "String",
                  "Description": {
                    "en": "Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole."
                  },
                  "Required": true
                },
                "BufferSize": {
                  "Type": "Number",
                  "Description": {
                    "en": "The shipping size. The value of this parameter determines the maximum size of raw log data that is shipped and stored in an object. Unit: MB. If the size of log data that you want to ship reaches the specified value, a shipping job is automatically created."
                  },
                  "Required": true,
                  "MinValue": 5,
                  "MaxValue": 256
                },
                "TimeZone": {
                  "Type": "String",
                  "Description": {
                    "en": "The time zone. Partition paths vary based on time zones. The value should be -1200 to +1400. For example, +0800."
                  },
                  "Required": false
                },
                "Suffix": {
                  "Type": "String",
                  "Description": {
                    "en": "The suffix of the OSS object name."
                  },
                  "Required": false
                },
                "Endpoint": {
                  "Type": "String",
                  "Description": {
                    "en": "The endpoint of the OSS bucket."
                  },
                  "Required": false
                },
                "DelaySeconds": {
                  "Type": "Number",
                  "Description": {
                    "en": "The shipping latency. The period of time after which data is shipped. Valid values: 900 to 63244800. Unit: seconds.\nNote: Do not specify a latency that exceeds the data retention period of the Logstore. We recommend that you reserve a buffer period of a few days for the latency. Otherwise, data loss may occur."
                  },
                  "Required": false,
                  "MinValue": 900,
                  "MaxValue": 63244800
                },
                "Bucket": {
                  "Type": "String",
                  "Description": {
                    "en": "The name of the OSS bucket."
                  },
                  "Required": true
                },
                "CompressionType": {
                  "Type": "String",
                  "Description": {
                    "en": "The compression method of OSS data. Valid values: none, snappy, gzip, and zstd. The value none indicates that raw data is not compressed. The value snappy, gzip, or zstd indicates that a specified algorithm is used to compress data, which can reduce the storage space usage of OSS buckets. The compression methods vary based on storage formats."
                  },
                  "AllowedValues": [
                    "none",
                    "snappy",
                    "gzip",
                    "zstd"
                  ],
                  "Required": false
                }
              }
            },
            "Type": "Json",
            "Description": {
              "en": "The sink name."
            },
            "Required": true
          },
          "Logstore": {
            "Type": "String",
            "Description": {
              "en": "The logstore name of the project."
            },
            "Required": true
          },
          "FromTime": {
            "Type": "Number",
            "Description": {
              "en": "The start time of the time range."
            },
            "Required": false,
            "Default": 0
          },
          "RoleArn": {
            "Type": "String",
            "Description": {
              "en": "Used to control permissions of writing data to OSS and reading data in Logstores. Example: acs:ram::13234:role/aliyunlogdefaultrole."
            },
            "Required": true
          }
        }
      },
      "Type": "Json",
      "Description": {
        "en": "The configuration of the export job."
      },
      "Required": true
    },
    "ProjectName": {
      "Type": "String",
      "Description": {
        "en": "The project name of SLS."
      },
      "Required": true,
      "AllowedPattern": "^[a-zA-Z0-9_-]+$",
      "MinLength": 3,
      "MaxLength": 63
    },
    "DisplayName": {
      "Type": "String",
      "Description": {
        "en": "The display name of the export job. It must be 4 to 100 characters in length."
      },
      "Required": true,
      "MinLength": 4,
      "MaxLength": 100
    },
    "ExportName": {
      "Type": "String",
      "Description": {
        "en": "The name of the export job. This value should be unique. It must be 2 to 64 characters in length and can contain lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit."
      },
      "Required": true,
      "AllowedPattern": "^[a-zA-Z0-9_-]+$",
      "MinLength": 2,
      "MaxLength": 64
    }
  },
  "Resources": {
    "OssExport": {
      "Type": "ALIYUN::SLS::OssExport",
      "Properties": {
        "Configuration": {
          "Ref": "Configuration"
        },
        "ProjectName": {
          "Ref": "ProjectName"
        },
        "DisplayName": {
          "Ref": "DisplayName"
        },
        "ExportName": {
          "Ref": "ExportName"
        }
      }
    }
  },
  "Outputs": {
    "ProjectName": {
      "Description": "The project name of SLS.",
      "Value": {
        "Fn::GetAtt": [
          "OssExport",
          "ProjectName"
        ]
      }
    },
    "ExportName": {
      "Description": "The name of the export job.",
      "Value": {
        "Fn::GetAtt": [
          "OssExport",
          "ExportName"
        ]
      }
    }
  }
}