When you use Logtail to collect logs, you can use the processor_log_to_sls_metric plug-in to convert the collected logs to SLS metrics. This topic describes the parameters of the processor_log_to_sls_metric plug-in and provides examples on how to configure the plug-in.
Entry point
If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview of Logtail plug-ins for data processing.
Configuration description
Logtail V1.7.1 and later support the processor_log_to_sls_metric plug-in.
Form configuration
Set the Processor Type parameter to Log to Metric. Then, configure other parameters based on the following table.
Parameters
Parameter
Description
MetricTimeKey
Specify the time field in the log. The time field is mapped to the
__time_nano__field in a metric. By default, the value of the__time__field in the log is extracted.Make sure that the value of the time field is a valid formatted timestamp. Timestamps that are accurate to the second, millisecond, microsecond, and nanosecond are supported. Timestamps that are accurate to the second are 10 bits in length. Timestamps that are accurate to the millisecond are 13 bits in length. Timestamps that are accurate to the microsecond are 16 bits in length. Timestamps that are accurate to the nanosecond are 19 bits in length.
MetricLabelKeys
Specify keys for the
__labels__field in a metric. The keys must match the^[a-zA-Z_][a-zA-Z0-9_]*$regular expression. The values of the keys cannot contain vertical bars (|) or#$#. For more information, see Metric.You cannot specify the
__labels__field as a key of the MetricLabelKeys parameter. If the raw log contains the__labels__field, the value of the __labels__ field in the log is added to the value of the__labels__field in a metric.MetricValues
Specify the name and value of a metric.
The
__name__field in a metric specifies the metric name. The metric name must match the^[a-zA-Z_:][a-zA-Z0-9_:]*$regular expression.The
__value__field in a metric specifies the metric value. The value must be of the Double type.
For more information, see Metric.
CustomMetricLabels
Specify custom keys for the
__labels__field in a metric. The keys of the field must match the^[a-zA-Z_][a-zA-Z0-9_]*$regular expression. The values of the keys cannot contain vertical bars (|) or#$#. For more information, see Metric.IgnoreError
Specify whether to generate error logs if no logs are matched.
Configuration example
Raw log
{ "script_md5":"8c6aebe9****c27f", "remote_addr":"123.XX.XX.123", "remote_user":"-", "time_local":"15/Aug/2023:12:03:20", "method":"GET", "url":"/www.a.com", "protocol":"HTTP/1.1", "status":"404", "body_bytes_sent":"4146", "http_referer":"-", "http_user_agent":"curl/7.74.0", "http_x_forwarded_for":"-", "__labels__":"field1#$#field1|field2#$#field2|tag1#$#tag1|tag2#$#tag2", "body_bytes_sent_name":"body_bytes_sent", "status_name":"status", "httptime":"1692101000", "__time__":"1692101000" }Logtail plug-in configuration for data processing
Results
The first metric
{ "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com", "__name__":"body_bytes_sent", "__value__":"4146", "__time_nano__":"1692101000000000000", "__time__":"1692101000" }The second metric
{ "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com", "__name__":"status", "__value__":"404", "__time_nano__":"1692101000000000000", "__time__":"1692101000" }
Editor configuration in JSON
Set the type parameter to processor_log_to_sls_metric. Then, configure other parameters in the detail parameter based on the following table.
Parameters
Parameter
Type
Required
Description
MetricTimeKey
String
No
Specify the time field in the log. The time field is mapped to the
__time_nano__field in a metric. By default, the value of the__time__field in the log is extracted.Make sure that the value of the time field is a valid formatted timestamp. Timestamps that are accurate to the second, millisecond, microsecond, and nanosecond are supported. Timestamps that are accurate to the second are 10 bits in length. Timestamps that are accurate to the millisecond are 13 bits in length. Timestamps that are accurate to the microsecond are 16 bits in length. Timestamps that are accurate to the nanosecond are 19 bits in length.
MetricLabelKeys
[]String
Yes
Specify keys for the
__labels__field in a metric. The keys must match the^[a-zA-Z_][a-zA-Z0-9_]*$regular expression. The values of the keys cannot contain vertical bars (|) or#$#. For more information, see Metric.You cannot specify the
__labels__field as a key of the MetricLabelKeys parameter. If the raw log contains the__labels__field, the value of the __labels__ field in the log is added to the value of the__labels__field in a metric.MetricValues
Map
Yes
Specify the name and value of a metric.
The
__name__field in a metric specifies the metric name. The metric name must match the^[a-zA-Z_:][a-zA-Z0-9_:]*$regular expression.The
__value__field in a metric specifies the metric value. The value must be of the Double type.
For more information, see Metric.
CustomMetricLabels
Map
No
Specify custom keys for the
__labels__field in a metric. The keys of the field must match the^[a-zA-Z_][a-zA-Z0-9_]*$regular expression. The values of the keys cannot contain vertical bars (|) or#$#. For more information, see Metric.IgnoreError
Boolean
No
Specify whether to generate error logs if no logs are matched. Default value: false. The value false indicates that error logs are not exported.
Configuration example
Raw log
{ "script_md5":"8c6aebe9****c27f", "remote_addr":"123.XX.XX.123", "remote_user":"-", "time_local":"15/Aug/2023:12:03:20", "method":"GET", "url":"/www.a.com", "protocol":"HTTP/1.1", "status":"404", "body_bytes_sent":"4146", "http_referer":"-", "http_user_agent":"curl/7.74.0", "http_x_forwarded_for":"-", "__labels__":"field1#$#field1|field2#$#field2|tag1#$#tag1|tag2#$#tag2", "body_bytes_sent_name":"body_bytes_sent", "status_name":"status", "httptime":"1692101000", "__time__":"1692101000" }Logtail plug-in configuration for data processing
{ "processors":[ { "detail":{ "CustomMetricLabels":{ "nginx":"test" }, "IgnoreError":false, "MetricLabelKeys":[ "url", "method" ], "MetricTimeKey":"httptime", "MetricValues":{ "body_bytes_sent_name":"body_bytes_sent", "status_name":"status" } }, "type":"processor_log_to_sls_metric" } ] }Results
The first metric
{ "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com", "__name__":"body_bytes_sent", "__value__":"4146", "__time_nano__":"1692101000000000000", "__time__":"1692101000" }The second metric
{ "__labels__":"field1#$#field1|field2#$#field2|method#$#GET|nginx#$#test|tag1#$#tag1|tag2#$#tag2|url#$#/www.a.com", "__name__":"status", "__value__":"404", "__time_nano__":"1692101000000000000", "__time__":"1692101000" }