Convert logs to Simple Log Service traces - Simple Log Service

When you use Logtail to collect logs, you can use the processor_otel_trace plug-in to convert the collected logs to SLS traces. This topic describes the parameters of the processor_otel_trace plug-in and provides examples on how to configure the plug-in.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview of Logtail plug-ins for data processing.

Configuration description

Important

Logtail V1.7.1 and later support the processor_otel_trace plug-in.

Form configuration

Set the Processor Type parameter to Log to Trace. Then, configure other parameters based on the following table.

Parameters

Parameter	Description
SourceKey	The name of the original field.
Format	The format of data after conversion. Valid values: protobuf json protojson
NoKeyError	Specifies whether to report an error if the raw log does not contain the original field. Default value: false.
TraceIDNeedDecode	Specifies whether to decode trace IDs by using the Base64 algorithm. Default value: false. If you set the Format parameter to protojson and trace IDs are encoded by using the Base64 algorithm, you must set this parameter to true. Otherwise, the conversion fails.
SpanIDNeedDecode	Specifies whether to decode span IDs by using the Base64 algorithm. Default value: false. If you set the Format parameter to protojson and span IDs are encoded by using the Base64 algorithm, you must set this parameter to true. Otherwise, the conversion fails.
ParentSpanIDNeedDecode	Specifies whether to decode parent span IDs by using the Base64 algorithm. Default value: false. If you set the Format parameter to protojson and parent span IDs are encoded by using the Base64 algorithm, you must set this parameter to true. Otherwise, the conversion fails.

Configuration example

Raw log

INFO OtlpJsonLoggingSpanExporter - {"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"io.opentelemetry.spring-webmvc-3.1","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"eecaa7e1b63c9f61","parentSpanId":"fa85fcee7fe0e0e8","name":"HelloWorldController.sayHello","kind":1,"startTimeUnixNano":"1696659118615748056","endTimeUnixNano":"1696659128701715998","attributes":[{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}}],"events":[],"links":[],"status":{}}]},{"scope":{"name":"io.opentelemetry.tomcat-7.0","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"fa85fcee7fe0e0e8","name":"GET /hello","kind":2,"startTimeUnixNano":"1696659118479360000","endTimeUnixNano":"1696659128703706165","attributes":[{"key":"http.status_code","value":{"intValue":"200"}},{"key":"user_agent.original","value":{"stringValue":"curl/7.84.0"}},{"key":"net.host.name","value":{"stringValue":"localhost"}},{"key":"http.response_content_length","value":{"intValue":"12"}},{"key":"http.target","value":{"stringValue":"/hello"}},{"key":"net.sock.peer.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.host.port","value":{"intValue":"18080"}},{"key":"net.sock.peer.port","value":{"intValue":"53649"}},{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}},{"key":"http.route","value":{"stringValue":"/hello"}},{"key":"net.sock.host.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.protocol.name","value":{"stringValue":"http"}},{"key":"net.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"http.method","value":{"stringValue":"GET"}}],"events":[],"links":[],"status":{}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.19.0"}

Logtail plug-in configuration for data processing
Results

Editor configuration in JSON

Set the type parameter to processor_otel_trace. Then, configure other parameters in the detail parameter based on the following table.

Parameters

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the original field.
Format	String	Yes	The format of data after conversion. Valid values: protobuf, json, and protojson.
NoKeyError	Boolean	No	Specifies whether to report an error if the raw log does not contain the original field. Default value: false.
TraceIDNeedDecode	Boolean	No	Specifies whether to decode trace IDs by using the Base64 algorithm. Default value: false. If you set the Format parameter to protojson and trace IDs are encoded by using the Base64 algorithm, you must set this parameter to true. Otherwise, the conversion fails.
SpanIDNeedDecode	Boolean	No	Specifies whether to decode span IDs by using the Base64 algorithm. Default value: false. If you set the Format parameter to protojson and span IDs are encoded by using the Base64 algorithm, you must set this parameter to true. Otherwise, the conversion fails.
ParentSpanIDNeedDecode	Boolean	No	Specifies whether to decode parent span IDs by using the Base64 algorithm. Default value: false. If you set the Format parameter to protojson and parent span IDs are encoded by using the Base64 algorithm, you must set this parameter to true. Otherwise, the conversion fails.

Configuration example

Raw log

INFO OtlpJsonLoggingSpanExporter - {"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"io.opentelemetry.spring-webmvc-3.1","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"eecaa7e1b63c9f61","parentSpanId":"fa85fcee7fe0e0e8","name":"HelloWorldController.sayHello","kind":1,"startTimeUnixNano":"1696659118615748056","endTimeUnixNano":"1696659128701715998","attributes":[{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}}],"events":[],"links":[],"status":{}}]},{"scope":{"name":"io.opentelemetry.tomcat-7.0","version":"1.26.0-alpha","attributes":[]},"spans":[{"traceId":"a7adbcb0921fa4a5b90615d0ad38ef95","spanId":"fa85fcee7fe0e0e8","name":"GET /hello","kind":2,"startTimeUnixNano":"1696659118479360000","endTimeUnixNano":"1696659128703706165","attributes":[{"key":"http.status_code","value":{"intValue":"200"}},{"key":"user_agent.original","value":{"stringValue":"curl/7.84.0"}},{"key":"net.host.name","value":{"stringValue":"localhost"}},{"key":"http.response_content_length","value":{"intValue":"12"}},{"key":"http.target","value":{"stringValue":"/hello"}},{"key":"net.sock.peer.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.host.port","value":{"intValue":"18080"}},{"key":"net.sock.peer.port","value":{"intValue":"53649"}},{"key":"thread.id","value":{"intValue":"48"}},{"key":"thread.name","value":{"stringValue":"http-nio-18080-exec-1"}},{"key":"http.route","value":{"stringValue":"/hello"}},{"key":"net.sock.host.addr","value":{"stringValue":"127.0.0.1"}},{"key":"net.protocol.name","value":{"stringValue":"http"}},{"key":"net.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"http.method","value":{"stringValue":"GET"}}],"events":[],"links":[],"status":{}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.19.0"}

Logtail plug-in configuration for data processing

{
  "processors": [
    {
      "type": "processor_split_log_regex",
      "detail": {
        "PreserveOthers": true,
        "SplitKey": "content",
        "SplitRegex": "INFO\\sOtlpJsonLoggingSpanExporter\\s.+"
      }
    },
    {
      "type": "processor_regex",
      "detail": {
        "SourceKey": "content",
        "Regex": "INFO\\sOtlpJsonLoggingSpanExporter\\s-(\\s.+)",
        "Keys": [
          "trace_data"
        ]
      }
    },
    {
      "detail": {
        "SourceKey": "trace_data",
        "Format": "protojson",
        "NoKeyError": true
      },
      "type": "processor_otel_trace"
    }
  ]
}

Results