Collect logs whose timestamps are accurate to the nanosecond

Sample raw log:

2023.11.06-15.12.12.123456,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

1. In the Logtail Configuration step of the wizard, click Other Global Configurations. In the Other Global Configurations section, turn on Advanced Parameters. In the Advanced Parameters field, enter { "EnableTimestampNanosecond": true}.

   For more information about how to collect logs, see Collect text logs from servers.

   

2. Add the Data Parsing (Delimiter Mode) plug-in. For more information, see Parsing in delimiter mode.

   

3. Add the Time Parsing plug-in. For more information, see Time parsing.

   You must configure a time format for the Time Parsing plug-in. For example, if the time field that is parsed by the Data Parsing (Delimiter Mode) plug-in from the raw log is 2023.10.26-20.58.12.123456, you must set the time format to %Y.%m.%d-%H.%M.%S.%f. %f indicates the fractional part of the second. The highest precision that is supported by the Time Parsing plug-in is the nanosecond. You must configure a time format that is consistent with the time format of the raw log. If the formats are inconsistent, the timestamp that is accurate to the nanosecond cannot be parsed. For more information, see FAQ. For more information about time formats, see Time formats.

   

4. View the parsing results in the Simple Log Service console. After you configure indexes, you can view the parsed timestamp that is accurate to the nanosecond. For more information, see Create indexes.

   

Sample raw log:

{
  "asctime": "2023-10-25 23:51:10,199999999",
  "filename": "generate_data.py",
  "levelname": "INFO",
  "lineno": 51,
  "module": "generate_data",
  "message": "{\"no\": 14, \"inner_loop\": 166, \"loop\": 27451, \"uuid\": \"9be98c29-22c7-40a1-b7ed-29ae6c8367af\"}",
  "threadName": "MainThread"
}

1. In the Logtail Configuration step of the wizard, click Other Global Configurations. In the Other Global Configurations section, turn on Advanced Parameters. In the Advanced Parameters field, enter { "EnableTimestampNanosecond": true}.

   For more information about how to collect logs, see Collect text logs from servers.

   

2. Add the Expand JSON Field plug-in. For more information, see Expand JSON fields.

   Note

   The sample raw low is in the single-level JSON format. When you expand JSON data in this format, you can use "" to connect expanded keys. When you expand JSON data in the multi-level JSON format, you can use "_" to connect expanded keys.

   

3. Add the Extract Log Time (strptime Time Format) plug-in. For more information, see Time format supported by strptime.

   You must configure a time format for the Extract Log Time (strptime Time Format) plug-in. For example, if the time field that is parsed from the raw log is "asctime": "2022-04-29 21:37:40,251", you must set the time format to %Y-%m-%d %H:%M:%S,%f. %f indicates the fractional part of the second. The highest precision that is supported by the Extract Log Time (strptime Time Format) plug-in is the nanosecond. You must configure a time format that is consistent with the time format of the raw log. For more information, see Time formats.

   

4. View the parsing results in the Simple Log Service console. After you configure indexes, you can view the parsed timestamp that is accurate to the nanosecond. For more information, see Create indexes.

   

Sample raw log:

{
  "asctime": "2023-10-25 23:51:10,199999999",
  "filename": "generate_data.py",
  "levelname": "INFO",
  "lineno": 51,
  "module": "generate_data",
  "message": "{\"no\": 14, \"inner_loop\": 166, \"loop\": 27451, \"uuid\": \"9be98c29-22c7-40a1-b7ed-29ae6c8367af\"}",
  "threadName": "MainThread"
}

1. In the Logtail Configuration step of the wizard, click Other Global Configurations. In the Other Global Configurations section, turn on Advanced Parameters. In the Advanced Parameters field, enter { "EnableTimestampNanosecond": true}.

   For more information about how to collect logs, see Collect text logs from servers.

   

2. Add the Expand JSON Field plug-in. For more information, see Expand JSON fields.

   Note

   The sample raw low is in the single-level JSON format. When you expand JSON data in this format, you can use "" to connect expanded keys. When you expand JSON data in the multi-level JSON format, you can use "_" to connect expanded keys.

   

3. Add the Extract Log Time (Go Time Format) plug-in. For more information, see Time format supported by Go.

   You must configure a time format for the Extract Log Time (Go Time Format) plug-in based on the time format specification of Go. Go uses a specific layout reference to represent a time format instead of the common %Y-%m-%d %H:%M:%S format. The reference time in Go is 2006-01-02 15:04:05 -0700 MST. For example, the time value 2023-10-25 01:36:10,199999999 is parsed based on the 2006-01-02 15:04:05,999999999 format.

   The following code provides the example layouts of time formats in Go:

   const (
       Layout      = "01/02 03:04:05PM '06 -0700" // The reference time, in numerical order.
       ANSIC       = "Mon Jan _2 15:04:05 2006"
       UnixDate    = "Mon Jan _2 15:04:05 MST 2006"
       RubyDate    = "Mon Jan 02 15:04:05 -0700 2006"
       RFC822      = "02 Jan 06 15:04 MST"
       RFC822Z     = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
       RFC850      = "Monday, 02-Jan-06 15:04:05 MST"
       RFC1123     = "Mon, 02 Jan 2006 15:04:05 MST"
       RFC1123Z    = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
       RFC3339     = "2006-01-02T15:04:05Z07:00"
       RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
       Kitchen     = "3:04PM"
       // Handy time stamps.
       Stamp      = "Jan _2 15:04:05"
       StampMilli = "Jan _2 15:04:05.000"
       StampMicro = "Jan _2 15:04:05.000000"
       StampNano  = "Jan _2 15:04:05.000000000"
   )

   

4. View the parsing results in the Simple Log Service console. After you configure indexes, you can view the parsed timestamp that is accurate to the nanosecond. For more information, see Create indexes.