LoongCollector collects text logs from servers (manual installation)

Step 1: Install LoongCollector

The following table describes the methods that you can select to download and install LoongCollector.

#Install LoongCollector by using the Internet
chmod +x loongcollector.sh; ./loongcollector.sh install ${region_id}-internet
#Install LoongCollector by using the transfer acceleration feature. If the server on which you want to install LoongCollector is from a third-party cloud service provider outside China or in a data center outside China, data transfer over the Internet may cause high network latency and unstable connections. In this case, we recommend that you use the transfer acceleration feature to transfer logs. Regions on Financial Cloud are not supported.
chmod +x loongcollector.sh; ./loongcollector.sh install ${region_id}-acceleration

#Install LoongCollector by using the Internet
chmod +x loongcollector.sh; ./loongcollector.sh install-local ${region_id}-internet
#Install LoongCollector by using the transfer acceleration feature. If the server on which you want to install LoongCollector is from a third-party cloud service provider outside China or in a data center outside China, data transfer over the Internet may cause high network latency and unstable connections. In this case, we recommend that you use the transfer acceleration feature to transfer logs. Regions on Financial Cloud are not supported.
chmod +x loongcollector.sh; ./loongcollector.sh install-local ${region_id}-acceleration

Step 3: Configure log collection

1. Click the Logstore in the Log Storage section. In the Logtail Configurations section, click Add. Click Import Data. In this example, select Regex - Text Log.

2. Click Host > ECS. Select the machine group that you created and click the > icon to add the machine group to the Applied Server Group section. Then, click Next.

3. In the Global Configurations section, configure the Configuration Name parameter.

   

4. In the Input Configurations section, configure the File Path parameter. The File Path parameter specifies the directory used to store the logs that you want to collect. The file path must start with a forward slash (/). In this example, the File Path parameter is set to /data/wwwlogs/main/**/*.Log, which indicates that logs are collected from files suffixed with .Log in the /data/wwwlogs/main directory. You can configure the Maximum Directory Monitoring Depth parameter to specify the maximum number of levels of the subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of the subdirectories that the ** wildcard characters can match in the value of the File Path parameter. The value 0 specifies that only the specified log file directory is monitored.

   

5. In the Processor Configurations section, configure the Log Sample, Multi-line Mode, and Processing Method parameters.

   1. Log Sample: Enter a sample log that is collected from an actual scenario in the Log Sample field. Sample logs can help you easily configure log processing-related parameters.

   2. Multi-line Mode: Turn on Multi-line Mode based on your business requirements. A multi-line log spans multiple consecutive lines. If you turn off Multi-line Mode, Simple Log Service collects logs in single-line mode. Each log is placed in a line. If you turn on Multi-line Mode, you must configure the following parameters:

      • Type:

        • Custom: If the format of raw data is not fixed, configure the Regex to Match First Line parameter to match the beginning of the first line of a log. If you set the Regex to Match First Line parameter to \[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*, the raw data in the following sample code is split into two logs. Note that the value of the Regex to Match First Line parameter must match the entire line of data.

          [2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
              at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
              at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
              at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
          [2023-10-01T10:31:01,000] [INFO] java.lang.Exception: exception happened

        • Multi-line JSON: If the raw data is in the standard JSON format, set the Type parameter to Multi-line JSON. LoongCollector automatically processes the line feeds that occur within a JSON-formatted log.

      • Processing Method If Splitting Fails

        • Discard: discards the text.

        • Retain Single Line: saves each line of the text as a log.

   3. Processing Method: This parameter specifies the method that is used to split a log. In this example, LoongCollector collects text logs in full regex mode and a Data Parsing (Regex Mode) processing plug-in is automatically generated. You can use other processing plug-ins based on your business requirements.

      The following section describes the settings of common processing plug-ins. For more information about the capabilities of processing plug-ins such as Time Parsing, Data Filtering, and Data Masking, see Overview of Logtail plug-ins for data processing. Simple Log Service also provides Simple Log Service Processing Language (SPL)-based data processing. SPL-based data processing has the processing capabilities of traditional processing plug-ins but is more efficient than the processing plug-ins. For more information, see Use Logtail SPL to parse logs.

      Data Parsing (Regex Mode) plug-in

      Data Parsing (JSON Mode) plug-in

      Data Parsing (Apache Mode) plug-in

      Data Parsing (NGINX Mode) plug-in

      Data Parsing (IIS Mode) plug-in

      Data Parsing (Delimiter Mode) plug-in

      SPL-based data processing

      Select Data Parsing (Regex Mode) from the Processor Type drop-down list to go to the detailed configuration page of the plug-in.

      On the page, configure the Regular Expression parameter and specify keys based on the extracted values. Click Generate below the Regular Expression field, select specific content in the sample log based on the following figure, and then click Generate Regular Expression in the popover that appears. Then, Simple Log Service automatically generates a regular expression for the selected content.

      

      After the regular expression is generated, specify keys based on the extracted values in the Extracted Field parameter. The key-value pairs can be used to create indexes. After you complete the settings, click OK. Then, click Next.

      Important

      If your want to process the collected JSON logs, you can add a Data Parsing (JSON Mode) plug-in.

      JSON logs can be written in the object or array structure. A log in the object structure contains key-value pairs, and a log in the array structure contains an ordered list of values. You can use a Data Parsing (JSON Mode) plug-in to parse JSON logs of the object type and extract key-value pairs from the first layer of each object. The extracted keys are used as field names, and the extracted values are used as field values. You cannot use a Data Parsing (JSON Mode) plug-in to parse JSON logs of the array type. To parse data in a fine-grained manner, see Expand JSON fields.

      Turn on Multi-line Mode based on your business requirements. If you turn on Multi-line Mode, you must configure the following parameters:

      • Type: Set the parameter to Multi-line JSON.

      • Processing Method If Splitting Fails: Set the parameter to Retain Single Line.

      Delete the Data Parsing (Regex Mode) plug-in from the Processing Method list. Add a Data Parsing (JSON Mode) plug-in and click OK. Then, click Next.

      

      Note

      You can use a Data Parsing (Apache Mode) plug-in to parse Apache logs into structured data based on the log format that you specify in the Apache configuration file. In this case, a log is parsed into multiple key-value pairs.

      Delete the Data Parsing (Regex Mode) plug-in from the Processing Method list and then add a Data Parsing (Apache Mode) plug-in.

      The following table describes the parameters that you must configure to add a Data Parsing (Apache Mode) plug-in. After you complete the settings, click OK. Then, click Next.

      Parameter	Description
      Log Format	The log format that you specify in the Apache configuration file. Valid values: common, combined, and Custom.
      APACHE LogFormat Configuration	The log configuration section that you specify in the Apache configuration file. In most cases, the log configuration section starts with LogFormat. If you set the Log Format parameter to common or combined, the system automatically assigns a value into this field. Check whether the value is the same as that you specify in the Apache configuration file. If you set the Log Format parameter to Custom, specify a value based on your business requirements. For example, you can enter LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized.
      Original Field	The original field that stores the log content before parsing. Default value: content.
      Regular Expression	The regular expression that is used to extract Apache logs. Simple Log Service automatically generates a regular expression based on the value of the APACHE LogFormat Configuration field.
      Extracted Field	The key that is automatically extracted based on the value of the APACHE LogFormat Configuration field.
      Retain Original Field if Parsing Fails	Specifies whether to retain the original field in the new log that is obtained after the raw log fails to be parsed.
      Retain Original Field if Parsing Succeeds	Specifies whether to retain the original field in the new log that is obtained after parsing.
      New Name of Original Field	The new name of the original field that you want to retain. If you select Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds, you can rename the original field that stores the original log content.

      Note

      You can use a Data Parsing (NGINX Mode) plug-in to parse NGINX logs into structured data based on log_format. In this case, a log is parsed into multiple key-value pairs.

      Delete the Data Parsing (Regex Mode) plug-in from the Processing Method list and then add a Data Parsing (NGINX Mode) plug-in. The following table describes the parameters that you must configure to add a Data Parsing (NGINX Mode) plug-in. After you complete the settings, click OK. Then, click Next.

      Parameter	Description
      NGINX Log Configuration	The log configuration section that you specify in the NGINX configuration file. The log configuration section starts with log_format. Example: log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$request_time $request_length ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent"'; For more information, see Introduction to NGINX logs.
      Original Field	The original field that stores the log content before parsing. Default value: content.
      Regular Expression	The regular expression that is used to extract NGINX logs. Simple Log Service automatically generates a regular expression based on the value of the NGINX Log Configuration field.
      Extracted Field	The key that is automatically extracted based on the value of the NGINX Log Configuration field.
      Retain Original Field if Parsing Fails	Specifies whether to retain the original field in the new log that is obtained after the raw log fails to be parsed.
      Retain Original Field if Parsing Succeeds	Specifies whether to retain the original field in the new log that is obtained after parsing.
      New Name of Original Field	The new name of the original field that you want to retain. If you select Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds, you can rename the original field that stores the original log content.

      Note

      You can use a Data Parsing (IIS Mode) plug-in to parse Internet Information Services (IIS) logs into structured data based on the log format that you specify. In this case, a log is parsed into multiple key-value pairs.

      Delete the Data Parsing (Regex Mode) plug-in from the Processing Method list and then add a Data Parsing (IIS Mode) plug-in.

      The following table describes the parameters that you must configure to add a Data Parsing (IIS Mode) plug-in. After you complete the settings, click OK. Then, click Next.

      Parameter	Description
      Log Format	The format of the logs that are generated on the IIS server. Valid values: IIS: Microsoft IIS log file format NCSA: NCSA Common log file format W3C: W3C Extended log file format
      IIS Configuration Fields	The IIS configuration fields. If you set the Log Format parameter to IIS or NCSA, the system automatically specifies the IIS configuration fields. If you set the Log Format parameter to W3C, enter the content that is specified in the logExtFileFlags parameter of the IIS configuration file. logExtFileFlags="Date, Time, ClientIP, UserName, SiteName, ComputerName, ServerIP, Method, UriStem, UriQuery, HttpStatus, Win32Status, BytesSent, BytesRecv, TimeTaken, ServerPort, UserAgent, Cookie, Referer, ProtocolVersion, Host, HttpSubStatus" Default path of the IIS5 configuration file: C:\WINNT\system32\inetsrv\MetaBase.bin Default path of the IIS6 configuration file: C:\WINDOWS\system32\inetsrv\MetaBase.xml Default path of the IIS7 configuration file: C:\Windows\System32\inetsrv\config\applicationHost.config
      Original Field	The original field that stores the log content before parsing. Default value: content.
      Regular Expression	The regular expression that is used to extract IIS logs. Simple Log Service automatically generates a regular expression based on the value of the IIS Configuration Fields field.
      Extracted Field	The key that is automatically extracted based on the value of the IIS Configuration Fields field.
      Retain Original Field if Parsing Fails	Specifies whether to retain the original field in the new log that is obtained after the raw log fails to be parsed.
      Retain Original Field if Parsing Succeeds	Specifies whether to retain the original field in the new log that is obtained after parsing.
      New Name of Original Field	The new name of the original field that you want to retain. If you select Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds, you can rename the original field that stores the original log content.

      Note

      You can use a Data Parsing (Delimiter Mode) plug-in to parse logs into structured data by using delimiters. In this case, a log is parsed into multiple key-value pairs.

      Delete the Data Parsing (Regex Mode) plug-in from the Processing Method list and then add a Data Parsing (Delimiter Mode) plug-in.

      The following table describes the parameters that you must configure to add a Data Parsing (Delimiter Mode) plug-in. After you complete the settings, click OK. Then, click Next.

      Parameter	Description
      Original Field	The original field that stores the log content before parsing. Default value: content.
      Delimiter	The delimiter. Select a delimiter based on the actual log content. For example, you can select Vertical Bar (|). Note If you set the Delimiter parameter to Non-printable Character, you must enter a character in the following format: 0x<Hexadecimal ASCII code of the non-printable character>. For example, if you want to use a non-printable character whose hexadecimal ASCII code is 01, you must enter 0x01.
      Quote	The quote. If a log field contains delimiters, you must specify a quote to enclose the field. Simple Log Service parses the content that is enclosed in a pair of quotes into a complete field. Select a quote based on the format of the logs that you want to collect. Note If you set the Quote parameter to Non-printable Character, you must enter a character in the following format: 0x<Hexadecimal ASCII code of the non-printable character>. For example, if you want to use a non-printable character whose hexadecimal ASCII code is 01, you must enter 0x01.
      Extracted Field	If you specify a sample log, Simple Log Service can automatically extract log content based on the specified sample log and the delimiter. Configure the Key parameter for each Value parameter. The Key parameter specifies the field name. The Value parameter specifies the extracted content. If you do not specify a sample log, the Value column is unavailable. You must specify keys based on the actual logs and the delimiter. A key can contain only letters, digits, and underscores (_) and must start with a letter or an underscore (_). A key can be up to 128 bytes in length.
      Allow Missing Field	Specifies whether to upload a log to Simple Log Service if the number of extracted values is less than the number of specified keys. If you select the Allow Missing Field parameter, the log is uploaded to Simple Log Service. In this example, the log is 11|22|33|44, the Delimiter parameter is set to Vertical Bar (|), and the keys are set to A, B, C, D, and E. The value of the E field is empty. If you select the Allow Missing Field parameter, the log is uploaded to Simple Log Service. If you do not select the Allow Missing Field parameter, the log is discarded.
      Processing Method of Field to which Excess Part is Assigned	The method that is used to process excess values that are extracted if the number of extracted values is greater than the number of specified keys. Valid values: Expand: retains the excess values and adds the values to the fields in the __column$i__ format. $i specifies the sequence number of the excess field. The sequence number starts from 0. Examples: __column0__ and __column1__. Retain: retains the excess values and adds the values to the __column0__ field. Drop: discards the excess values.
      Retain Original Field if Parsing Fails	Specifies whether to retain the original field in the new log that is obtained after the raw log fails to be parsed.
      Retain Original Field if Parsing Succeeds	Specifies whether to retain the original field in the new log that is obtained after parsing.
      New Name of Original Field	The new name of the original field that you want to retain. If you select Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds, you can rename the original field that stores the original log content.

      Simple Log Service also provides custom SPL-based data processing. Compared with traditional processing plug-ins, SPL-based data processing features faster processing speed and higher processing efficiency, and is more intelligent and easier to use. As a result, SPL-based data processing greatly improves the overall capabilities of Simple Log Service. You can process data based on specific SPL statements and the computing capabilities of Simple Log Service. For more information, see Use Logtail SPL to parse logs.

6. Configure data query and analysis

   Approximately 1 minute is required to create a LoongCollector configuration. If you create a LoongCollector configuration for a Logstore for the first time and specific conditions are met, the LoongCollector configuration is created. The following conditions are included: Automatic refresh is complete. Incremental logs exist in the specified log file directory. Data can be previewed. After the LoongCollector configuration is created, click Next. The LoongCollector configuration-related settings are complete.

   By default, full-text indexing is enabled for Simple Log Service. In this case, full-text indexes are created. You can query all fields in logs based on the indexes. You can also manually create indexes for fields based on the collected logs. Alternatively, you can click Automatic Index Generation. Then, Simple Log Service generates indexes for fields. You can query data in an accurate manner based on field indexes. This reduces indexing costs and improves query efficiency. For more information, see Create indexes.

   If you use a Data Parsing (Regex Mode) plug-in to process the collected logs, the extracted key-value pairs are automatically displayed in the Field Search section.