Simple Log Service:Use the NGINX configuration mode to collect text logs

Solution overview

Logtail processes log content in NGINX configuration mode based on the log_format directive. The primary directives for NGINX access logs, log_format and access_log, are typically set in the /etc/nginx/nginx.conf file. The log_format directive defines the log's format, while access_log specifies the storage directory for NGINX log files.

• Log Format and Storage Path

  The example below uses the default values for log_format and access_log.

  log_format main  '$remote_addr - $remote_user [$time_local] "$request" '
                   '$request_time $request_length '
                   '$status $body_bytes_sent "$http_referer" '
                   '"$http_user_agent"';
  
  access_log  /var/log/nginx/access.log  main;

  The table below details the log fields:

  Field name	Description
  remote_addr	The IP addresses of the client.
  remote_user	The username that is used by the client to send the request.
  time_local	The system time of the server. The value must be enclosed in brackets [].
  request	The URI and HTTP protocol of a request.
  request_time	The time that is required to process a request. Unit: seconds.
  request_length	The length of a request. The request line, request headers, and request body are all counted.
  status	The status of a request.
  body_bytes_sent	The number of bytes in a response that is sent to the client. The response header is not counted.
  http_referer	The URL of the source web page.
  http_user_agent	The browser information of the client.

• Raw Log

  NGINX generates logs according to the log_format definition:

  192.168.1.1 - - [11/Dec/2024:11:21:03 +0800] "GET /nginx-logo.png HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

• Logs collected in the Logstore of Simple Log Service:

Procedure

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you want to manage.

3. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to manage.

4. Expand the Logstore tab. Click Logtail Configuration and then click Add Logtail Configuration.

5. On the Quick Data Access page, select NGINX - Text Logs > Access Now.

6. Select the created machine group in the machine group configuration step.

7. Configure the following options in the Logtail configuration step.

   1. Configuration Name: Enter the name for the Logtail collection configuration. For example, use nginx-logs.

   2. File Path: Specify the log storage path. For instance, /var/log/nginx/**/access* refers to the /var/log/nginx directory and all its subdirectories containing files starting with access.

   3. Processing Configuration: Select NGINX Mode Parsing. In the Processing Plug-in tab, input the log configuration from the standard NGINX configuration file, typically starting with log_format. Simple Log Service will automatically parse the corresponding fields. For example:

      log_format main  '$remote_addr - $remote_user [$time_local] "$request" '
                       '$request_time $request_length '
                       '$status $body_bytes_sent "$http_referer" '
                       '"$http_user_agent"';

      

   4. Maintain the default settings for other configuration items. For more details, see Collect Text Logs from Hosts.

8. In the query analysis configuration step, click Refresh to preview the collected data.

9. Click Next to finalize the configuration. To query the collected logs, click Query Logs and navigate to the Logstore query analysis page. Please wait approximately 1 minute for the index to activate, after which you can view the logs on the Raw Log tab. For more details, see Quick Start for Query and Analysis.

References

• Linux users can utilize the Logtail automatic diagnosis tool provided by Simple Log Service for quick issue resolution. For more information, see How to Use the Logtail Automatic Diagnosis Tool.

• If the preview page is empty or the query page shows no data after log collection with Logtail, follow the troubleshooting steps in Troubleshoot Logtail Collection Failures.

• Errors may occur during log collection with Logtail, such as issues with parsing regular expressions, invalid file paths, or shard processing capacity being exceeded. To view Logtail collection errors, see How to View Logtail Collection Errors. For common error types in data collection, see Common Error Types of Data Collection in Simple Log Service.

• By default, a single Logtail configuration is used to collect logs from a file. To collect multiple copies of the same log, refer to How to Collect Multiple Copies of Logs from a File.

• For collecting logs from corporate intranet servers to Simple Log Service, see Collect Logs from Corporate Intranet Servers.

• To distinguish logs from different servers with identical storage paths or filenames, see Machine Group Topic. For differentiating logs by users or instances, see File Path Extraction.

• For insights on analyzing website access, diagnosing and optimizing websites, and setting alerts for key scenarios, see Analyze NGINX Access Logs.