Use Logtail to collect syslogs - Simple Log Service - Alibaba Cloud Documentation Center

This topic describes how to create a Logtail configuration in the Simple Log Service console to collect syslogs.

Prerequisites

Logtail V0.16.13 or later is installed on a Linux server, or Logtail V1.0.0.8 or later is installed on a Windows server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.

Overview

Linux servers allow you to use syslog agents such as rsyslog to forward on-premises syslogs to a server by using a specified IP address and port. After you apply a Logtail configuration to the specified server, Logtail receives the forwarded syslogs over TCP or UDP. Logtail parses the syslogs based on the specified syslog protocol, and extracts the facility, tag(program), severity, and content fields from the syslogs. The syslog protocols defined in RFC 3164 and RFC 5424 are supported.

Implementation

Logtail listens to a specific IP address and port, and receives syslog data. 实现原理

Procedure

Log on to the Simple Log Service console.
In the Import Data section, select Custom Data Plug-in.
Select the project and Logstore. Then, click Next.
Create a machine group.
- If a machine group is available, click Use Existing Machine Groups.
- If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
  1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.
    For more information, see Install Logtail on ECS instances.
    Important
    If your server is an ECS instance that belongs to a different Alibaba Cloud account from Simple Log Service, a server in a data center, or a server from a third-party cloud service provider, you must manually install Logtail on the server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server. After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.
  2. After you confirm the settings, click OK.
  3. After Logtail is installed, click Complete Installation.
  4. In the Create Machine Group step, configure the Name parameter and click Next.
    Simple Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.
Confirm that the machine group is displayed in the Applied Server Groups section and click Next.
Important
If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?

In the Specify Data Source step, configure the Config Name and Plug-in Config parameters. Then, click Next.

The following example shows how to configure Logtail to listen to 127.0.0.1 over UDP and TCP:

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "tcp://0.0.0.0:9000",
                 "ParseProtocol": "rfc3164"
             }
         },
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://0.0.0.0:9001",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Parameter	Type	Required	Description
`type`	string	Yes	The type of the data source. Set the value to `service_syslog`.
`Address`	string	No	The listening protocol, address, and port on which Logtail listens. Logtail listens on and obtains data based on the Logtail configuration. The value of the parameter is in the `[tcp/udp]://[ip]:[port] format`. Default value: `tcp://127.0.0.1:9999`. This value indicates that Logtail receives only logs that are forwarded from the on-premises server. Note The listening protocol, IP address, and port that you specify must be the same as those specified in the forwarding rule that is added to the configuration file of rsyslog. If the Logtail server uses multiple IP addresses to receive data, set the Address parameter to `0.0.0.0`. The value indicates that Logtail listens to all IP addresses of the server.
`ParseProtocol`	string	No	The protocol that you want Logtail to use to parse syslogs. By default, this parameter is empty, which indicates that syslogs are not parsed. Valid values: Empty: Logtail does not parse syslogs. rfc3164: Logtail uses the RFC 3164 protocol to parse syslogs. rfc5424: Logtail uses the RFC 5424 protocol to parse syslogs. auto: Logtail automatically selects a protocol based on the content of syslogs.
`IgnoreParseFailure`	boolean	No	Specifies whether to perform an operation on a syslog after the syslog fails to be parsed. Default value: `true`. This value indicates that the system does not parse the syslog and adds the syslog to the content field. If you set the value to `false`, the syslog is discarded after it fails to be parsed.

Preview data, configure indexes, and then click Next.
By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
Important
If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
Click Log Query. You are redirected to the query and analysis page of your Logstore.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Troubleshooting

If no data is displayed on the preview page or query page after logs are collected by using Logtail, you can run logger commands to send data to local Logtail. Then, you can check whether the issue occurred due to the writer, network, or Logtail.

If local Logtail can receive data, the issue occurred due to the writer or network.

logger -n localhost -P 9000 -T "This is a TCP syslog message"
logger -n localhost -P 9001 -d "This is a UDP syslog message"

For more information, see What do I do if errors occur when I use Logtail to collect logs?

Sample log

The following figure shows a sample syslog.

Logtail配置生效

Field	Description
`_hostname_`	The hostname. If no hostname is included in the log, the hostname of the current host is obtained.
`_program_`	The tag field in the syslog protocol.
`_priority_`	The priority field in the syslog protocol.
`_facility_`	The facility field in the syslog protocol.
`_severity_`	The severity field in the syslog protocol.
`_unixtimestamp_`	The timestamp of the log.
`_content_`	The content of the log. If the log fails to be parsed, this field contains the content of the raw log.
`_ip_`	The IP address of the current host.
`_client_ip_`	The IP address of the syslog client that transfers logs.

Examples

Use rsyslog to forward syslogs to Simple Log Service

Add a forwarding rule for rsyslog.
1. Modify the /etc/rsyslog.conf configuration file of rsyslog on the server from which you want to collect syslogs. Add a forwarding rule to the end of the configuration file.
  After the forwarding rule is added, rsyslog forwards syslogs to a specified IP address and port.
  - If Logtail resides on the syslog server, you must specify the IP address 127.0.0.1 and a non-well-known port that is unoccupied in the forwarding rule.
  - If Logtail resides on a different server from the syslog server, you must specify the public IP address of the different server and an idle non-well-known port in the forwarding rule.
  The following example shows a forwarding rule, which allows all syslogs to be forwarded to 127.0.0.1:9000 over TCP. For more information about the configuration file, see RSyslog Documentation.
```
*.* @@127.0.0.1:9000
```
2. Run the following command to restart rsyslog and validate the forwarding rule:
```
sudo service rsyslog restart
```

Create a Logtail configuration. For more information, see Procedure.

The following code shows the Logtail configuration:

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "tcp://0.0.0.0:9000",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Collect NGINX logs

NGINX servers allow you to forward access logs to specified IP addresses and ports by using the syslog protocol. If you want to deliver all data of a server as syslogs to Simple Log Service, you can create a Logtail configuration to collect the data. The data includes NGINX access logs.

Add a forwarding rule for NGINX.
1. Add a forwarding rule to the nginx.conf configuration file on the NGINX server. For more information, see NGINX Beginner's Guide.
  The following sample code provides an example of a forwarding rule:
```
http {
    ...

    # Add this line.
    access_log syslog:server=127.0.0.1:9000,facility=local7,tag=nginx,severity=info combined;

    ...
}
                                
```
2. Run the following command to restart the NGINX service and validate the forwarding rule:
```
sudo service nginx restart
```

Create a Logtail configuration. For more information, see Procedure.

The following code shows the Logtail configuration:

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://127.0.0.1:9000",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Collect firewall logs

In most cases, firewalls allow you to forward access logs to specified IP addresses and ports by using the syslog protocol. If you want to deliver the logs of a firewall as syslogs to Simple Log Service to build a security information and event management (SIEM) solution, you can create a Logtail configuration to collect the data.

Add a forwarding rule for the firewall based on the firewall manual.
In this example, the IP address of the server on which Logtail is installed is 10.20.30.40.

Create a Logtail configuration. For more information, see Procedure.

The following code shows the Logtail configuration:

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "tcp://0.0.0.0:9000",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }