Collect logs from servers in a corporate intranet - Simple Log Service

This topic describes how to configure a forward proxy server and collect logs from servers in a corporate intranet to Simple Log Service in proxy mode. In this topic, NGINX is used as an example.

Prerequisites

A project and a Logstore are created. For more information, see Create a project and Create a Logstore.
Logtail V1.5.0 or later is installed on your Linux server or Logtail V1.5.0.0 or later is installed on your Windows server. For more information, see Install Logtail on a Linux server and Install Logtail on a Windows server.

Background information

For example, you deployed multiple servers in a corporate intranet, and the servers do not have access to the Internet. If you want to collect logs from the servers to Simple Log Service for query and analysis, you can authorize one of the servers to access the Internet and collect logs from the other servers to Simple Log Service in proxy mode. You can configure an intranet server that has access to the Internet as the forward proxy server.

Implementation

Logtail exchanges data with Simple Log Service, including control data, service data, and monitoring data. Control data includes the information that is related to Logtail configuration delivery and authentication. By default, service data and monitoring data are transmitted over HTTP. HTTPS is also supported. Therefore, a proxy server must support both HTTP and HTTPS.

NGINX is an open source and high-performance HTTP proxy server. By default, NGINX supports HTTP but does not support HTTPS because of authentication-related reasons. To enable NGINX to support HTTPS, you must install an HTTPS patch for NGINX.

Step 1: Configure a proxy server

The following procedure describes how to use NGINX to configure a server that has access to the Internet in a corporate intranet as a forward proxy server:

Log on to the server that you want to configure as a forward proxy server.
Download the NGINX installation package and an HTTPS patch.
1. Download an HTTPS patch.
```
git clone https://github.com/chobits/ngx_http_proxy_connect_module.git
```
2. Download the NGINX installation package and decompress the package.
  ${version} specifies the version of NGINX. You must replace ${version} with the actual value. For more information about the latest version of NGINX, see nginx: download.
```
wget http://nginx.org/download/nginx-${version}.tar.gz
tar -xzvf nginx-${version}.tar.gz
cd nginx-${version}/
```
Add the HTTPS patch to NGINX.
${patchfile} specifies the path to the HTTPS patch file. You must specify a patch file based on the version of NGINX. For more information, see Select patch.
```
patch -p1 < ../ngx_http_proxy_connect_module/patch/${patchfile}.patch
```

Install NGINX.

./configure --add-module=../ngx_http_proxy_connect_module
make && make install

Add the following settings to the nginx.conf file.

You must replace ${Listening port of the proxy server} and ${DNS server address} with the actual values.

server {
     listen                         ${Listening port of the proxy server};
     resolver                       ${DNS server address};

     # Specify the proxy for non-HTTP requests. 
     proxy_connect;
     proxy_connect_allow            443;
     proxy_connect_connect_timeout  10s;
     proxy_connect_data_timeout     10s;

     # Specify the proxy for HTTP requests. 
     location / {
         proxy_pass http://$host;
         proxy_set_header Host $host;
     }
}

Start the NGINX server.

Step 2: Configure proxy-related environment variables

Linux operating system

The following table describes the solutions that you can use to configure proxy-related environment variables.

Solution	Advantage	Disadvantage	Scenario
Solution 1	The configurations take effect only for the Logtail process. The impact is within an acceptable scope.	The configuration method is relatively complex.	Suitable for server users who are not familiar with the overall server network.
Solution 2	The configuration method is simple.	The configurations take effect on the entire server. The impact is significant.	Suitable for server administrators who are familiar with the request status of all processes on the server.

Solution 1

Log on to a server in the corporate intranet.

Open the /etc/init.d/ilogtaild file, add the following environment variables to the start() function, and then save the file.

For more information about environment variables, see Appendix: Proxy-related environment variables.

start()
{
    cd $BIN_DIR
    umask $UMASK
    # Add proxy-related environment variables before $BIN_DIR/ilogtail. 
    # In this example, the ALL_PROXY environment variable is added. The IP address of the proxy server is 192.168.1.0, and the listening port is 9000. 
    # The server in the corporate intranet communicates with the proxy server over HTTP. 
    ALL_PROXY=http://192.168.1.0:9000 $BIN_DIR/ilogtail
    RETVAL=$?
}

Run the following command to restart Logtail:
```
/etc/init.d/ilogtaild restart
```
Repeat Steps 1 to 3 to configure proxy-related environment variables for the other servers in the corporate intranet.

Solution 2

Important

You can use this solution if you want to forward all requests of the servers in a corporate intranet by using a proxy server. You can also use this solution if you want to forward only requests from Logtail processes by using a proxy server. To use this solution, make sure that you are familiar with the addresses to which the requests are sent from the other processes on the servers. In other scenarios, use Solution 1.

Log on to a server in the corporate intranet.
Run the export command to add proxy-related environment variables to the ~/.bash_profile or /etc/profile startup file.
For more information about environment variables, see Appendix: Proxy-related environment variables.
Run the following command to allow the environment variables to take effect.
In this example, the ~/.bash_profile startup file is used.
```
source ~/.bash_profile
```
Run the following command to restart Logtail:
```
/etc/init.d/ilogtaild restart
```
Repeat Steps 1 to 4 to configure proxy-related environment variables for the other servers in the corporate intranet.

Windows operating system

Open the Run window, enter regedit in the Open field, and then click OK.
In the Registry Editor window, search for Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\LogtailDaemon and click LogtailDaemon.
Right-click LogtailDaemon and choose New > Multi-String Value. Then, specify Environment for Value name.
Double-click Environment. In the Edit String dialog box, enter proxy-related environment variables in the Value data field and click OK.
In this example, the ALL_PROXY environment variable is added. The IP address of the proxy server is 192.168.1.0, and the listening port is 9000. The server in the corporate intranet communicates with the proxy server over HTTP. For more information about environment variables, see Appendix: Proxy-related environment variables.
Open the Run window, enter services.msc in the Open field, and then click OK.
In the Services window, click the required Logtail service.
- For Logtail V0.x.x.x, click the LogtailWorker service.
- For Logtail V1.0.0.0 or later, click the LogtailDaemon service.
Right-click the service and click Restart.

Step 3: Test network connectivity

Log on to a server in the corporate intranet.

Run the following commands.

In the following commands, ${region} specifies the region of the project that is used, and ${project_name} specifies the name of the project. Replace the variables with the actual values.

curl http://logtail.${region}.log.aliyuncs.com
curl https://logtail.${region}.log.aliyuncs.com
curl http://${project_name}.${region}.log.aliyuncs.com
curl http://ali-${region}-sls-admin.${region}.log.aliyuncs.com

If information similar to the following code is returned, the network is connected:

{"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"62591BC7C08B7BD4AA99FCD4"}}

Repeat Steps 1 and 2 to test the network connectivity of the other servers in the corporate intranet.

Appendix: Proxy-related environment variables

This section describes the environment variables that you can configure for an intranet server.

Important

The environment variables can be in uppercase or lowercase. The environment variables in uppercase take precedence.

To send HTTP and HTTPS data to the same proxy server, add the ALL_PROXY environment variable.
```
ALL_PROXY: ${Address of the forward proxy server}
```
To send HTTP and HTTPS data to different proxy servers, add the HTTP_PROXY and HTTPS_PROXY environment variables.
```
HTTP_PROXY: ${Address of the HTTP proxy server}
HTTPS_PROXY: ${Address of the HTTPS proxy server}
```
The address of a proxy server must be in the [Protocol://[Username:Password@]]Address[:Port] format.
- Protocol: the communication protocol that is used between the current server in the corporate intranet and the proxy server. Valid values: http, https, and socks5. This field is optional. If you do not configure this field, http is used by default.
- Username and Password: the username and password that are used to log on to the proxy server. These fields are optional.
- Address: the IP address of the proxy server. This field is required.
- Port: the listening port of the proxy server that is configured in the nginx.conf file. This field is optional. For more information, see Step 5 in the "Configure a proxy server" section. If you do not configure this field, port 80 is used by default.
If you do not want to send specific data from a server in the corporate intranet to the proxy server, you can add the NO_PROXY environment variable. The NO_PROXY environment variable specifies the addresses to which data can be sent without using the proxy server. You must separate multiple addresses with commas (,). Supported formats:
- IP addresses.
- Domain names. The domain names can start with a period (.). A domain name and its subdomains can be matched.
- Asterisk (*). If you specify an asterisk, the proxy server is disabled.