Simple Log Service:Collect Log4j logs

Prerequisites

• A project and a Logstore are created. For more information, see Create a project and Create a Logstore.

• If you use an appender to collect logs, a Resource Access Management (RAM) user must be created and the permissions to write data to Simple Log Service must be granted to the RAM user. For more information, see Configure the permission assistant feature

Procedure

Step 1: Add the configuration file of Log4j2 to the project

1. Add dependencies to the Maven project.

   <dependencies>
     <dependency>
       <groupId>org.apache.logging.log4j</groupId>
       <artifactId>log4j-api</artifactId>
       <version>2.23.1</version>
     </dependency>
     <dependency>
       <groupId>org.apache.logging.log4j</groupId>
       <artifactId>log4j-core</artifactId>
       <version>2.23.1</version>
     </dependency>
   </dependencies>

2. Modify the configuration file.

   By default, the system searches for the configuration file in the classpath path. If the configuration file does not exist, you must manually create the file. In the following example, two appenders are defined in the log4j2.xml file. The appenders are used to output logs to the Simple Log Service console and the app.log file, respectively. The log level of the root logger is defined as ERROR, and the logs are output to the Simple Log Service console. The log level of the logger named com.example.demo.log is defined as WARN, and the logs are output to the app.log file. For more information about how to configure Log4j2, see Log4j - Configuring Log4j 2 (apache.org)

   !" -- status="WARN". The internal status of Log4j2. This parameter is not used to capture and report potential issues during configurations.  -->
   <Configuration status="WARN">
       <! -- Define the two appenders  start  -->
       <Appenders>
           <! -- Define an appender named Console to print logs to system standard output.  -->
           <Console name="Console" target="SYSTEM_OUT">
               <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss:SSS zzz} [%t] %-5level %logger{36} - %msg%n"/>
           </Console>
           <! -- Define an appender named MyFile to print logs to the app.log file. -->
           <File name="MyFile" fileName="app.log">
               <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss:SSS zzz} [%t] %-5level %logger{36} - %msg%n"/>
           </File>
       </Appenders>
       <! -- Define the two appenders   end  -->
   
       <!-- Configure loggers      start  -->
       <Loggers>
           <! -- Define a logger named com.example.demo.log and set the log level to TRACE. The com.example.demo.log logger references the MyFile appender. The logger outputs logs whose levels are TRACE and higher to the MyFile appender.  -->
           <Logger name="com.example.demo.log" level="warn">
               <AppenderRef ref="MyFile" />
           </Logger>
           <!-- Define the root logger and set the log level to ERROR. The root logger references to the Console appender. The logger outputs logs whose levels are ERROR and higher to the Simple Log Service console.  -->
           <Root level="error">
               <AppenderRef ref="Console" />
           </Root>
       </Loggers>
       <!-- Configure loggers      end  -->
   </Configuration>

3. Start to output logs.

   Write test code to read the defined appenders and output logs. Sample code:

       public void logExampleDemo() {
           // Obtain the logger named com.example.demo.log.
           Logger logger = LogManager.getLogger("com.example.demo.log");
   
           // Record log data at different levels.
           logger.trace("trace level");
           logger.debug("debug level");
           logger.info("info level");
           logger.warn("warn level");
           logger.error("error level");
           logger.fatal("fatal level");
       }

4. The following sample code provides an example of the log output:

   2024-05-28 13:37:16:295 CST [http-nio-8080-exec-8] TRACE com.example.demo.log - trace level
   2024-05-28 13:37:16:296 CST [http-nio-8080-exec-8] DEBUG com.example.demo.log - debug level
   2024-05-28 13:37:16:296 CST [http-nio-8080-exec-8] INFO  com.example.demo.log - info level
   2024-05-28 13:37:16:296 CST [http-nio-8080-exec-8] WARN  com.example.demo.log - warn level
   2024-05-28 13:37:16:296 CST [http-nio-8080-exec-8] ERROR com.example.demo.log - error level
   2024-05-28 13:37:16:296 CST [http-nio-8080-exec-8] FATAL com.example.demo.log - fatal level

Step 2: Collect Log4j logs to Simple Log Service

Use a Log4j2 appender to collect Log4j logs

1. Add dependencies to the Maven project.

   <dependency>
       <groupId>com.google.protobuf</groupId>
       <artifactId>protobuf-java</artifactId>
       <version>2.5.0</version>
   </dependency>
   <dependency>
       <groupId>com.aliyun.openservices</groupId>
       <artifactId>aliyun-log-log4j2-appender</artifactId>
       <version>0.1.12</version>
   </dependency>

2. Modify the configuration file.

   In the following example, an XML configuration file named log4j2.xml is used to configure appenders and loggers related to LogHub. If the configuration file does not exist, create the file in the root directory of the project. Sample code:

   <Appenders>
       <Loghub name="Loghub"
               project="your project"
               logStore="your logStore"
               endpoint="your project endpoint"
               accessKeyId="your accessKey id"
               accessKeySecret="your accessKey secret"
               totalSizeInBytes="104857600"
               maxBlockMs="0"
               ioThreadCount="8"
               batchSizeThresholdInBytes="524288"
               batchCountThreshold="4096"
               lingerMs="2000"
               retries="10"
               baseRetryBackoffMs="100"
               maxRetryBackoffMs="100"
               topic="your topic"
               source="your source"
               timeFormat="yyyy-MM-dd'T'HH:mmZ"
               timeZone="UTC"
               ignoreExceptions="true">
           <PatternLayout pattern="%d %-5level [%thread] %logger{0}: %msg"/>
       </Loghub>
   </Appenders>
   <Loggers>
       <Root level="warn">
           <AppenderRef ref="Loghub"/>
       </Root>
   </Loggers>

   In the preceding sample code, the project, logStore, endpoint, accessKeyId, and accessKeySecret parameters are required. Other parameters are optional, and you can use the default values. The following table describes the parameters.

   Parameter	Description
   project	The name of the Simple Log Service project.
   logStore	The name of the Simple Log Service Logstore.
   endpoint	The public endpoint of Simple Log Service. For information about how to obtain the endpoint, see Endpoints.
   accessKeyId	The AccessKey ID. For information about how to obtain an AccessKey ID, see Create an AccessKey pair.
   accessKeySecret	The AccessKey secret. For information about how to obtain an AccessKey secret, see Create an AccessKey pair.

3. Test the demo.

   import org.apache.logging.log4j.LogManager;
   import org.apache.logging.log4j.Logger;
   
   public class Log4j2AppenderExample {
   
       private static final Logger LOGGER = LogManager.getLogger(Log4j2AppenderExample.class);
   
       public static void main(String[] args) throws InterruptedException {
           LOGGER.trace("log4j2 trace log");
           LOGGER.debug("log4j2 debug log");
           LOGGER.info("log4j2 info log");
           LOGGER.warn("log4j2 warn log");
           LOGGER.error("log4j2 error log", new RuntimeException("Runtime Exception"));
           Thread.sleep(1000 * 5);
       }
   }

Use Logtail to collect Log4j logs

Important

When you configure Logtail, you must provide the file path. Make sure that application logs can be written to log files as expected.

1. Install Logtail

   For information about how to install Logtail, see Install Logtail on a Linux server.

2. Configure Logtail

   Simple Log Service provides configuration wizards to help you configure Logtail. For more information, see Collect text logs from servers.

Step 3: View logs

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you want to manage.

   

3. In the left-side navigation pane, click Log Storage. In the Logstores list, click the Logstore that you want to manage.

   

4. Check whether logs are uploaded to the Logstore. Query and analyze logs in the Logstore. Then, click Consumption Preview to view the logs.

   

5. Query and analyze logs

   Important

   After logs are uploaded to the Logstore, you must create indexes to query and analyze the logs. For more information, see Create indexes. For information about how to create an index, see Create indexes.

   Enter a query statement in the search box, click Last 15 Minutes, and then specify a query time range. The following sample code provides an example of the query results:

   Appender

   The following sample code provides an example of a log:

   level: ERROR
   location: com.aliyun.openservices.log.log4j.example.Log4jAppenderExample.main(Log4jAppenderExample.java:16)
   message: error log
   throwable: java.lang.RuntimeException: xxx
   thread: main
   time: 2018-01-02T03:15+0000
   log: 0 [main] ERROR com.aliyun.openservices.log.log4j.example.Log4jAppenderExample - error log
   __source__: xxx
   __topic__: yyy

   Query and analyze logs on the Logstore page.

   

   Logtail

   Query and analyze logs on the Logstore page.

   

   You can enter a query statement to view, search for, and filter log data. For more information, see Query and analyze logs. The following statements are provided as examples.

   • Query the three positions at which the most errors occur within the previous hour.

     level: ERROR | select location ,count(*) as count GROUP BY  location  ORDER BY count DESC LIMIT 3

   • Query the numbers of logs at each log level that are generated within the previous 15 minutes.

     | select level ,count(*) as count GROUP BY level ORDER BY count DESC

References

• For information about how to use Logtail to collect logs from servers, see Collect text logs from servers. Multiple plug-ins are available. For more information, see Parsing in regex mode.

• If the preview page is blank or the query page displays no data when you use Logtail to collect logs, you can troubleshoot the issue by following the instructions provided in the What do I do if errors occur when I use Logtail to collect logs? topic. If you use Logtail to collect logs, errors may occur. For example, regular expressions may fail to be parsed, invalid file paths may exist, and traffic may exceed the processing capabilities of shards. You can view Logtail collection errors by following the instructions provided in the How do I view Logtail collection errors? topic. For information about the common errors in data collection, see How do I troubleshoot the common errors that may occur when Simple Log Service collects logs?.

• For information about use cases and frequently asked questions (FAQ) about using Log4j2 appenders to collect Log4j logs, see Log4j2 Appender. For information about use cases and FAQ about using Log4j versions earlier than Log4j2 to collect Log4j logs, see Log4j Appender.

• When you use a Log4j2 appender to collect Log4j logs, you must configure an AccessKey pair. For information about how to obtain an AccessKey pair, see Create an AccessKey pair. For information about how to configure the endpoint provided by Simple Log Service, see Endpoints.