How to use Logtail to collect Python logs - Simple Log Service

This topic describes how to use Logtail to collect Python logs.

Background information

The Python logging module provides a general logging system, which can be used by third-party modules or applications. The logging module defines multiple log levels and logging methods. The logging module consists of the following components: loggers, handlers, filters, and formatters.

Formatters specify the output format of logs. The fields that are configured for a formatter are in the %(key)s format. Sample code:

import logging  
import logging.handlers  
LOG_FILE = 'tst.log'  
handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes = 1024*1024, backupCount = 5) # Create a handler.   
fmt = '%(asctime)s - %(filename)s:%(lineno)s - %(levelno)s %(levelname)s %(pathname)s %(module)s %(funcName)s %(created)f %(thread)d %(threadName)s %(process)d %(name)s - %(message)s' // Define the output format of logs. 
formatter = logging.Formatter(fmt)   # Create a formatter.  
handler.setFormatter(formatter)      # Add the formatter to the handler. 
logger = logging.getLogger('tst')    # Retrieve a logger that is named tst.   
logger.addHandler(handler)           # Add the handler to the logger.  
logger.setLevel(logging.DEBUG)  
logger.info('first info message')  
logger.debug('first debug message')

The following table describes the fields in the sample code.

Field	Description
%(name)s	The name of the logger that is used to log the call.
%(levelno)s	The log level in the numeric format.
%(levelname)s	The log level in the text format. Valid values: DEBUG, INFO, WARNING, ERROR, and CRITICAL.
%(pathname)s	The log file path.
%(filename)s	The log file name.
%(module)s	The name of the source module for log collection.
%(funcName)s	The name of the function that contains the logging call.
%(lineno)d	The source line number of the statement that includes the function.
%(created)f	The time at which a log is generated. The time is a UNIX timestamp.
%(relativeCreated)d	The difference between the time at which a log is generated and the time at which the logging module is loaded. Unit: milliseconds.
%(asctime)s	The time at which a log is generated. Example: 2003-07-08 16:49:45,896, where 896 indicates the millisecond portion of the time.
%(msecs)d	The millisecond portion of the time at which a log is generated.
%(thread)d	The thread ID.
%(threadName)s	The thread name.
%(process)d	The process ID.
%(message)s	The log information.

Procedure

Simple Log Service provides configuration wizards to help you create Logtail configurations. The following example describes how to collect Python logs in full regex mode.

Log on to the Simple Log Service console.
In the Import Data section, select RegEx - Text Log.
Select the project and Logstore. Then, click Next.
Create a machine group.
- If a machine group is available, add the machine group to Applied Server Groups.
- If no machine group is available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
  1. Select Servers for Scenario.
  2. Select ECS for Installation Environment.
  3. Click Create Machine Group next to Select Machine Group.
  4. In the Create Machine Group panel, select the ECS instance and click Install and Create Machine Group.
    For more information, see Install Logtail on ECS instances.
  5. Check whether Logtail is installed on the ECS instance. If Success is displayed in the Logtail Installation Status column, Logtail is installed.
  6. In the Create Machine Group panel, enter a name for the machine group and retain the default settings of other parameters. Then, click OK.
    For more information, see Create an IP address-based machine group.
Confirm that the machine group is displayed in the Applied Server Groups section and click Next.
Important
If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
In the Logtail Configuration step, create a Logtail configuration.
For more information about Logtail-related parameters, see Logtail configuration.
Preview data, configure indexes, and then click Next. For more information about indexes, see Create indexes.
Click Log Query. You are redirected to the query and analysis page of your Logstore.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.