Simple Log Service:Import historical logs from log files

Procedure

1. Obtain the unique identifier of the Logtail configuration.

   You can obtain the unique identifier of a Logtail configuration from the user_log_config.json file in the installation directory of Logtail. For example, you can run the following command to obtain the unique identifier of a Logtail configuration on a Linux server:

   grep "##" /usr/local/ilogtail/user_log_config.json | awk '{print $1}'

2. Add a local event.

   1. Create the local_event.json file in the installation directory of Logtail.

   2. Add the local event in the JSON format to the local_event.json file.

      Important

      To prevent Logtail from loading an invalid JSON event, we recommend that you save the configuration of the local event in a temporary file for editing and then copy the configuration to the local_event.json file after editing.

      [ 
        {
          "config" : "${your_config_unique_id}",
          "dir" : "${your_log_dir}",
          "name" : "${your_log_file_name}"
         },
        {
         ...
         }
         ...
      ]

      Parameter	Description
      config	Enter the unique identifier of the Logtail configuration that is obtained in Step 1. Example: ##1.0##log-config-test$ecs-test.
      dir	Specify the directory of the log file from which you want to import historical logs. Example: /data/logs. Important Do not end the value with a forward slash (/). Do not specify the installation directory of Logtail, such as /usr/local/ilogtail.
      name	Specify the name of the log file from which you want to import historical logs. Wildcard characters are supported. Examples: access.log.2018-08-08 and access.log*.

      The following sample code provides an example on how to configure a local event in Linux:

      $ cat /usr/local/ilogtail/local_event.json
      [
        {
          "config": "##1.0##log-config-test$ecs-test",
          "dir": "/data/log",
          "name": "access.log*"
        },
        {
          "config": "##1.0##log-config-test$tmp-test",
          "dir": "/tmp",
          "name": "access.log.2017-08-09"
        }
      ]                            

FAQ

• How do I check whether Logtail loads a Logtail configuration?

  In most cases, after you save the local_event.json file, Logtail loads the content of the file to memory within 1 minute and then clears the file.

  You can use the following methods to check whether a Logtail configuration is loaded:

  1. If no content exists in the local_event.json file, Logtail has read the event information.

  2. Check whether the ilogtail.LOG file in the installation directory of Logtail contains the process local event parameter. If no content exists in the local_event.json file and the process local event parameter is not obtained, the content in the local_event.json file may be filtered out because the content is invalid.

• Why am I unable to collect data after a Logtail configuration is loaded?

  • The Logtail configuration is invalid.

  • The configuration in the local_event.json file is invalid.

  • The log file from which you want to collect logs is not stored in the directory that is specified in the Logtail configuration.

  • The logs of the log file from which you want to collect logs are collected.

What to do next

After historical logs are imported, you can query and analyze the logs. For more information, see Query and analyze logs in index mode.