Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog An Introduction to Logrotate: Manage Log Files on Alibaba Cloud

An Introduction to Logrotate: Manage Log Files on Alibaba Cloud

This article looks at Logrotate and goes through how you can configure log rotation for a custom application to manage log files on Alibaba Cloud.

By Alex, Alibaba Cloud Community Blog author.

If you are looking for an efficient system utility to rotate and compress your log files, then your search ends here with Logrotate. The importance of efficient management of log files is the proper utilization of disk space, hence it necessitates to rotate, compress and prune log files. Alibaba Cloud ECS instance running Ubuntu 18.04 has this utility as one of the default installed packages. By default, it takes care of log rotation for all the installed applications including the system log processor. This article deep dives into Logrotate configurations and also demonstrates how to create some custom configurations for log rotations.

Prerequisites

For this tutorial, you'll need an Alibaba Cloud ECS instance with Ubuntu 18.04 up and running and a sudo non-root user with limited access to the server instance. Although other Linux distributions may have some pre-installed version of Logrotate, the configurations may not suit the requirements of this tutorial. To begin with, use the sudo non-root user to log into the ECS instance and follow the steps described in the following sections

Verify Logrotate Setup and Status

Verify whether the Logrotate is up and running using the following command.

sudo logrotate

The following snippet shows the output for the above command.

logrotate 3.11.0 - Copyright (C) 1995-2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License

Usage: logrotate [-dfv?] [-d|--debug] [-f|--force] [-m|--mail=command]
        [-s|--state=statefile] [-v|--verbose] [-l|--log=STRING] [--version]
        [-?|--help] [--usage] [OPTION...] <configfile>

The absence of any such output implies that the Logrotate is not included in the Ubuntu 18.04 installation, by default. So, you'll want to install Logrotate using the following command.

 sudo apt-get install logrotate -y

Next, run the command below to check the installed version.

logrotate --version

Output
logrotate 3.11.0

To overcome the configuration related challenges owing to different Logrotate versions, check out the documentation for the particular version on Git repository, or run the command below.

man logrotate

Now, let's proceed to discover some configurations for Logrotate.

Logrotate Configurations

The /etc/logrotate.conf file contains the default configuration settings for Logrotate. The file contains a number of global variables that dictate how log rotation occurs. Further settings are located in the /etc/logrotate.d/ directory, which holds files for all daemon processes.

Run the command below to check some global commands in the /etc/logrotate.conf file.

sudo nano /etc/logrotate.conf

The following snippet list the output for the preceding command.

weekly
su root syslog
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

The /etc/logrotate.d/ directory lists configurations referenced in the /etc/logrotate.conf file by the include /etc/logrotate.d statement as shown above.

sudo nano /etc/logrotate.d/dpkg
/var/log/dpkg.log {
        monthly
        rotate 12
        compress
        delaycompress
        missingok
        notifempty
        create 644 root root
}

There are directives to manage the log file's size and rotation frequency. For instance, the /etc/logrotate.d/dpkg file dictates a monthly rotation and saves 12 of the last rotated logs. Log files are recreated with permission 644 and root user in the root group.

However, it is still possible to have even more specific configurations to override the set global configurations.

Email Rotated Logs

A mail directive allows mailing logs after certain rotations as well as support their subsequent removal. First, logs rotate as per the definition in the rotate directive, and once the specified rotation number exceeds, the log is removed. While handling a small number of processes, logs do not consume a lot of disk space. However, when a large number of processes generate logs, they take up disk space really quickly.

mail mail@user.com

For the email function, set up an emailing client as shown below.

echo test | mail -s test user@example.com

Logrotate uses the /usr/bin/mail as a default means to send mail, therefore it is critical to ensure that it works on your system.

Define Rotation Requirements

A log rotation directive helps to choose the period between rotations as defined in the /etc/logrotate.conf file.

Choose either to rotate logs weekly, monthly, or yearly. Furthermore, use the size directive to induce rotation of logs once they take some space on the disk.

size [value]

Since the assumption is that the disk size is usually in bytes, it is better to state the specific size of the file in (k) for kilobytes, (M) for megabytes, or (G) for gigabytes.

Log Compression

Using the compress directive, Logrotate helps to further compress rotated logs. Compressing all the generated logs depends on whether the directive is placed in the main configuration file. A nocompress directive disables compression in a specific log. While the default is the gzip command, the compresscmd directive allows specifying different commands such as bzip2 or xz.

Use delaycompress directive to postpone compression in case there is no need to compress logs immediately after rotation. The default is a one-cycle delay.

Log File Extension

The extension directive allows Logrotate to maintain a common extension to all rotated files. For instance, the facebook.log would be rotated as facebook.log, logrotate.conf.

extension log

The anticipated output could be facebook.log.gz. Let's proceed to create a sample configuration.

Sample Logrotate Configuration

Custom Configurations Snapshot

This section describes some custom configurations and their applications.

  • The following custom configuration helps to rotate the log files if the size exceeds 100 MB.
custom.conf
/var/log/remote/IP-ADDRESS/auth.log {
        maxsize 100M
        daily
        missingok
        notifempty
        rotate 7
        compress
}
  • The following custom configuration removes rotated files that are older for more than 60 days.
 custom.conf
/var/log/remote/IP-ADDRESS/auth.log {
        size 1G
        maxage 60
        missingok
        notifempty
        rotate 4
        compress
}
  • The following custom configuration specifies the compression command as xz.
custom.conf
/var/log/remote/IP-ADDRESS/auth.log {
        size 1G
        maxage 60
        missingok
        notifempty
        rotate 8
        compress
        compresscmd /bin/xz
        compressext .xz
}
  • The following custom configuration includes instructions to run the postrotate script after compressing old logs.
custom.conf
/var/log/remote/IP-ADDRESS/auth.log {
        rotate 4
        monthly
        sharedscripts
        postrotate
        /usr/bin/killall -HUP rsyslogd
        endscript
}

Create Custom Configurations for a User

Assume that there's a user named flasky, which has a running application that stores its logs in /home/flasky/logs/. The intention is to rotate those logs every six hours. Since this is a non-root user, log rotation configurations are outside the default /etc/logrotate.d provided on the ECS instance.

To begin with, run the following command to create a new configuration file in the home directory.

nano /home/flasky/logrotate.conf

Add the following code lines in the new file that opens in the Nano editor.

/home/flasky/logrotate.conf
/home/flasky/logs/*.log {
    daily
    missingok
    rotate 4
    compress
    create
    size 1G
    maxage 60
    notifempty

}

Save the changes and exit the editor. The new configurations rotate files every six hours and maintain four old logs at all times. The period could be hourly or even daily depending on the needs of the application. Now, follow the next steps to verify whether it works.

cd ~

mkdir logs

touch logs/access.log

The command creates a new empty log file. To run the command, it is mandatory to declare a state file in command which contains all the information about previous Logrotate runs as well as informs about the next steps. As shown earlier, the process is completely automated. Now run the following command.

logrotate /home/flasky/logrotate.conf --state /home/flasky/logrotate-state --verbose

The following snippet shows the output for the preceding command.

reading config file /home/flasky/logrotate.conf
Reading state from file: /home/flasky/logrotate-state
Allocating hash table for state file, size 64 entries

Handling 1 logs

rotating pattern: /home/flasky/logs/*.log  after 1 days (4 rotations)
empty log files are rotated, old logs are removed
considering log /home/flasky/logs/access.log
Creating new state
  Now: 2019-04-05 10:04
  Last rotated at 2019-04-05 10:00
  log does not need rotating (log has been already rotated)

The --verbose command is used to print the current status of Logrotate. The information output indicates that the specified file is less than 6 hours old and thus, requires no rotation. Now run the following command to inspect the state file and see all recorded run information.

cat /home/flasky/logrotate-state

The following snippet shows the output for the preceding command.

logrotate state -- version 2
"/home/flasky/logs/access.log" 2019-4-5-10:0:0

The output indicates that Logrotate ran and considered the file for rotation. For a forced rotation, employ the --force flag as shown in the example below.

logrotate /home/flasky/logrotate.conf --state /home/flasky/logrotate-state --verbose ¨Cforce

The following snippet shows the output for the preceding example.

creating new /home/flasky/logs/access.log mode = 0664 uid = 1000 gid = 1000
compressing log with: /bin/gzip

Now, let's set up a cron job that even runs six hours for the Logrotate. Execute the following command.

crontab -e

A text file may open after the command. Include the following syntax at the end of the file to run the Logrotate job.

crontab
30 */6 * * */usr/sbin/logrotate /home/flasky/logrotate.conf --state /home/flasky/logrotate-state

With this, the task will run every 30th minute of the 6th hour every day. Save the file and exit the editor to initialize the crontab that runs the job. Now, check new rotated and compressed log files in the log directory after the lapse of at least six hours.

Conclusion

This tutorial explores the basic functions of Logrotate and comprehensively explains how to install Logrotate, check the installed version and illustrate the configurations for a sample application. Alibaba Cloud Ubuntu 16.04 ECS offers a full set of features, and Logrotate is one of the major features. For more detailed information about the complex Logrotate setup, quickly access the documentation through the man logrotate command.

Don't have an Alibaba Cloud account? Sign up for an account and try over 40 products for free worth up to $1200. Get Started with Alibaba Cloud to learn more.

OSS ECS MaxCompute Web Hosting Tech Share Container Service Tutorial Logrotate Log Files Logging
0 0 0
Share on

Read previous post:

How to Configure TeamViewer on Alibaba Cloud

Read next post:

An Introduction to jQuery

Alex

53 posts | 10 followers

You may also like

Comments

Alex

53 posts | 10 followers

Related Products

More Posts by Alex

See All