How to Use Journalctl to Consult Server Logs

By Alexandru Andrei, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Linux instances and other *nix (Unix-like) based operating systems in general (e.g. the BSD releases) come with great logging utilities preinstalled. These catch all of the important boot, kernel, system and program messages, errors, warnings, notices, and everything that might be useful to help monitor and debug the software running on the platform. Traditionally, this has been handled by syslog, syslog-ng and rsyslog daemons. Each version brought incremental improvements, rsyslog being the log daemon that has been used by most of the Linux based operating systems in the latest years.

Since most of the popular Linux distributions have migrated to systemd, there is an ongoing shift to another logging daemon, systemd-journald. While rsyslog separates log messages to different files such as /var/log/auth.log, /var/log/syslog and so on, journald centralizes everything in one place. It also uses the binary format instead of the text format to store data. This can offer some advantages. For example, it makes it easier to see how different events from multiple system areas relate, since messages are sorted by time and displayed in a single log. This can make it easier to connect the dots and follow the chain of errors to see where a particular problem started. Filtering and sorting messages, changing output type and various tasks are also easier to do because of the way systemd-journald stores data. Another helpful feature is that the utility used to display logs, automatically formats text, highlighting important pieces of information in bold letters or different colors; e.g. errors are displayed in red, making it easier to notice them as you're scrolling through text.

Note: The steps in this tutorial have been tested on Debian and Ubuntu. Although highly unlikely, other distributions may require adapting the commands presented here. This can happen if the distribution uses an older version of systemd or defines different paths for configurations files. Some commands require administrator privileges, so you should be logged in as the root user or prefix the command with sudo if required.

Enable Persistent Logs

On most Linux distributions, rsyslog is running in parallel with systemd's logging daemon. rsyslog log files are kept in the /var/log directory. For this reason, you will usually find that systemd-journald will only keep volatile log messages for the current boot (logs are kept in memory instead of disk). To enable persistent logging, so that you can use journalctl to consult logs from previous boots, check the following:

    grep Storage /etc/systemd/journald.conf

If the output is Storage=auto or #Storage=auto, it means that if the directory /var/log/journal exists, then journald will keep persistent logs at that location. If it doesn't exist, it will keep logs only in volatile memory, which will be lost when rebooting or shutting down the instance.

Check if the directory exists:

    ls /var/log/journal

If it exists you can skip creating it and restarting the journal daemon. If it doesn't, create it, to keep logs on disk:

    mkdir -p /var/log/journal

Then restart the daemon to apply the changes:

    systemctl restart systemd-journald

If Storage is set to another value such as volatile or none, you can edit the configuration file and set it to persistent:

    sed -i '/Storage/ c\Storage=persistent' /etc/systemd/journald.conf

The command above searches for any line containing the string "Storage" and replaces it with "Storage=persistent".

Restart the journal to apply the changes:

    systemctl restart systemd-journald

To learn more about the values you can set in journald.conf:

    man journald.conf

Press "q" to quit the manual.

You can edit the configuration file with:

    nano /etc/systemd/journald.conf

Check the variables that set limits on how many log archives can be kept on the system. Here's an example from the Debian 9 operating system:

#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100

This limits the amount of log files that can be kept to 100. You can also limit by disk space if you want to. To edit a variable, delete the preceding # sign. "#" signals that the line is commented and not active, but these commented lines show you the default values that are used even if not set in this file. To change the maximum number of journal files to be kept around, you would edit the file like this:

#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
SystemMaxFiles=200

You would then press CTRL+X, then "y", then ENTER, to save the file and you would apply the changes with:

    systemctl restart systemd-journald

Set Timezone

Log messages are prefixed by date and time of event. But for most people, the timezone used on an Alibaba Cloud ECS instance won't coincide with the timezone the user is currently located in. Check the timezone set on the instance with:

    timedatectl status

To see the available timezones you can apply:

    timedatectl list-timezones

You can press PAGE UP and PAGE DOWN to navigate this list. Press "q" to quit.

The command for setting the timezone is timedatectl set-timezone Contintent/City. The name of the continent and name of city have to start with capital letters. Example command: timedatectl set-timezone Europe/Amsterdam. Now the journalctl utility will show the timestamps of events in your local time.

Filtering Journal Output

The command you use to interact with journal files is journalctl. To be able to read the system journals, you either have to be logged in as the root user or prefix the command with sudo, if you're logged in as a regular user that has sudo privileges. Another way to view system log files, from a regular user account, is to add that user to the systemd-journal group. You can do this with a command such as adduser name_of_user systemd-journal. Then log out if you're already logged in as that user and log back in. Now you can view journal files without having to prefix the command with sudo and enter your password.

If you run:

    journalctl

The available journal files will be concatenated and displayed to the user. The less utility is used to display the text, which means you can use some of its keyboard shortcuts. For example, to go to the end of the text displayed press >. To go back to the beginning you can press <.

You can search for text from the position that is currently displayed, and continuing down, by pressing /, entering the text you want to search for, e.g. "ssh" and then pressing ENTER. Matched text will be highlighted in white. Pressing n will search for the next occurrence and N will search for the previous. To search upwards instead of downwards, you can press ? and then enter your text. Press q to quit the utility. Consult man less for more information.

Filter by Boot Number

Every time you boot your system, a new boot entry is created in the journal. You can see all of the available ones with:

    journalctl --list-boots

Example output:

root@instance1:~# journalctl --list-boots
-4 b995cd57ae72431ba9be9560f28f80f7 Wed 2018-08-08 22:59:57 EEST—Wed 2018-08-08 23:00:04 EEST
-3 9b069b7c07a74e1a9d07d7ba830c22a8 Wed 2018-08-08 23:00:07 EEST—Wed 2018-08-08 23:32:35 EEST
-2 99439be7b99a4f67ad7b36768a508adc Wed 2018-08-08 23:32:39 EEST—Wed 2018-08-08 23:33:31 EEST
-1 0a261487baad44a394f86475ee5392f3 Wed 2018-08-08 23:33:35 EEST—Wed 2018-08-08 23:33:52 EEST
 0 6ee25c9b1cd94c7aaed4db674c14d431 Wed 2018-08-08 23:33:57 EEST—Wed 2018-08-08 23:49:17 EEST

They're all prefixed by a number, starting from 0, which means current boot. "-1" signifies, one boot ago, "-2" two boots ago and so on. To view messages from one particular boot you can use journalctl -b 6ee25c9b1cd94c7aaed4db674c14d431. But this requires copy-pasting that series of characters. You can also pick a boot by using its index number:

    journalctl -b -1

Press "q" to quit.

Filter by Time

To filter by time of events, you can use the -S and -U command line switches. -S can be replaced with --since and -U with --until.

You can use year-month-day hour:minute:second as arguments. Or you can use just the date part or just the time part. You can also type today instead of 00:00. tomorrow and yesterday are also understood. And probably the most useful type of time arguments are relative values such as -1h which refers to 1 hour ago.

To see messages since the end of January 2018:

    journalctl -S 2018-01-31

To see messages generated since midnight:

    journalctl -S 00:00

Equivalent command:

    journalctl -S today

To see messages from the last hour:

    journalctl -S -1h

Such commands cut content from the beginning. But you may want to cut from the end. If you don't want to see messages from the last hour:

    journalctl -U -1h

These filters can be combined. If you want to see only the content generated yesterday, removing everything else:

    journalctl -S yesterday -U 00:00

Filter by Priority

Filtering by message priority can often be useful, especially in cases where you need to debug something. When you're looking for things that are malfunctioning, normal status messages are of no use. To only list errors:

    journalctl -p err

To list warnings:

    journalctl -p warning

Other types of priority levels are: alert, crit, debug, emerg, err, info, notice, warning. These are explained in the syslog manual which can be consulted with man 3 syslog.

You can combine different types of filters together. To get errors generated in the last hour:

    journalctl -p err -S -1h

Filter by Unit, File or Device Path

Sometimes you will only want to see logs associated with a particular service on your system. For example, maybe you want to see everything related to the SSH daemon:

    journalctl -u ssh.service

To see what services generate log messages, you can type journalctl -u at the terminal, add a space after -u and then press TAB two times.

To filter messages based on the running process' file path:

    journalctl /usr/sbin/sshd

If you want to see log messages related to a device, for example the first virtual disk on your system:

    journalctl /dev/vda

To show only logs generated by the Linux kernel:

    journalctl -k

Control journalctl Output

By default, the journal is displayed starting with the oldest entry available. But most of the times, you will want to see the recent entries first. To do this, you can use the -e switch in a command such as journalctl -b 0 -e to go to the end of the journal from the last boot. This will also limit the output to 1000 lines. To display more lines you can use the -n switch. To see the output from the last boot, go to the end of it and limit the number of lines to 10000:

    journalctl -b 0 -e -n 10000

At times, you may find yourself trying to reproduce errors or test functionality. Instead of looking at what happened in the past, in such cases you'll want to see log output live and then interact with a program so that it generates events you can see in real time. The journalctl utility has a "follow" option:

    journalctl -f

To exit this mode press CTRL+C.

journalctl output can also be reversed so that messages are displayed starting with the newest entry and ending with the oldest:

    journalctl -r

By default, journalctl displays text that can be easily read by humans. But at times, you may want text that can be easily read and processed by other programs. For example, you might want to integrate the output in a web application so you might find JavaScript Object Notation (JSON) useful:

    journalctl -o json

To make it "pretty", meaning also easily readable by humans:

    journalctl -o json-pretty

To see all the details the logging daemon actually has about each event:

    journalctl -o verbose

Log messages can also be enriched by the so-called Message Catalog. For specific events found in the catalog, this will add explanations, possible solutions, links, or any kind of additional information that might be useful to the user.

    journalctl -x

Manage Disk Space Used by the Journal

To see how much storage space the journal files are using:

    journalctl --disk-usage

To free up space, you have two options, either to delete files based on time, or delete files based on how much space they are using. The following commands will only delete archived journal files. Actively used journal files won't be touched. To delete anything older than a certain time:

    journalctl --vacuum-time=3months

You can also use "days", "weeks" and "years".

To delete based on how much disk space is used:

    journalctl --vacuum-size=2G

This will keep at most 2 gigabytes of archived journal files and delete the rest. M can be used for megabytes.

To learn more about journalctl, consult the manual with man journalctl.

Community

How to Use Journalctl to Consult Server Logs

Enable Persistent Logs

Set Timezone

Filtering Journal Output

Filter by Boot Number

Filter by Time

Filter by Priority

Filter by Unit, File or Device Path

Control journalctl Output

Manage Disk Space Used by the Journal

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

ECS(Elastic Compute Service)

OSS(Object Storage Service)

Simple Log Service