Realtime Compute for Apache Flink lets you view the logs of historical job instances on the job log interface, in Object Storage Service (OSS), or in Simple Log Service (SLS). This topic describes how to view the logs of historical job instances using these methods.
Background information
You may need to view the logs of a historical job instance in the following scenarios:
-
You want to view, search for, and analyze the operational logs of a historical job instance. By default, the system retains the latest 5 MB of operational logs.
-
The Flink UI cannot be opened, and you need to view the Job Manager logs to identify the cause of the issue.
This topic describes three ways to view the logs of a historical job instance:
Prerequisites
-
To view the logs of historical job instances on the Job Log tab, the log archiving feature must be enabled in the log configuration. By default, this feature is enabled and archived logs are retained for seven days.
-
To view the logs of historical job instances in OSS or SLS, you must configure log output to OSS or SLS in Realtime Compute for Apache Flink. For more information, see Configure job log output.
Notes
-
When a log file reaches the specified size, the system automatically archives it. The system also periodically deletes expired archived log files. Archived logs are retained if a job is stopped, but are deleted if the job is deleted.
NoteAfter the new version is released in November 2024, all existing job logs in the Realtime Compute for Apache Flink management console will be deleted after their retention period expires. This includes archived logs that were retained after a job was stopped.
-
Logs are written to SLS or OSS only if a Task Manager or the Job Manager starts successfully. If neither of them starts properly, logs are not written to SLS or OSS.
-
If you cannot find the corresponding log file in OSS because the Job Manager and Task Managers failed to start, contact technical support.
-
By default, the log configuration retains two files, each 5 MB in size. Do not change this value. If this log retention size does not meet your needs, you can export the logs to OSS.
NoteThe number of files shown in the image does not include the file that is being written to. For example, if you set this value to 1, a maximum of two files are saved: one in memory and one on your local disk.
View logs on the job log interface
-
Log on to the Realtime Compute for Apache Flink console.
-
In the Actions column of the target workspace, click Console.
-
In the navigation pane on the left, choose .
-
Click the name of the target job.
-
Click the Job Log tab.
-
On the Operational Log tab, click the drop-down list next to Job and select a historical job instance.
-
View the Job Manager and Task Manager logs.
On the Job Manager or Running Task Managers tab, click the name of the log file that you want to view.
Viewing on OSS
-
Log on to the OSS console.
-
In the navigation pane on the left, click Buckets.
-
Click the Bucket Name that you specified when you created your Flink workspace.
-
In the Files section, view the logs in the corresponding folders.
Folder
Description
artifacts
Uploaded JAR packages are stored in this directory.
flink-jobs
Stores the checkpoints of Flink jobs.
flink-savepoints
On the page of the development console of Realtime Compute, when you click Create Snapshot for a job, a Savepoint operation is triggered and the final Savepoint file is stored in this directory.
logs
If you set Log Template to OSS for your job, the job logs are stored in this folder.
NoteThe storage path for jobs in the logs folder is logs/${Workspace ID}/${Job ID}/${Instance ID}. You can use this path structure to find the Job Manager (JM) and Task Manager (TM) logs for the corresponding job.
Viewing in SLS
-
Log on to the Simple Log Service console.
-
In the Projects section, click the target project.
-
On the Logstores tab, click the target Logstore.
-
Enter a query statement, select a time range, and click Query/Analyze.
For more information, see Quick start for query and analysis.