This topic describes how to gracefully shut down jobs and tasks executed in an application.
Background information
In actual business scenarios, scheduled jobs are continuously executed in application processes at fixed intervals. When an application is restarted during release, the application process must be temporarily shut down. If you close the application, the ongoing scheduled jobs are interrupted, which may cause incomplete data or other issues. To prevent this situation, SchedulerX allows you to implement graceful shutdown for scheduled jobs. You must wait until the running jobs are complete before you safely close the application.
Limits
The version of the SchedulerX client must be 1.10.8 or later.
Graceful shutdown diagram
How to Configure
SchedulerX provides the graceful shutdown feature for jobs that are executed in standalone mode and distributed mode. Unlike real-time online jobs, offline scheduled jobs may require a long period of time to complete. However, application processes do not wait infinitely for the offline scheduled jobs to complete once you enable the graceful shutdown feature. You can run the following code to prolong the period of time before the application processes are shut down:
# The graceful shutdown mode. WAIT_ALL: Wait for all jobs to complete. WAIT_RUNNING: Wait for running jobs to complete.
# If you do not configure this parameter, the graceful shutdown feature is disabled. By default, the graceful shutdown feature is disabled.
spring.schedulerx2.graceShutdownMode=WAIT_ALL
# The timeout period for graceful shutdown. Unit: seconds. By default, no timeout period for graceful shutdown is specified.
spring.schedulerx2.graceShutdownTimeout=10
# Specifies whether to enable the port required by the HTTP service. Default value: false.
spring.schedulerx2.httpServerEnable=true
# The port over which you can call the shutdown-purposed HTTP interface. Default port: 51886.
spring.schedulerx2.httpServerPort=52333
Graceful shutdown mode | Description |
| In this mode, an application exits only after all jobs and tasks in the application complete. This mode is recommended. |
| In this mode, an application exits after running jobs and tasks to which threads are allocated in the application complete. Jobs and tasks in queue are dropped. |
Procedure
To facilitate the integration of release processes in various deployment modes into your application, you can use one of the following methods to enable the graceful shutdown feature:
Method 1: Gracefully shut down an application process by using kill -15
SchedulerX SDKs provide the hook function required by shutting down Java virtual machine (JVM) processes, which makes applications gracefully exit. If you want to shut down an application process, you can run the kill -15 Process ID
command. In this case, the running process is gracefully shut down after jobs and tasks are complete based on the configured policies.
When you execute a script to shut down an application, we recommend that you do not directly run the kill -9
command. Otherwise, graceful shutdown does not work. For some applications, we recommend that you first run the kill -15
command, monitor the application for a period of time, and then run the kill -9
command to forcibly shut down the application. We recommend that you specify a period of wait time for graceful shutdown based on your job attributes to prevent an extremely slow release process.
Method 2: Gracefully shut down an application process by using the SpringBoot shutdown event
If you connect an initialized SpringBoot application to SchedulerX, you can use the actuator feature provided by SpringBoot to implement graceful shutdown. The actuator feature can gracefully shut down scheduled jobs while responding to Spring container shutdown events.
To enable the SpringBoot actuator feature, perform the following steps:
Run the following code to add dependencies to the
pom.xml
file of the desired application:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Run the following code to enable the shutdown port:
management.endpoint.shutdown.enabled=true
management.endpoints.web.exposure.include=shutdown
Run the following code to implement graceful shutdown:
curl -X GET http://NodeIP:Port/actuator/shutdown
Method 3: Gracefully shut down an application process by using the HTTP interface
Configure the following parameters to enable the port required by the HTTP service to respond to the external requests for shutting down the scheduled jobs in the current application process. Parameters:
# Specifies whether to enable the port required by the HTTP service. Default value: false.
spring.schedulerx2.httpServerEnable=true
# The port over which you can call the shutdown-purposed HTTP interface. Default port: 51886.
spring.schedulerx2.httpServerPort=51886
You can call the following HTTP interface to gracefully shut down scheduled jobs:
curl -X GET http://NodeIP:51886/schedulerx/worker/shutdown
Integration solutions
In most cases, the graceful shutdown feature needs to be integrated into the O&M processes released on a daily basis to prevent scheduled jobs from being damaged upon application restart. You can use one of the following regular integration solutions based on your business requirements.
Scenario 1: Integrate graceful shutdown into a self-built CD process
In most cases, a self-built continuous delivery (CD) process includes a node where you can stop the application process. You can build and execute the stop.sth script on the node to make your application process exit. You must include the processing logic of graceful shutdown in the script content. You can refer to one of the preceding methods based on your business requirements.
Sample CD process:
The following code provides a sample script that is used to shut down an application process:
# After you start the application, the process ID is written into the app.pid file.
PID="{Application deployment path}/app.pid"
FORCE=1
if [ -f ${PID} ]; then
TARGET_PID=`cat ${PID}`
kill -15 ${TARGET_PID}
loop=1
while(( $loop<=5 ))
do
# health: Specifies whether the current application process ends. You can configure the health field based on your business requirements.
health
if [ $? == 0 ]; then
echo "check $loop times, current app has not stop yet."
sleep 5s
let "loop++"
else
FORCE=0
break
fi
done
if [ $FORCE -eq 1 ]; then
echo "App(pid:${TARGET_PID}) stop timeout, forced termination."
kill -9 ${TARGET_PID}
if
rm -rf ${PID}
echo "App(pid:${TARGET_PID}) stopped successful."
fi
Scenario 2: Integrate graceful shutdown into the PreStop event of the Kubernetes containerized deployment
You can use the PreStop event that is used to destroy pods in Kubernetes clusters to trigger graceful shutdown for your application. In addition, you can use exec to execute scripts or initiate HTTP requests to invoke the PreStop hook function to implement graceful shutdown.
Exec script: Directly shut down the application process. Alternatively, you can execute the preset
stop.sh
script to make the application process exit.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app-container
image: my-app-image:latest
lifecycle:
preStop:
exec:
# command: ["/bin/sh", "-c", "kill -15 PID && sleep 30"]
command: ["/bin/sh", "-c", "Script path /stop.sh"]
HTTP interface:
When you enable the SpringBoot actuator shutdown feature, you can set Path to
/actuator/shutdown
.For non-Spring applications, you can set Path to
/schedulerx/worker/shutdown
after you enable the port over which you can call the HTTP interface by using SchedulerX SDKs to implement graceful shutdown.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app-container
image: my-app-image:latest
lifecycle:
preStop:
httpGet:
path: /schedulerx/worker/shutdown
port: 51886
scheme: HTTP
Scenario 3: Integrate graceful shutdown into an application release platform in Alibaba Cloud
If you release your application by using Alibaba Cloud Enterprise Distributed Application Service (EDAS) or by using Microservices Engine (MSE) and Container Service for Kubernetes (ACK), the graceful shutdown feature of SchedulerX is automatically integrated into the platform that you use after you enable the graceful shutdown feature on the platform. For more information, see Gracefully shut down Spring Cloud applications and Graceful shutdown.