All Products
Search
Document Center

SchedulerX:Gracefully shut down jobs and tasks executed in an application

Last Updated:Feb 18, 2024

This topic describes how to gracefully shut down jobs and tasks executed in an application.

Background information

In actual business scenarios, scheduled jobs are continuously executed in application processes at fixed intervals. When an application is restarted during release, the application process must be temporarily shut down. If you close the application, the ongoing scheduled jobs are interrupted, which may cause incomplete data or other issues. To prevent this situation, SchedulerX allows you to implement graceful shutdown for scheduled jobs. You must wait until the running jobs are complete before you safely close the application.

Limits

The version of the SchedulerX client must be 1.10.8 or later.

Graceful shutdown diagram

image

How to Configure

SchedulerX provides the graceful shutdown feature for jobs that are executed in standalone mode and distributed mode. Unlike real-time online jobs, offline scheduled jobs may require a long period of time to complete. However, application processes do not wait infinitely for the offline scheduled jobs to complete once you enable the graceful shutdown feature. You can run the following code to prolong the period of time before the application processes are shut down:

# The graceful shutdown mode. WAIT_ALL: Wait for all jobs to complete. WAIT_RUNNING: Wait for running jobs to complete. 
# If you do not configure this parameter, the graceful shutdown feature is disabled. By default, the graceful shutdown feature is disabled. 
spring.schedulerx2.graceShutdownMode=WAIT_ALL
# The timeout period for graceful shutdown. Unit: seconds. By default, no timeout period for graceful shutdown is specified. 
spring.schedulerx2.graceShutdownTimeout=10

# Specifies whether to enable the port required by the HTTP service. Default value: false. 
spring.schedulerx2.httpServerEnable=true
# The port over which you can call the shutdown-purposed HTTP interface. Default port: 51886. 
spring.schedulerx2.httpServerPort=52333

Graceful shutdown mode

Description

WAIT_ALL

In this mode, an application exits only after all jobs and tasks in the application complete. This mode is recommended.

WAIT_RUNNING

In this mode, an application exits after running jobs and tasks to which threads are allocated in the application complete. Jobs and tasks in queue are dropped.

Procedure

To facilitate the integration of release processes in various deployment modes into your application, you can use one of the following methods to enable the graceful shutdown feature:

Method 1: Gracefully shut down an application process by using kill -15

SchedulerX SDKs provide the hook function required by shutting down Java virtual machine (JVM) processes, which makes applications gracefully exit. If you want to shut down an application process, you can run the kill -15 Process ID command. In this case, the running process is gracefully shut down after jobs and tasks are complete based on the configured policies.

Important

When you execute a script to shut down an application, we recommend that you do not directly run the kill -9 command. Otherwise, graceful shutdown does not work. For some applications, we recommend that you first run the kill -15 command, monitor the application for a period of time, and then run the kill -9 command to forcibly shut down the application. We recommend that you specify a period of wait time for graceful shutdown based on your job attributes to prevent an extremely slow release process.

Method 2: Gracefully shut down an application process by using the SpringBoot shutdown event

If you connect an initialized SpringBoot application to SchedulerX, you can use the actuator feature provided by SpringBoot to implement graceful shutdown. The actuator feature can gracefully shut down scheduled jobs while responding to Spring container shutdown events.

To enable the SpringBoot actuator feature, perform the following steps:

  1. Run the following code to add dependencies to the pom.xml file of the desired application:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
  1. Run the following code to enable the shutdown port:

management.endpoint.shutdown.enabled=true
management.endpoints.web.exposure.include=shutdown
  1. Run the following code to implement graceful shutdown:

curl -X GET http://NodeIP:Port/actuator/shutdown

Method 3: Gracefully shut down an application process by using the HTTP interface

Configure the following parameters to enable the port required by the HTTP service to respond to the external requests for shutting down the scheduled jobs in the current application process. Parameters:

# Specifies whether to enable the port required by the HTTP service. Default value: false. 
spring.schedulerx2.httpServerEnable=true
# The port over which you can call the shutdown-purposed HTTP interface. Default port: 51886. 
spring.schedulerx2.httpServerPort=51886

You can call the following HTTP interface to gracefully shut down scheduled jobs:

curl -X GET http://NodeIP:51886/schedulerx/worker/shutdown

Integration solutions

In most cases, the graceful shutdown feature needs to be integrated into the O&M processes released on a daily basis to prevent scheduled jobs from being damaged upon application restart. You can use one of the following regular integration solutions based on your business requirements.

Scenario 1: Integrate graceful shutdown into a self-built CD process

In most cases, a self-built continuous delivery (CD) process includes a node where you can stop the application process. You can build and execute the stop.sth script on the node to make your application process exit. You must include the processing logic of graceful shutdown in the script content. You can refer to one of the preceding methods based on your business requirements.

Sample CD process:

image

The following code provides a sample script that is used to shut down an application process:

# After you start the application, the process ID is written into the app.pid file.
PID="{Application deployment path}/app.pid"
FORCE=1
if [ -f ${PID} ]; then
  TARGET_PID=`cat ${PID}`
  kill -15 ${TARGET_PID}
  loop=1
  while(( $loop<=5 ))
  do
    # health: Specifies whether the current application process ends. You can configure the health field based on your business requirements.
    health
    if [ $? == 0 ]; then
      echo "check $loop times, current app has not stop yet."
      sleep 5s
      let "loop++"
    else
      FORCE=0
      break
    fi
  done
  if [ $FORCE -eq 1 ]; then
  	echo "App(pid:${TARGET_PID}) stop timeout, forced termination."
    kill -9 ${TARGET_PID}
  if
  rm -rf ${PID}
  echo "App(pid:${TARGET_PID}) stopped successful."
fi

Scenario 2: Integrate graceful shutdown into the PreStop event of the Kubernetes containerized deployment

You can use the PreStop event that is used to destroy pods in Kubernetes clusters to trigger graceful shutdown for your application. In addition, you can use exec to execute scripts or initiate HTTP requests to invoke the PreStop hook function to implement graceful shutdown.

  • Exec script: Directly shut down the application process. Alternatively, you can execute the preset stop.sh script to make the application process exit.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: my-app-image:latest
        lifecycle:
          preStop:
            exec:
              # command: ["/bin/sh", "-c", "kill -15 PID && sleep 30"]
              command: ["/bin/sh", "-c", "Script path /stop.sh"] 
  • HTTP interface:

    • When you enable the SpringBoot actuator shutdown feature, you can set Path to /actuator/shutdown.

    • For non-Spring applications, you can set Path to /schedulerx/worker/shutdown after you enable the port over which you can call the HTTP interface by using SchedulerX SDKs to implement graceful shutdown.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: my-app-image:latest
        lifecycle:
          preStop:
            httpGet:
              path: /schedulerx/worker/shutdown
              port: 51886
              scheme: HTTP

Scenario 3: Integrate graceful shutdown into an application release platform in Alibaba Cloud

If you release your application by using Alibaba Cloud Enterprise Distributed Application Service (EDAS) or by using Microservices Engine (MSE) and Container Service for Kubernetes (ACK), the graceful shutdown feature of SchedulerX is automatically integrated into the platform that you use after you enable the graceful shutdown feature on the platform. For more information, see Gracefully shut down Spring Cloud applications and Graceful shutdown.