Microservices Engine:Java SDK

Feature

Type

Description

References

Workflow

New feature

Workflow IDs can be obtained by using JobContext.

JobContext parameters

Optimized SDK

Optimized feature

• Unnecessary dependencies are removed from SchedulerX SDK for Java.

• Log4j2.xml is removed from SchedulerX SDK for Java, which prevents log configuration conflicts.

N/A

Feature

Type

Description

References

Kubernetes job

New feature

Job parameters can be read from environment variables.

N/A

XXL-JOB log

New feature

Log isolation by application is supported.

N/A

Security and log display

Fixed issue

• Security vulnerabilities in Netty are fixed.

• Security vulnerabilities in Logback are fixed.

• The following issue is fixed: Logs of the SchedulerX framework cannot be displayed due to log conflicts.

N/A

Feature

Type

Description

References

Graceful shutdown

New feature

The graceful shutdown feature is supported. An application can be shut down only after all the ongoing jobs of the application are complete. This ensures service availability during the release or restart process.

Gracefully shut down jobs and tasks executed in an application

Push model job

New feature

Push model jobs are optimized to support the configuration of job distribution rate.

N/A

H2 database

New feature

H2 databases are optimized to support the configurations of username and password.

• spring.schedulerx2.h2DatabaseUser: the username of the H2 built-in database.

• spring.schedulerx2.h2DatabasePassword: the password of the H2 built-in database.

Connect a Spring Boot application to SchedulerX

Business log

Optimized feature

Business logs can be isolated by instance group.

N/A

Broadcast job

Optimized feature

Broadcast jobs are optimized to support sorting and distribution by worker IP address.

N/A

Spring job

Optimized feature

• Spring jobs are optimized to support Spring 4.

• You can specify scheduling methods for Spring jobs to return any processing results.

Spring jobs

Optimized SDK

Optimized feature

• SchedulerX SDK for Java is optimized to prevent heartbeat threads from being repeatedly created when Spring containers are repeatedly initialized.

• Akka Actor supports automatic restart upon unexpected termination.

N/A

Job execution

Fixed issue

• The following issue is fixed: An exception is returned after Spring fails to read a processor bean.

• The following issue is fixed: Map jobs are stuck in V 1.10.x.

• The following issue is fixed: The instance of a broadcast job cannot be terminated when an exception occurs during the execution of preProcess of the broadcast job.

• The following issue is fixed: An alert is triggered due to a job execution timeout error caused by residual threads when the job instance is stopped unexpectedly.

• The following issue is fixed: The status of an application is unknown or stuck upon application startup in the shared thread pool mode.

• The following issue is fixed: The access configurations cannot be loaded as expected when multiple instance groups or AppKeys contain special characters.

• The following issue is fixed: You cannot use the features of a visual MapReduce job of the Professional edition.

N/A

Feature

Type

Description

References

Template for Kubernetes script jobs

New feature

The following types of templates are supported for Kubernetes script jobs:

• Standard templates: Images and image pull policies can be modified in standard templates.

• Custom templates: Custom templates are compatible with native Kubernetes. To use a custom template, you must update schedulerx2-plugin-kubernetes to V1.0.5 or later.

N/A

Distribution method for Map jobs

New feature

The distribution of Map jobs can be started from a random position. To use this feature, you must set the spring.schedulerx2.mapMasterDispatchRandom parameter to true. The default value of this parameter is false.

N/A

Built-in parameter for Shell jobs

New feature

The following regular built-in parameters are supported for Shell jobs:

• schedule.timestamp: the time when a job is scheduled.

• data.timestamp: the time offset of a job when the job is scheduled.

N/A

Feature

Type

Description

References

Adjustment for execution thread models

New feature

Adjustment for execution thread models is supported.

• Execution thread models can be switched to the unshared thread pool mode.

• Thread pool parameters can be shared by models in shared thread pool mode.

  • spring.schedulerx2.shareContainerPool=true: enables the shared thread pool.

  • spring.schedulerx2.sharePoolSize=32: the global size of the shared thread pool. Default value: 64.

  • spring.schedulerx2.sharePoolQueueSize=20480: the queue size of the shared thread pool. Default value: Integer.MAX_VALUE.

N/A

Idle master node and retry on failure for broadcast jobs

New feature

The following parameters can be specified to support the idle master node and retry on failures:

• spring.schedulerx2.broadcastMasterExecEnable=false: specifies whether the master node is required to execute a broadcast job. Default value: true.

• spring.schedulerx2.broadcastDispatchRetryTimes=5: the number of retries that are allowed after a broadcast job fails to be distributed.

N/A

Map job

Optimized feature

The performance of Map jobs during large-scale application release and scale-in is optimized.

N/A

Heartbeat checkpoint for application startup

Optimized feature

The heartbeat checkpoint for application startup is optimized to prevent time-consuming job execution that is caused by long startup time of containers.

N/A

Job execution

Fixed issue

• The following issue is fixed: A job is rejected after it runs for a long time in shared thread pool mode.

• The following issue is fixed: Empty pointer logs are occasionally generated when job status is being reported.

• The following issue is fixed: The number of broadcast job threads continues to grow after the release of V1.7.X.

• The following issue is fixed: Memory leaks occur in shared thread pool mode.

• The following issue is fixed: A dynamic proxy cannot implement the Reduce method.

• The following issue is fixed: A second-level job is stuck because specific responses are missing after the execution responses are combined.

• The following issue is fixed: An error occurs when Diamond reads environment addresses.

• The following issue is fixed: The heartbeat timeout period is excessively long when you connect to Alibaba Cloud.

• The following issue is fixed: Alerts are triggered in the log4j log format.

• The following issue is fixed: If a standalone job fails to be distributed, the job is stuck.

• The following issue is fixed: The new version is incompatible with the previous ClassName:BeanId configuration mode.

• The following issue is fixed: The maximum file size of a Logback standalone log cannot be increased.

N/A

Type

Description

References

New feature

• Native and scheduled Spring jobs can be integrated.

• End-to-end tracing is supported.

• Spring jobs

• Integrate tracing analysis

Optimized feature

The response is accelerated when an agent is stopped or disconnected.

N/A

Fixed issue

• The following issue is fixed: The Out of Memory (OOM) error occurs when a large number of Simple Log Service logs are generated for a job that is scheduled frequently.

• The following issue is fixed: Large latency occurs when many second-delay jobs are run together.

N/A

Type

Description

References

Fixed issue

The following issue is fixed: High CPU usage is caused by second-delay broadcast detection threads.

N/A

Type

Description

References

New feature

Kubernetes job scheduling capabilities can be integrated.

N/A

Optimized feature

• The processor class can be used to configure the Spring Aspect Orient Programming (AOP) proxy.

• The content in the error log that is generated occasionally for an available worker monitor timer is optimized.

• Logs of second_delay jobs can be collected.

N/A

Type

Description

References

New feature

You can dynamically enable or disable the logging feature.

N/A

Fixed issue

• The following issue is fixed: V1.5.0.x is incompatible with Spring Boot 2.0.3.

• The following issue is fixed: An error may be reported during JobSyncService Bean initialization in V1.5.0.x.

Type

Description

References

New feature

Spring Boot declarative jobs are supported. You can declare namespaces, applications, jobs, and alerts by using configuration files. This facilitates job management. You can modify configuration files based on your business requirements and launch your application with a few clicks in all environments.

Periodic jobs for Spring Cloud applications

Visualized MapReduce jobs, formerly known as parallel computing jobs, are supported. You can view the tasks of a specific visualized MapReduce job, view the logs of visualized MapReduce jobs, and rerun visualized MapReduce jobs. You can also add custom tags to tasks and filter tasks by tag.

Enterprise-class distributed batch processing solution

Type

Description

References

New feature

The logging feature is supported by SchedulerX V2.0. To enable the logging feature, you need to only add the Log4j or Logback configuration to the corresponding job. Then, you can view the log that is generated each time the job is triggered in the SchedulerX console. Distributed jobs also support the logging feature.

Integrate Simple Log Service with SchedulerX 2.0

You can view the stacks based on which jobs are run in the SchedulerX console. This helps troubleshoot jobs that are stuck. You must update your agent to 1.4.0 or a later version before you can view stacks.

N/A

SchedulerX can be interfaced with the executor that is developed for elastic jobs.

Type

Description

References

New feature

• DataWorks jobs are supported.

• schedulerx2-plugin-xxljob is added to support the XXL-JOB interface.

• DataWorks jobs

• XXL-JOB jobs

Type

Description

References

Optimized feature

The default dependency Log4j 2 is updated to V2.15.0.

N/A

Type

Description

References

New feature

Second-delay jobs enhanced in standalone mode: Second-delay jobs can be distributed to different idle workers in standalone mode. To distribute second-delay jobs to different idle workers in standalone mode, set the spring.schedulerx2.enableSecondDelayStandaloneDispatch parameter to true. Default value of the parameter: false.

N/A

Optimized feature

The response is accelerated when an agent is stopped or disconnected.

N/A

Fixed issue

• The following issue is fixed: The running workers send heartbeats only to the specified scheduled service if an agent belongs to multiple instance groups.

• The following issue is fixed: Threads are occasionally interrupted or the null pointer exception occasionally occurs when SchedulerX schedules second-delay jobs.

• The following issue is fixed: Jobs become stuck and time out if the SchedulerX agent is integrated with a Spring Cloud application.

• The following issue is fixed: Jobs are interrupted if SchedulerX runs the jobs in parallel in broadcast mode.

• The following issue is fixed: SchedulerX fails to update the worker list when SchedulerX runs second-delay jobs.

• The following issue is fixed: The scan thread is not terminated after a second-delay job that is scheduled in broadcast mode executes the terminate instruction.

• The following issue is fixed: A second-delay job that is scheduled in broadcast mode may become stuck when a large number of workers publish the job in batches.

N/A

Type

Description

References

Fixed issue

The following issue is fixed: Jobs become stuck and time out if the SchedulerX agent is integrated with a Spring Cloud application.

N/A

Type

Description

References

New feature

Second-delay jobs enhanced in standalone mode: Second-delay jobs can be distributed to different idle workers in standalone mode. To distribute second-delay jobs to different idle workers in standalone mode, set the spring.schedulerx2.enableSecondDelayStandaloneDispatch parameter to true. Default value of the parameter: false.

N/A

Fixed issue

The following issue is fixed: Jobs are interrupted if SchedulerX runs the jobs in parallel in broadcast mode and the list of available workers is not updated.

N/A

Type

Description

References

Fixed issue

The following issue is fixed: The scan thread is not terminated after a second-delay job that is scheduled in broadcast mode executes the terminate instruction.

N/A

Type

Description

References

Fixed issue

The following issue is fixed: A second-delay job that is scheduled in broadcast mode may become stuck when a large number of workers publish the job in batches.

N/A

Type

Description

References

New feature

• Enhanced broadcast mode: postProcess can be used to retrieve the status (JobContext.getTaskStatuses()) and execution results (JobContext.getTaskResults()) of all workers.

• Enhanced sharding broadcast mode: JobContext can be used to retrieve the number (shardingNum) and IDs (sharingId) of shards. You can use the index of your worker to process data in a distributed manner.

• Enhanced sharding mode: JobContext can be used to retrieve the number (shardingNum) of all shards.

N/A

Type

Description

References

New feature

A failover switch is added to allow you to enable or disable the failover feature.

N/A

Fixed issue

• The following issue is fixed: The socket may be disclosed if the heartbeat probe fails.

• The following issue is fixed: MapTaskMaster frequently performs failovers when MapTaskMaster needs to handle a large number of tasks.

N/A

Type

Description

References

New feature

Second-delay jobs can be triggered at a millisecond granularity.

N/A

Optimized feature

• The MapReduce model is optimized. Root tasks are scheduled only to the master worker. This can help you identify the cause when exceptions occur.

• The SchedulerX agent is no longer strongly reliant on Log4j 2. If you use Logback, you can remove the Log4j and Log4j 2 dependencies.

N/A

Type

Description

References

New feature

• The progress of jobs that are scheduled in broadcast mode can be reported.

• Heartbeat logging can be enabled or disabled.

N/A

Fixed issue

• The interval of heartbeat probes is changed to 10 seconds.

• The following issue is fixed: Broadcast jobs may be stuck.

N/A

Type

Description

References

Optimized feature

The following issue is fixed: Jobs cannot continue to be scheduled due to traffic surges during the persistent connection caused by heavy workloads on the SchedulerX agent.

N/A

Type

Description

References

New feature

• The actual CPU utilization of a container in a Container Service for Kubernetes (ACK) cluster can be collected. However, you must add the corresponding configurations. For example, if you use starter, you must perform the following steps: Set the spring.schedulerx2.enableCgroupMetrics parameter to true and the spring.schedulerx2.cgroupPathPrefix parameter to /sys/fs/cgroup/cpu/. If the /sys/fs/cgroup/cpu/ path exists on your container, you do not need to specify the spring.schedulerx2.cgroupPathPrefix parameter. Otherwise, you must specify the actual path of cgroup.

• Tags are supported. You can add tags to workers after you launch the SchedulerX agent. For example, if you use starter, you can set the spring.schedulerx2.labels parameter to xxx. You can specify a worker that has specific tags to run a job in the Task Management module. This feature is suitable for canary release and cell-based management scenarios.

N/A

Optimized feature

• The heartbeat thread and Akka thread of the SchedulerX agent can run independently. This ensures that heartbeat probes are not affected when the SchedulerX agent is busy.

• h2 is removed from shade. shade h2 may cause the failure to load the MySQL driver.

N/A

Type

Description

References

New feature

• The shared container pool feature is supported. All jobs that are scheduled by the SchedulerX agent can share the same thread pool. This increases the performance and stability of the SchedulerX agent when the agent needs to schedule a large number of jobs in parallel.

• Shade Scala is supported to eliminate scala conflicts.

• Authentication based on AppKeys is supported in the public cloud.

• The MapReduce model is enhanced and allows you to specify whether to perform the Reduce method only after all tasks are complete.

• The AppKey can be specified in the -Dschedulerx.appKey parameter on the SchedulerX agent.

N/A

Optimized feature

The Netty thread pool of the SchedulerX agent is optimized.

N/A

Fixed issue

• The following issue is fixed: Jobs that are scheduled in broadcast mode may become stuck.

• The following issue is fixed: The AccessKey ID or AccessKey secret of Simple Log Service is leaked.

N/A

Type

Description

References

New feature

• More than 100,000 jobs are supported by an application that is deployed in the public cloud.

• A logging switch is added to allow you to enable or disable logging for the SchedulerX agent.

• The status of a job can be configured if you use the SchedulerX API to create the job.

• The diamond-client, logger.API, and Log4j dependencies are removed.

N/A

Fixed issue

• The following issue is fixed: The SchedulerX agent disconnects from the server if you disable Internet access for the agent.

• The following issue is fixed: EDAS applications cannot read the AccessKey secret.

N/A

Type

Description

References

New feature

• Namespaces can be created.

• Multiple SchedulerX workers can be initialized at the same time.

• The MapReduce model is enhanced.

• The Reduce method can be performed even if a task fails.

• JobContext.getTaskStatuses can be used to obtain the status of a task. The key of the Map<Long,TaskStatus> structure indicates the ID of the task and the value indicates the status of the task.

N/A

Fixed issue

• The following issue is fixed: A null pointer error occurs if ProcessResult returns an empty result.

• The following issue is fixed: A job becomes stuck if thread-dispatcher-delivery becomes hung.

N/A

Type

Description

References

New feature

shade protobuf and netty from AccessKey IDka are used to eliminate more than 90% of JAR package conflicts when you connect an agent to SchedulerX.

N/A

Optimized feature

AppKeys are not supported when you create multiple groups.

N/A

Type

Description

References

New feature

• Support for sharding models in multiple languages (similar to elastic-job): ShedulerX2.0 supports sharding models in multiple languages.

• The AppKey of a group can be returned when you use the SchedulerX API to create the group.

• Job instances that are in the Successful state can be rerun. Job instances in a workflow can rerun their jobs and their downstream jobs.

N/A

Optimized feature

• The performance of the server is optimized. The method that the server uses to communicate with the SchedulerX agent is changed from synchronous communication to asynchronous communication. The configurations of the default dispatcher of Akka are optimized.

• The performance of heartbeat probes provided by the SchedulerX agent V1.1.0 is increased by three times.

• The Task Management page in the SchedulerX console is redesigned to display more information.

N/A

Fixed issue

• The following issue is fixed: Global tasks may not take effect if the pull model is used to distribute jobs.

• The following issue is fixed: The SchedulerX agent may fail to be launched in an isolated cell environment if no domain is specified.

N/A

Type

Description

References

New feature

• The BlockAppStart setting is added. The setting specifies whether to block the startup of the application when SchedulerX fails to start up. The default value is true.

• An API operation is added to allow you to query the status of a workflow. The API operation is GetWorkflowInstanceRequest.

• The jobName field is added to jobContext. This allows you to obtain the name of a running job.

N/A

Fixed issue

• The following issue is fixed: The result of Hessian deserialization on BigDecimal is 0.

• The following issue is fixed: A LocalDateTime error occurs after Hessian deserialization is complete.

• The following issue is fixed: Tasks of a job fail to be scheduled to specified workers. The following issue is fixed: If a job runs for a specific period of time, the tasks of the job are scheduled to workers that you did not specify.

• The following issue is fixed: The springContext.getBean of the SchedulerX agent throws an AnnotationConfigApplicationContext has not been refreshed yet exception.

• The following issue is fixed: The ServeletWebServer of Spring Boot is terminated if the configuration of the implementation class of the job is invalid. As a result, the web service is shut down but the application process continues to run.

• The following issue is fixed: A job becomes stuck if the system enables the user.dir='/' variable.

• The following issue is fixed: The springContext.getBean of the SchedulerX agent throws an AnnotationConfigApplicationContext has been closed already exception.

• The following issue is fixed: The workerId values that are generated by the SchedulerX agent may be duplicate. As a result, the job may be scheduled to workers that do not belong to the application to which the job belongs.

• The following issue is fixed: Custom class loaders are not supported by Spring applications.

• The following issue is fixed: The counter that counts the number of times a second-delay job in broadcast mode is triggered does not display the correct value.

• The following issue is fixed: jobContext.getScheduleTime is not updated when a second-delay job is periodically scheduled.

N/A

Type

Description

References

New feature

• Important changes: JobProcessor.postProcess is reconfigured. ProcessResult is added to return values. You need to modify the code of the job if you use the original postProcess interface.

• The broadcast mode is enhanced. BroadcastJobProcessor supports preProcess and postProcess. All workers must perform preProcess once before they perform process. All workers must perform postProcess once after they perform process.

• JobContext.getTaskAttempt can be used to retrieve the number of times that SchedulerX attempts to run the current task.

• Custom listening ports can be specified for the SchedulerX agent. Example: SchedulerxWorker.setPort.

• JobProcessor can be implemented by Java jobs. Java jobs no longer need to inherit JavaProcessor.

N/A

Fixed issue

• The following issue is fixed: Tasks whose TaskId is set to 1 cannot be automatically retried.

• The following issue is fixed: The cause of failure is not displayed if the root task of a distributed job fails.

• The following issue is fixed: Tasks in the task list of a job that is scheduled in parallel computing mode cannot be retried.

N/A

Type

Description

References

Optimized feature

This SchedulerX agent version is compatible with the SchedulerX 1.0 (DTS) interface. The agent version cannot rely on both the schedulerx-client and schedulerx-worker packages. The agent version can rely on only the schedulerx-worker package. Therefore, you must migrate all Data Transmission Service (DTS) jobs to SchedulerX 2.0.

N/A

Type

Description

References

New feature

• shade: aliyun-log, commons-validator, gson, fastjson, guava, and commons-collections are added.

• errorMsg can be displayed in logs in the SchedulerX console if you use ProcessResult(false,errorMsg) to return execution results.

Optimized feature

• The performance of at-least-once-delivery is optimized.

• Duplicate messages are no longer submitted in heavy load scenarios. Duplicate messages caused second-delay jobs to become stuck or interrupt application threads.

N/A

Fixed issue

• The following issue is fixed: Jobs that are scheduled in broadcast mode become stuck.

• The following issue is fixed: Second-delay jobs become stuck.

• The following issue is fixed: at-least-once-delivery may cause SchedulerX to keep retrying specific tasks.

• The following issue is fixed: The SchedulerX agent fails to start up if logcollector fails to initialize.

• The following issue is fixed: A workflow cannot be recovered because the database cleans the job instances in the workflow.

• The following issue is fixed: kill is not supported if the application is launched by using Spring.

N/A

Type

Description

References

New feature

• The execution results of all tasks can be returned by the MapReduce model and processed by using the Reduce method.

• The pull model is supported for jobs that use distributed programming models. This model can eliminate the barrel effect that occurs due to the limited performance of standalone computing. This model allows workers to dynamically scale out to pull tasks.

• The global degree of concurrency can be configured for tasks if you use the pull mode. This helps you limit the number of concurrent tasks.

• wfInstanceId is added to JobContext.

• The startup of the Java virtual machine (JVM) is blocked when the SchedulerX agent fails to be started and an exception occurs. This allows you to troubleshoot issues at the earliest opportunity.

• The SchedulerX agent prints the JAR version and path on which Maven relies when the SchedulerX agent starts up. This helps you troubleshoot JAR package conflicts.

• Distributed programming models can return details about tasks. You can view the queue of tasks that are cached on each worker.

N/A

Fixed issue

• Workers that are quarantined by mistake due to high workloads on servers can be automatically recovered without restarting the SchedulerX agent.

• Distributed programming models can return details about tasks. You can view the number of tasks that run on each worker.

• The server automatically cleans the resources on slave nodes to prevent memory leaks if the master node fails.

N/A

Type

Description

References

New feature

• The following expressions are supported when you create a periodic scheduling job: crontab and fixed_rate.

• Workflows are supported. You can create workflows to orchestrate jobs.

• Second-delay expressions are supported. You can use second-delay expressions to create second-delay jobs.

• Jobs that use the following programming languages are supported: Java, Shell, Python, and Go.

• The following execution modes are supported: standalone, broadcast, parallel computing, memory grid, and grid computing.

• The following distributed programming models are supported: Map and MapReduce.

• SchedulerX can be configured to automatically retry failed job instances or tasks. By default, this feature is disabled.

• Time-series data is supported and the outputs of jobs can be updated.

N/A