This topic describes how to enable the remote shuffle service in a batch deployment. After the remote shuffle service is enabled in the batch deployment, the issue of insufficient disk capacity is mitigated and the deployment can run efficiently and stably.
Background information
In traditional batch processing scenarios of Realtime Compute for Apache Flink, local disks are used to store shuffled data. This makes full use of the advantages of data localization and significantly improves the processing efficiency. However, this solution is limited by the capacity of local physical disks. When a large amount of data needs to be processed, the computing resources and storage resources of Realtime Compute for Apache Flink are generally increased at the same time. This method may result in a waste of computing resources and may incur high overheads.
To resolve the previous issue, Alibaba Cloud launches the remote shuffle service. The remote shuffle service is free of charge in public preview. After the remote shuffle service is enabled in a batch deployment of Realtime Compute for Apache Flink, shuffled data is stored in high-performance Apache Celeborn clusters and the deployment is no longer limited by the disk capacity of compute nodes of Realtime Compute for Apache Flink. This enhances the capability to process a large amount of data and ensures the high stability and cost-effectiveness of the deployment.
The remote shuffle service is in public preview. Exercise caution when you use the remote shuffle service in the production environment. If you have questions, submit a ticket. Alibaba Cloud will respond in a timely manner and provide professional assistance.
The remote shuffle service is in public preview and is supported only in specific regions. For more information about the supported regions, see Supported regions of the remote shuffle service. If the remote shuffle service is not supported in the region in which your deployment resides, contact Alibaba Cloud for technical support. Alibaba Cloud will provide the service for you at the earliest opportunity.
Limits
Only batch deployments that are not deployed in session clusters are supported.
The remote shuffle service is supported only by Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 8.0.9 or later.
The remote shuffle service is supported only in specific regions. For more information about the supported regions, see Supported regions of the remote shuffle service.
Procedure
Go to the parameter configuration page of a batch deployment.
Log on to the management console of Realtime Compute for Apache Flink.
On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane of the development console of Realtime Compute for Apache Flink, choose .
On the Deployments page, select BATCH from the drop-down list next to Create Deployment.
Find the deployment that you want to manage and click Details. In the panel that appears, find the Parameters section and click Edit in the upper-right corner.
In the Other Configuration field, add the following code:
execution.batch.remote-shuffle.enabled: trueClick Save.
After the parameter modification is saved, it takes effect in the newly started batch deployment.
Supported regions of the remote shuffle service
China (Shanghai)
China (Ulanqab)
If the remote shuffle service is not supported in the region in which your deployment resides, contact Alibaba Cloud for technical support.