Realtime Compute for Apache Flink:Configure resources for a deployment

You can use the following formulas when you configure resources for a deployment:

• Number of CUs to be configured for a deployment = MAX(Total number of CPU cores of the JobManager and TaskManagers, Total memory size of the JobManager and TaskManagers/4).

• Number of IP addresses required for each deployment = Number of JobManagers + Actual number of TaskManagers. Only one JobManager exists in each deployment.

  Note

  If you purchase a workspace after May 30, 2024, you do not need to pay attention to this formula. For more information, see Network architecture upgrade.

• Actual number of TaskManagers = Value of Parallelism/Value of TaskManager Slots. This formula is used when the total number of CPU cores of each TaskManager is less than or equal to the default maximum number of CPU cores of each TaskManager or the total memory size is less than or equal to the default maximum memory size of each TaskManager.

• Actual number of TaskManagers = MAX(⌈Total number of CPU cores/16⌉,⌈Total memory size/64⌉). This formula is used when the total number of CPU cores of each TaskManager is greater than the default maximum number of CPU cores of each TaskManager or the total memory size is greater than the default maximum memory size of each TaskManager.

  • Total number of CPU cores of each TaskManager = Value of Parallelism/Value of TaskManager Slots × Value of Task Manager CPU.

  • Total memory size of each TaskManager = Value of Parallelism/Value of TaskManager Slots × Value of Task Manager Memory.

  • The default maximum number of CPU cores of each TaskManager is 16.

  • The default maximum memory size of each TaskManager is 64 GiB.

• Actual number of slots that can be allocated to each TaskManager = ⌈Value of Parallelism/Actual number of TaskManagers⌉.

For example, you can set Parallelism to 12 and TaskManager Slots to 4. The following figure shows the configurations for this example.

You can find that the actual number of TaskManagers is 3 and the actual number of slots for each TaskManager is 4 in the development console of Realtime Compute for Apache Flink.

The following formulas are used to calculate the actual number of TaskManagers and the actual number of slots for each TaskManager:

1. Actual number of TaskManagers = [Value of Parallelism/Value of TaskManager Slots] = [12/4] = 3.

2. Actual number of slots for each TaskManager = ⌈Value of Parallelism/Actual number of TaskManagers⌉ = ⌈12/3⌉ = 4.

Configure basic resources

Configure slot resources

1. In the expert mode, click Get Plan Now in the Resources section and configure a resource configuration plan.

   

2. In the upper-right corner of the SLOT box, click the icon.

3. Modify the slot configurations.

   The Parallelism parameter that you configure in the Modify SLOT(default) dialog box is the parallelism of all operators in the SSG. After you configure the Parallelism parameter, the system automatically performs the following operations:

   • The system automatically applies the parallelism to all operators in the SSG.

   • The system automatically calculates the memory size that is required by the state backend, Python, and operators based on the computational logic of the deployment. Manual configurations are not required.

   Note

   • Make sure that the number of partitions can be divided by the deployment parallelism for source nodes without leaving a remainder. For example, if a Kafka cluster has 16 partitions, we recommend that you set the Parallelism parameter to 16, 8, or 4 to prevent data skew. We also recommend that you do not set the Parallelism parameter for a source node to an excessively small value. If the Parallelism parameter is set to an excessively small value, the source node may read excessive data. In this scenario, a data input bottleneck may occur and the deployment throughput may be reduced.

   • We recommend that you configure the deployment parallelism for all nodes except source nodes based on the amount of data. If the amount of data on a node is large, we recommend that you set the Parallelism parameter to a large value. If the amount of data on a node is small, we recommend that you set the Parallelism parameter to a small value.

   • We recommend that you adjust the size of heap memory and off-heap memory only when an exception occurs or the deployment throughput is low. For example, an out of memory (OOM) error or a severe garbage collection issue occurs in your deployment. If you adjust the size of heap memory and off-heap memory when your deployment runs normally, the deployment throughput cannot be significantly increased.

4. Click OK.

Configure operator resources

By default, all operators are placed in one SSG. You cannot separately modify the resource configuration of each operator. If you want to configure resources for individual operators, you must enable the Multiple SSG mode to ensure that each operator has an independent slot. This way, you can configure resources for each operator in a slot. To configure resources for an operator, perform the following steps:

1. In the upper-right corner of the Resources section of the Configuration tab, click Edit, and set Mode to Expert.

2. (Optional) If no resource plans are available, click Get Plan Now.

   

3. Turn on Multiple SSG and click Re-fetch.

   Each operator in an SSG is assigned its own slot.

   

4. Click the icon in the SLOT box of the desired operator to modify the resource configuration of the operator.

   

5. Click OK.

Configure the parallelism, chaining strategy, and TTL of an operator

You can configure the parallelism, chaining strategy, and TTL for a single operator.

1. Click the icon to display vertex details in the VERTEX box.

   

   Note

   You can click the icon in the VERTEX box and configure the Parallelism parameter for all operators in the vertex at a time.

2. Click the icon of an operator.

   

3. Configure operator resources.

   

   The following table describes the parameters.

   Parameter	Description
   Parallelism	The parallelism of the operator.
   Chaining Strategy	A chain is a logical computing chain that is formed by multiple operators. A chain can help improve the execution efficiency and performance of deployments and reduce the overhead of data transmission and serialization between operators. In specific scenarios, you may need to disconnect the chain to control the execution flow of deployments and improve deployment performance in an efficient manner. The following chaining strategies are supported: ALWAYS: The operator can always be chained together with the upstream and downstream operators. This is the default value. HEAD: The operator serves as the head node of a chain. The upstream operators are disconnected from the chain. The downstream operators are chained with the current operator. NEVER: The operator cannot be chained with the upstream and downstream operators.
   State Expiration Time Settings	You can configure the time-to-live (TTL) in seconds, minutes, hours, or days. The default value of this parameter is the expiration time of the deployment. If you do not specify an expiration time for a deployment, 1.5 days is automatically used. For more information about how to specify an expiration time for a deployment, see the "Parameters section" of the Configure a deployment topic. Note Only Realtime Compute for Apache Flink that uses VVR 8.0.7 or later supports this parameter. Only stateful operators support the TTL configuration.

4. Click OK.