Alibaba Cloud Linux 2 ECS instances fail to create a large number of processes after a large number of processes are created - Alibaba Cloud Linux

Note

Disclaimer: this document may contain information about third-party products that are for reference only. Alibaba Cloud does not make any guarantee, express or implied, with respect to the performance and reliability of third-party products, as well as potential impacts of operations on the products.

Problem Description

When a system call (fork/clone) is executed Alibaba Cloud Linux 2 an ECS instance of the system, process creation fails and" -1 EAGAIN (Resource temporarily unavailable)"Error. Two scenarios are available:

Scenario 1: When the shell commands are used for interaction, the system returns a" bash: fork: retry: No child processes"Error.
Scenario 2: Some applications fail to create processes or threads while instances in other applications of the system can be created successfully.

Cause of problem

The cause of the problem varies according to the scenario.

Scenario 1: The number of user-created threads in the system reaches the maximum. For more information about how to view the maximum value, use the ulimit -u command.
Scenario 2: The number of processes created by the users who run the application (nr_user_process) is greater than the app_limit value of the application, leading to an error.

Solution

Note

Alibaba Cloud reminds you that:

If you have any risky operations on an instance or data, pay attention to the disaster tolerance and fault tolerance capabilities of the instance to ensure data security.
If you modify the configuration and data of an instance (including but not limited to ECS and RDS), we recommend that you create snapshots or enable RDS log backup.
If you have granted permissions on the Alibaba Cloud platform or submitted security information such as the logon account and password, we recommend that you modify the information as soon as possible.

Temporary solution

The solution varies according to the scenario.

Scenario 1: Follow these steps to adjust the maximum number of threads you can create:
1. Switch to the root user to view all applications of the user whose process creation failed.
2. By kill to forcibly close the application that occupies the majority of threads.
3. Run the following command to specify the maximum number of threads that you can create:
  Note
  Note: kill create a snapshot for the ECS instance or back up important files to ensure data security.
```
ulimit -u [$Num_Of_Process]
```
  Note
  Note:[$Num_Of_Process] is the maximum thread value after adjustment.
Scenario 2: Follow these steps to use the limit action of a process:
1. Run the following command to install util-linux tool:
```
yum install -y util-linux
```
  Note
  Note: skip this step if the tool is already installed.
2. Run the following command to adjust the limit resource for the process:
```
prlimit --pid [$PID] --nproc=unlimited
```
  Note
  Note:[$PID] indicates the process ID of a process or thread that fails to be created.

Fixed solution

We recommend that the administrator plan ulimit resources properly. For more information about how to plan the resource, see the following documents.

How to set ulimit values

Applicable to

Elastic Compute Service