How to Implement Parallelism and Concurrency Control (Queue) in Shell

By digoal

Background

1. Implement parallelism in the shell. Execute psql -c "select 1", psql -c "select 2", and psql -c "select 3" in parallel, and always keep the concurrency control to 2. For example, if select 1 has been executed but select 2 has not yet finished, select 3 should be executed immediately.

You can use GNU Parallel to implement parallel execution and concurrency control. First, make sure that GNU Parallel is installed on your system, and then you can use the following command to implement the required functionality:

parallel -j 2 ::: "psql -c 'select 1'" "psql -c 'select 2'" "psql -c 'select 3'"

In this command, the -j 2 indicates that the concurrency is 2, and ::: is followed by the commands to be executed in parallel. When one of the commands completes its execution, GNU Parallel will immediately execute the next command, ensuring that the concurrency is always maintained at 2.

Using this command, you can achieve parallel execution and control the concurrency to ensure that once select 1 has completed, select 3 will be executed immediately, without waiting for select 2 to finish.

1.1. Examples

parallel -j 2 ::: "psql -c 'select pg_sleep(5)'" "psql -c 'select pg_sleep(2)'" "psql -c 'select pg_sleep(3)'"  
  
time parallel -j 2 ::: "psql -c 'select pg_sleep(5)'" "psql -c 'select pg_sleep(2)'" "psql -c 'select pg_sleep(3)'"  
 pg_sleep   
----------  
   
(1 row)  
  
 pg_sleep   
----------  
   
(1 row)  
  
 pg_sleep   
----------  
   
(1 row)  
  
  
real  0m5.157s  
user  0m0.079s  
sys 0m0.089s

2. In the file.txt file, each line contains an executable command. It does not rely on other tools (such as parallel) to realize parallelism in the shell. Commands are extracted from file.txt in sequence and the concurrency is controlled to be 2. The concurrency is always maintained at 2. For example, if select 1 has been executed but select 2 has not yet been completed, then select 3 should be executed immediately

You can use the following shell script to do this:

#!/bin/bash  
  
# Read the commands from file.txt and use the & symbol to execute them in the background  
while IFS= read -r cmd; do  
    # When two commands are being executed, wait until one of them is executed  
    while (( $(jobs | wc -l) >= 2 )); do  
        sleep 1  
    done  
    {  
        $(cmd)  
    } &  
done < file.txt  
  
# Wait for all commands to finish execution  
wait

The above script first reads the command from file.txt and executes it in the background. After executing each command, it checks the number of commands currently executed in the background, and if there are more than 2 commands, it will wait. Finally, use the wait command to wait for all commands to complete execution.

2.1. Examples

b.sh

#!/bin/bash  
  
# Read the commands from file.txt and use the & symbol to execute them in the background  
while IFS= read -r cmd; do  
    # When two commands are being executed, wait until one of them is executed  
    # jobs -p -r  
    while (( $(jobs -p -r | wc -l) >= 2 )); do  
        sleep 0.01  
    done  
    {  
  psql -c "${cmd}"  
    } &  
done < file.txt  
  
# Wait for all commands to finish execution  
wait

file.txt

select pg_sleep(5);  
select pg_sleep(2);  
select pg_sleep(3);  
select pg_sleep(4);  
select pg_sleep(3);

time ./b.sh   
 pg_sleep   
----------  
   
(1 row)  
  
 pg_sleep   
----------  
   
(1 row)  
  
 pg_sleep   
----------  
   
(1 row)  
  
pg_sleep   
----------  
   
(1 row)  
  
pg_sleep   
----------  
   
(1 row)  
  
  
real  0m9.097s  
user  0m0.319s  
sys 0m2.189s

Community

How to Implement Parallelism and Concurrency Control (Queue) in Shell

Background

Read previous post:

Read next post:

digoal

You may also like

Comments

digoal

Related Products

PolarDB for PostgreSQL

PolarDB for Xscale

PolarDB for MySQL

AnalyticDB for PostgreSQL