Batch Compute:Submit job

This article describes the commands used to submit various types of jobs.

1. Submit a simplest job

If you have configured the default image, instance type, and network type according to Preparations, use the following simple command to submit a job.

bcs sub "echo 123" # Submit and run a single-task job "echo 123".

If default settings are unavailable for the BatchCompute-cli tool, you must specify more parameters when submitting a job.

bcs sub "echo 123" -t ecs.sn1ne.large -i img-ubuntu-vpc --vpc_cidr_block 192.168.0.0/16

For the remaining examples in this article, it is assumed that you have configured the default settings.

2. Submit a Python script job

bcs sub "python test.py" -p ./test.py # "-p" indicates that test.py is packed into worker.tar.gz and then uploaded to OSS before the job is submitted.

3. Submit a directory (multi-file) job

Generally, the src directory contains multiple files, for example:

src
  |-- test.py
  |-- dep.py

Test.py must depend on dep.py. Therefore, the entire directory can be packed.

bcs sub "python test.py" -p ./src/ # Pack all files under the src directory into worker.tar.gz, upload it to the OSS, and then submit the job.

Specify multiple files at a time and separate these files by commas (,):

cd src  # Switch to the src directory.
bcs sub "python test.py" -p test.py,dep.py # Pack the two files into worker.tar.gz.

If you do not switch to the src directory, submit the job as follows:

bcs sub "python test.py" -p src/test.py,src/dep.py # Pack the two files into worker.tar.gz.

Then, use the following command to view the content of worker.tar.gz:

tar -tvf worker.tar.gz

The content of worker.tar.gz is as follows:

test.py
dep.py

4. Submit a job by mounting a task program

Assume that you have uploaded test.py to oss://mybucket/test/test.py.You can mount the OSS directory “oss://mybucket/test/“ to a VM local directory, for example, “/home/admin/test/“. Then use the command “python /home/admin/test/test.py” to run the program.

Submit a job as follows:

bcs sub "python /home/admin/test/test.py" -r oss://mybucket/test/:/home/admin/test/

The parameter -r oss://mybucket/test/:/home/admin/test/ indicates read-only mounting and “oss://mybucket/test/“ is mounted onto “/home/admin/test/“.

In this way, you do not need to pack test.py and dep.py into worker.tar.gz.

For more information about how to submit a job, run “bcs sub -h” to view the help information.

Other tasks

1. Mount the input data

Assume that your data has been uploaded to the directory “oss://my-bucket/inputs/“.

bcs sub "python test.py" -p ./src/  -r oss://my-bucket/inputs/:/home/admin/inputs/

• “-r” indicates read-only mounting and “oss://my-bucket/inputs/“ is mounted to “/home/admin/inputs/“. In this way, your program can read files under the directory “/home/admin/inputs/“ same as reading local files.

• All mounted directories cannot be system directories, such as “/bin” and “/usr”. The directory “/home/“ is recommended.

• To mount multiple directories, separate them by using commas (,). For example, -r oss://my-bucket/inputs/:/home/admin/inputs/,oss://my-bucket/inputs2/:/home/admin/inputs2/.

2. Use the mounting method to automatically upload the program running results

Your program will write the running results to the directory “/home/admin/outputs/“. To upload all data under this directory to OSS, run the following command:

bcs sub "python test.py" -p ./src/ -r oss://my-bucket/inputs/:/home/admin/inputs/ -w oss://my-bucket/outputs/:/home/admin/outputs/

• “-w” indicates writable mounting and all data written to this directory will be automatically uploaded to the corresponding OSS directory after the program finishes running.

• All mounted directories cannot be system directories, such as “/bin” and “/usr”. The directory “/home/“ is recommended.

• To mount multiple directories, separate them by using commas (,). For example, -w oss://my-bucket/outputs/:/home/admin/outputs/,oss://my-bucket/outputs2/:/home/admin/outputs2/.

3. Use the custom image and instance type

bcs sub "python test.py" -p ./src/ -c img=img-ubuntu-vpc:type=ecs.sn1ne.large

• “-c” indicates that a cluster is used. Next to “-c”, you can specify the cluster ID or the AutoCluster configuration. The AutoCluster configuration format is as follows: img=${ImageId}:type=${InstanceType}.

• “img=img-ubuntu-vpc” indicates that an image is used. You can specify a custom image.

• “type=ecs.sn1ne.large” indicates that an instance type is used. You can run bcs it to view the list of the supported instance types.

• Specify only -c img=${ImageId} or only -c type=${InstanceType}. If none of them are specified, the Default image and default instance types are used.

4. Use a cluster

bcs sub "python test.py" -p ./src/ -c cls-xxxxxxx

• “-c cls-xxxxxxx” indicates that the cluster with the ID set to cls-xxxxxxx is used.

For more information about the cluster, see Use a cluster.

5. Use Docker

bcs sub "python test.py" -p ./src/ --docker myubuntu@oss://my-bucket/dockers/

• “myubuntu” is short for “localhost:5000/myubuntu”. “oss://my-bucket/dockers/“ indicates the path of the OSS Docker registry.

• For more information, see Use the Docker.

6. Custom disks

bcs sub "python test.py" --disk system:cloud_efficiency:50,data:cloud_efficiency:200

• The command is valid only when AutoCluster is used. System disks and one data disk (optional) can be configured. The format is similar to -disk system:cloud_efficiency:40,data:cloud_efficiency:50:/home/disk1. The system disk configuration is separated from the data disk configuration by a comma (,). You can specify only the system disk or only the data disk, for example:-disk system:cloud_efficiency:40.

• By default, only one system disk is mounted, and the size is 40 GB.

• The format of the system disk configuration is as follows: system:< cloud_efficiency|cloud_ssd>:<40-500>. For example, “system:cloud_efficiency:40” indicates that the system disk is mounted to a 40 GB ultra cloud disk.

• The format of the data disk configuration is as follows: data:< cloud_efficiency|cloud_ssd>:<5-2000>:For example, “data:cloud_ssd:50:/home/disk1” indicates that a 50 GB SSD cloud disk is mounted as a data disk. In the Windows operating system, the data disk can be mounted only to the drive. For example, “data:cloud_ssd:50:E” indicates that the data disk is mounted to the E drive.

View job information

For more information, see How to view the job information.