How to create a workflow and configure CPU and memory requests for the containers in the workflow - Container Service for Kubernetes

Workflow clusters are developed based on open source Argo Workflows. You can refer to the documentation of Argo Workflows to customize workflows. Workflows can be applied to Continuous Integration and Continuous Delivery (CICD) pipelines, data processing, machine learning, and simulation calculation. This topic provides an example on how to use the Alibaba Cloud Argo CLI to create a workflow and configure CPU and memory requests for the containers in the workflow.

Prerequisites

Usage notes

Workflow service accounts

Workflows allow you to specify service accounts to access other Kubernetes resources. You can also create a service account. The workflow cluster automatically grants permissions to the service account that you create. If the service account cannot provide all permissions required by the workflow, join DingTalk group 35688562 and request technical support.

Alibaba Cloud Argo CLI

The Alibaba Cloud Argo CLI is completely compatible with the open source Argo CLI, and has optimized metrics. You can use the Alibaba Cloud Argo CLI to query the CPU usage, memory usage, and operating cost of a workflow. The Alibaba Cloud Argo CLI enhances the logging capability to allow you to obtain the logs of pods that have been deleted from a workflow.

Perform the following steps:

Note

In the following example, Linux is used. You can download one of the following Alibaba Cloud Argo CLI packages based on the operating system that is used:

Darwin: argo-cli-aliyun-darwin
Linux: argo-cli-aliyun-linux

Run the following command to download the Alibaba Cloud Argo CLI:

wget https://ack-one.oss-cn-hangzhou.aliyuncs.com/cli/v3.4.12/argo-cli-aliyun-linux

Run the following command to make argo-cli-aliyun-linux executable:
```
chmod +x argo-cli-aliyun-linux
```
Move the executable file to a directory contained in a environment variable. Example: /usr/local/bin/.
```
mv argo-cli-aliyun-linux /usr/local/bin/argo
```

Download one of the following Alibaba Cloud Argo CLI packages based on the operating system that is used, and rename the package as Argo.

Darwin: Download the Alibaba Cloud Argo CLI.
Linux: Download the Alibaba Cloud Argo CLI.

Create a workflow

You can use the Alibaba Cloud Argo CLI or kubectl to create a workflow.

Use the Alibaba Cloud Argo CLI to manage a workflow

Create a file named helloworld-workflow.yaml and add the following content to the file:

apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec.
metadata:
  generateName: hello-world-    # name of the workflow spec.
spec:
  entrypoint: whalesay          # invoke the whalesay template.
  templates:
    - name: whalesay              # name of the template.
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "hello world" ]

Run the following command to submit the workflow:
```
argo submit helloworld-workflow.yaml
```

Query the status of the workflow.

Run the following command to query a list of workflows:

argo list

Expected output:

NAME                STATUS      AGE   DURATION   PRIORITY
hello-world-lgdpp   Succeeded   2m    37s        0

Run the following command to query the status of the workflow:

argo get hello-world-lgdpp

Expected output:

Name:                hello-world-lgdpp
Namespace:           default
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Succeeded
Conditions:
 PodRunning          False
 Completed           True
....
Duration:            37 seconds
Progress:            1/1
ResourcesDuration:   17s*(1 cpu),17s*(100Mi memory)

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ✔ hello-world-lgdpp  whalesay  hello-world-lgdpp  27s

Use kubectl to manage a workflow

After you configure the kubeconfig file, you can use kubectl to manage workflow clusters. However, unlike regular Kubernetes clusters, some operations are limited when you use kubectl. The following table describes the permissions that you have to manage different resources by using kubectl.

Resource	Permission
priorityclasses	The permissions to manage PriorityClasses and customize PriorityClasses in workflows to control pod scheduling based on pod priorities.
namespaces	The permissions to create namespaces and full permissions on all resources in self-managed namespaces. However, you cannot access resources in system namespaces. System namespaces refer to namespaces whose names start with kube-. Important The namespace named after the cluster ID is the system namespace of Argo. You can manage this namespace. For example, you can modify Argo workflow settings in workflow-controller-configmap.
persistentvolumes	Full permissions.
persistentvolumeclaims	Full permissions on resources in self-managed namespaces.
secretsconfigmapsserviceaccounts	Full permissions on resources in self-managed namespaces.
pods	Read permissions on resources in self-managed namespaces.
pods/logevents	Read permissions on resources in self-managed namespaces.
pods/exec	Permissions to create resources in self-managed namespaces.
Argo: workflows workflowtasksets workflowtemplates cronworkflows	Full permissions on resources in self-managed namespaces.

Configure CPU and memory requests for the containers in the workflow

Workflow clusters preferably use preemptible elastic container instances to run workflows. When preemptible elastic container instances are out of stock, pay-as-you-go elastic container instances are used. This efficiently reduces the cost of workflows. The protection period of preemptible elastic container instances is 1 hour. Make sure that each step in your workflow can be completed within 1 hour.

Preemptible elastic container instances support only containers each of which requests two or more vCPUs. If a workflow creates containers each of which requests less than two vCPUs, the workflow cannot be run on preemptible elastic container instances.

If no resource requests are configured for the containers in a workflow, each container in the workflow requests 2 vCPUs and 4 GiB of memory by default.
The system automatically overwrites workflows whose container resource requests are lower than 2 vCPUs/4 GiB with 2 vCPUs/4 GiB.
The system automatically selects matching elastic container instances for workflows whose container resource requests are equal to or higher than 2 vCPUs/4 GiB. The following table describes the supported CPU and memory requests. We recommend that you do not set the CPU request to more than eight vCPUs.
vCPU
Memory (GiB)
2
4, 8, and 16
4
4, 8, 16, and 32
8
4, 8, 16, 32, and 64

Force a pay-as-you-go elastic container instance to run the workflow

If you do not want to use a preemptible elastic container instance to run key tasks in cost-prioritized mode, you can force a pay-as-you-go elastic container instance to run the workflow.

Configure the requests and limits parameters in the Container section, as shown in the following sample code:

apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec.
metadata:
  generateName: hello-world-    # name of the workflow spec.
spec:
  entrypoint: whalesay         # invoke the whalesay template.
  templates:
    - name: whalesay              # name of the template.
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "hello world" ]
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 0.5
            memory: 1Gi

vCPU	Memory (GiB)
2	4, 8, and 16
4	4, 8, 16, and 32
8	4, 8, 16, 32, and 64