Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Simplify Your Workflow: Advanced Pipeline YAML Reduces Code Repetition and Flexibly Arranges Multitasking

Simplify Your Workflow: Advanced Pipeline YAML Reduces Code Repetition and Flexibly Arranges Multitasking

This article introduces the template syntax feature in Alibaba Cloud DevOps Pipeline (Flow) that allows for dynamic rendering of pipeline YAML files using template languages.

By Muyan

Do you encounter the following problems when configuring pipelines in YAML?

• In scenarios where batch execution of similar tasks is required within a single pipeline, multiple similar Jobs need to be defined in the YAML file. The more Jobs there are, the longer the pipeline YAML configuration becomes, resulting in more repeated code and poor code reusability and readability.

• When administrators manage multiple pipelines with similar technology stacks and development processes, but with minor differences in build and deployment parameters, each pipeline requires separate modification of the differing parameters, resulting in high configuration costs. When each pipeline is configured independently, unified management becomes difficult, leading to poor maintainability.

• Pipeline YAML files are static; the task execution logic is fixed when the pipeline is saved. However, in some development scenarios, the execution flow of the pipeline needs to be determined at runtime, making it impossible to configure in advance. In this scenario, the common pipeline YAML cannot meet the requirements.

To address this, Alibaba Cloud DevOps Pipeline (Flow) introduces template syntax in pipeline YAML files, allowing the use of template languages to dynamically render pipeline YAML. This supports batch configuration of multiple similar or identical Jobs, as well as on-demand dynamic generation of Jobs, helping to reduce redundant code and enabling flexible task orchestration.

1. What is the Template Syntax?

A template is a template language used for defining and rendering text. It can incorporate variables, conditional statements, loops, and other constructs, allowing YAML files to generate diverse configuration outputs based on context or external data sources. This enables dynamic rendering of pipeline YAML at runtime.

Alibaba Cloud DevOps Pipeline introduces the template engine by specifying template=true in the first line comment of the pipeline YAML to enable template mode. It supports using {{ }} to define template language, following the native syntax of go template. It also allows the use of variables defined with variables to be used as parameters for rendering the pipeline. Typical application scenarios are as follows:

2. Core Usage Scenarios of the Template Syntax

2.1 Scenario 1: Test the Compatibility of Multiple Operating Systems and SDK Versions

In certain compatibility testing scenarios, such as when you need to test your code on n different operating systems and m different SDK versions, you would need to define n m Jobs in your pipeline, with each Job having the same execution logic. In this situation, the pipeline YAML would become very long and contain a large amount of repetitive code, making it difficult to maintain. Additionally, any modification to the execution logic of a Job would need to be made n m times.

Furthermore, by introducing template syntax, compatibility scenarios can be extracted into variables and the range syntax can be used to loop through scenarios, batch-generating multiple Jobs and significantly reducing the amount of YAML code. When the execution logic of a Job changes, it only needs to be modified in one place.

For example, in the following code, we loop through 2 operating systems ("Linux" and "Windows") and 3 JDK versions ("10", "11", and "17"), using template range syntax to easily generate 6 Jobs with the same logic.

# template=true
variables:
  - key: osList
    type: Object
    value: ["linux", "windows"]
  - key: jdkVersionList
    type: Object
    value: ["10", "11", "17"]

stages:
  build_stage:
    name: compatibility testing
    jobs:                             # Two-layer loop to generate 2*3 Jobs
      {{ range $os := .osList}}
        {{ range $jdk := $.jdkVersionList}}
        {{ $os }}_JDK{{ $jdk }}_job:
                   name: Test-{{ $os }}-JDK{{ $jdk }}
                   my_step:
                     name: command execution
                     step: Command
                     with:
                       run: |
                         echo 'test on {{ $os }}-JDK{{ $jdk }}"
        {{ end }}
        {{ end }}

The running result of the pipeline is as follows:

1

2.2 Scenario 2: Dynamic On-demand Build and Deployment for Multiple Applications

In a multi-application joint release scenario under a single system, a change in business requirements will only affect some of the applications within the system. Each release deployment only needs to trigger the build and deployment of the affected applications. In this case, the static YAML file configuration mode cannot meet the requirements of scenarios that need to dynamically generate application building and job deployment.

After introducing template syntax, applications can be extracted into variables, and the range syntax can be used to traverse configurations in a loop manner. Based on the list of applications provided at runtime, multiple build and deployment tasks for the applications can be dynamically generated as needed.

As shown in the example below, we have configured multiple application code sources, multiple application build tasks, and multiple application deployment tasks. The number of applications to be deployed can be dynamically determined based on the runtime environment variable appnames.

# template=true
variables:
  - key: appnames
    type: Object
    value: ["app1", "app2", "app3"]

sources:
  {{ range $app := .appnames }}
  repo_{{ $app }}:
    type: codeup
    name: the name of the code source-{{ $app }}
    endpoint: https://yunxiao-test.devops.aliyun.com/codeup/07880db8-fd8d-4769-81e5-04093aaf7b2b/c{{ $app }}.git
    branch: master
    certificate:
      type: serviceConnection
      serviceConnection: wwnbrqpihykbiko4
  {{ end }}

defaultWorkspace: repo_app1

stages:
  build_stage:
    name: build stage
    jobs:
      {{ range $app := .appnames }}
      build_job_{{ $app }}:
        name: build task-{{ $app }}
        sourceOption: ['repo_{{ $app }}']
        steps:
          build_{{ $app }}:
            step: Command
            name: build-{{ $app }}
            with:
              run: "echo start build {{ $app }}\n"
      {{ end }}
  deploy_stage:
    name: deployment stage
    jobs:
      {{ range $app := .appnames }}
      deploy_job_{{ $app }}:
        name: deployment task-{{ $app }}
        needs: build_stage.build_job_{{ $app }}
        steps:
          build_{{ $app }}:
            step: Command
            name: deployment-{{ $app }}
            with:
              run: "echo start deploy {{ $app }}\n"
      {{ end }}

The pipeline operation result is as follows:

2

3. How to Use Template Syntax in the Alibaba Cloud DevOps Pipeline

1) Visit the YAML editing page of the pipeline and switch to the template mode by commenting # template=true in the first line.

3

2) After you switch to the template mode, you can use the {{ }}template language to define pipelines.

• Use environment variables defined in variables as rendering parameters: Use environment variables defined in variables as rendering parameters of the template. Multiple types of environment variables are supported: String, Number, Boolean, and Object. You can overwrite the run time environment variable with the same name.

• Follow the go template native syntax: The Alibaba Cloud DevOps Pipeline template mode follows the native go template syntax. The common syntax is as follows:

{{/* a comment */}}  // Comment
{{pipeline}}         // Reference
{{if pipeline}} T1 {{end}}   // Condition judgment
{{if pipeline}} T1 {{else}} T0 {{end}}
{{if pipeline}} T1 {{else if pipeline}} T0 {{end}}
{{range pipeline}} T1 {{end}}          // Loop
{{range pipeline}} T1 {{else}} T0 {{end}}
{{break}}
{{continue}}

• In addition, the Alibaba Cloud DevOps Pipeline template syntax supports the following extension functions to meet more flexible orchestration scenarios.

# add: The integer addition function. The parameter receives two or more integers.
{{ add 1 2 }} // Sample return: 3
{{ add 1 2 2 }} // Sample return: 5

# addf: The floating-point number addition function. The parameter receives two or more numbers.
{{ add 1.2 2.3 }} // Sample return: 3.5
{{ add 1.2 2.3 5 }} // Sample return: 8.5

# replace: The string replacement function. It receives three parameters: the source string, the string to be replaced, and the replaced string.
{{ "Hallo World" | replace "a" "e" }}  // Sample return: Hello World
{{ "I Am Henry VIII" | replace " " "-" }}  // Sample return: I-Am-Henry-VIII

3) Configure environment variables for the pipeline run time to dynamically render YAML before the pipeline runs.

4

4) Click Preview Mode to check whether the pipeline meets expectations by using variables for pre-rendering.

5

5) After confirming that it is correct, save it and trigger the pipeline to run. You can modify run time variables as needed to dynamically render pipeline YAML based on run time environment variables and dynamically generate Jobs. If the JDK version is modified to ["17","21"] and the operating system remains ["Linux", "Windows"], four Jobs are dynamically generated.

6
7

DevOps continuous delivery cloud native Developer Tools YAML DevOps Flow Alibaba Cloud Flow Alibaba Cloud DevOps Pipeline Pipeline YAML Template Syntax
0 1 0
Share on

Read previous post:

Design and Build FinOps Processes, Teams, Systems, and Goals

Read next post:

ApsaraMQ Copilot for RocketMQ: A Health Manager of Message Data Integration Link

Alibaba Cloud Native

199 posts | 12 followers

You may also like

Comments

Alibaba Cloud Native

199 posts | 12 followers

Related Products

More Posts by Alibaba Cloud Native

See All