Building Automated CI with Argo Workflows and EventBridge

ACK One Serverless Argo Workflows and EventBridge deliver your applications simply, quickly, efficiently, and cost-effectively, enabling an automated ...

By Jing Cai

Argo Workflows is an open-source, cloud-native workflow engine for orchestrating jobs on Kubernetes. It easily automates and manages complex workflows on Kubernetes. It is also suitable for various scenarios such as scheduled tasks, machine learning, ETL and data analysis, model training, data flow pipeline, and CI/CD.

Kubernetes Jobs only provides basic task execution without defining step dependencies and order, workflow templates, a visual interface, or workflow-level error handling. This makes it inadequate for business scenarios such as batch processing, data processing, scientific computing, and continuous integration.

As a graduate project of CNCF, Argo Workflows is used in many scenarios, with Continuous Integration (CI) being one of its key application areas.

CI is an important part of the software development lifecycle, and a good CI system can greatly improve your delivery efficiency. Based on Alibaba Cloud EventBridge and ACK One Serverless Argo Workflows, you can build an event-driven automated CI Pipeline to deliver your applications easily, quickly, efficiently, and cost-effectively, enabling automatic builds and deliveries upon committing code.

EventBridge is a serverless event bus service provided by Alibaba Cloud. You can connect Alibaba Cloud services, custom applications, and SaaS applications to EventBridge in a standardized and centralized manner. It routes events among these applications using the standardized CloudEvents 1.0 protocol, helping you build loosely coupled and distributed event-driven architectures. Using EventBridge as an event-driven engine offers the following advantages:

1. Availability: It enables high-throughput event publishing and consumption, auto-scaling to handle sudden traffic spikes, disaster recovery deployment within the same region with strong data reliability, and multi-region deployment with low network latency.

2. Usability: It supports fully managed service with no need for deployment, operation, or maintenance, a wide range of event sources, including official Alibaba Cloud event sources and custom event sources, and seamless migration to the cloud with full compatibility with CloudEvents 1.0.

3. Security: It supports granting permissions from Alibaba Cloud accounts to RAM users, permissions across accounts, and different access permissions and HTTP/HTTPS protocols to different RAM users.

Event-Driven Automated CI Pipeline

ACK One fully manages Argo Workflows, which improves its performance, stability, observability, and O&M (Operation and Maintenance) capabilities. This article will introduce how to build an event-driven automated CI Pipeline based on EventBridge and ACK One Serverless Argo Workflows, achieving a CI system driven by Git events for building and pushing Docker Image and running pre-build tests.

An event-driven automated CI Pipeline consists of two parts:

A Git event trigger. An event is triggered when code is committed to the Git repository.
Pre-build testing. The CI pipeline performs a test before the CI system builds the Docker image and pushes the image to the image repository.

After the image is built, you can use the CD system (such as ACK One GitOps) to synchronize the new Image tag to the Kubernetes cluster.

EventBridge contains event sources, event buses, event rules, and event targets:

• Event Source: Publish event messages generated by applications such as Alibaba Cloud services and custom applications to an event bus. In this article, the event source is connected to GitHub.

• Event Bus: Store received event messages and route them to event targets according to event rules.

• Event Target: Consumers of the events. In this article, the event target is the ACK One Serverless Argo Workflows.

CI Pipeline based on the workflow cluster primarily involves three steps: Clone and switch branches of the Git code; run test cases for Go projects; and build and push the image to ACR EE. BuildKit Cache is used to accelerate image builds.

The main steps for CI driven by Git events are as follows:

A user commits code to the Git repository.
EventBridge, based on configured rules, is triggered to commit CI workflows to the ACK One workflow cluster when Git events occur.
ACK One creates a CI pipeline in the workflow cluster, builds a Docker image, and then pushes the image to the image repository. You can use GitOps to automatically synchronize image changes to the ACK cluster.

Prepare the Argo Workflows and CI Environment

Create a workflow cluster, ACR EE, and NAS file system.
Prepare the ACR EE access credential and NAS volumes required by the CI workflow in the workflow cluster. For more details, please refer to Steps 1 and 2 in the link for the CI Pipeline based on workflow clusters below.

(Note that the namespace needs to match the Workflow resource.)

https://www.alibabacloud.com/help/en/ack/distributed-cloud-container-platform-for-kubernetes/use-cases/building-a-ci-pipeline-of-golang-project-based-on-workflow-cluster

The Apps Code Repo example is the echo-server project. If you use this sample code, you need to fork it into your own Git repository. If it is a private Git repository, you can refer to Clone Private Git Repository in CI Pipeline to modify the WorkflowTemplate accordingly.

Use EventBridge to Implement the CI Pipeline Driven by Git Events

Below are the configuration steps for the CI Pipeline driven by GIt events for the specified branch (release-v1) of the echo-server project.

1) Activate EventBridge and grant permissions.

https://www.alibabacloud.com/help/en/eventbridge/getting-started/activate-eventbridge-and-grant-permissions-to-a-ram-user

2) Manage custom event buses.

https://www.alibabacloud.com/help/en/eventbridge/user-guide/manage-custom-event-buses#section-sfl-pcs-6rh

3) Integrate Github.

https://www.alibabacloud.com/help/en/eventbridge/use-cases/integrate-github

4) Edit event rules (optional).

• Event pattern: The following settings only trigger changes from the release-v1 branch.

{
    "source": [
        "github.event"
    ],
    "data": {
        "body": {
            "ref": [
                "refs/heads/release-v1"
            ]
        }
    }
}

5) Event target:

• Service Type: Select Container Service for Kubernetes.

• KubeConfig File: Enter the KubeConfig file of the ACK One workflow cluster.

• YAML Configuration: Template.

• Variables: Add the workflowName parameter, configured as the event id as follows.

{
    "workflowName": "$.id"
}

6) Template: Fill in the Workflow CI CR.

• You need to specify name and namespace instead of generateName, and declare the default namespace.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: ci-go-v1-eb-${workflowName}
  namespace: default
  labels:
    workflows.argoproj.io/workflow-template: ackone-ci
spec:
  arguments:
    parameters:
    - name: repo_url
      value: https://github.com/ivan-cai/echo-server.git
    - name: repo_name
      value: echo-server
    - name: target_branch
      value: release-v1
    - name: container_image
      value: "test-registry.cn-hongkong.cr.aliyuncs.com/acs/echo-server"
    - name: container_tag
      value: "v1.0.0"
    - name: dockerfile
      value: ./Dockerfile
    - name: enable_suffix_commitid
      value: "true"
    - name: enable_test
      value: "true"
  workflowTemplateRef:
    name: ci-go-v1
    clusterScope: true

Commit Code to Automatically Build and Push Images

1. Commit the GitHub code to trigger the automation process (according to the preceding configuration, you need to modify the release-v1 branch code to trigger the process).

You can view event traces in Event Tracking.

2. View the execution topology of the newly created Workflow in the ACK One workflow cluster or the workflow console.

3. View the pushed images in ACR EE.

Summary

As a fully managed Argo workflow service, ACK One Serverless Argo Workflows helps you implement a larger-scale, faster-running, and lower-cost CI Pipeline. Combined with EventBridge, which offers advantages in availability, usability, and security, you can deliver your applications simply, quickly, efficiently, and cost-effectively, enabling an automated CI system that triggers builds and deliveries upon committing code.

Community

Building Automated CI with Argo Workflows and EventBridge

Event-Driven Automated CI Pipeline

Prepare the Argo Workflows and CI Environment

Use EventBridge to Implement the CI Pipeline Driven by Git Events

Commit Code to Automatically Build and Push Images

Summary

Read previous post:

Read next post:

Alibaba Container Service

You may also like

Comments

Alibaba Container Service

Related Products

Serverless Workflow

Serverless Application Engine

Alibaba Cloud Flow

ACK One