Best Practices of ACK One GitOps

This article describes how to use ACK One GitOps to achieve application distribution in multiple clusters, version management, automatic update, permission control, and CI pipeline integration.

By Yu Zhuang and Liusheng

Alibaba Cloud ACK One is a distributed cloud container platform for scenarios (such as hybrid clouds, multi-clusters, and distributed computing). It allows users to manage the Kubernetes clusters that are on and at the edge of Alibaba Cloud and deployed in customer data centers and other clouds. It can simplify the cluster management interface. ACK One can associate and manage Kubernetes clusters of various types and provide a unified cluster control plane to implement unified application distribution, traffic management, O&M management, security management, and GitOps capabilities across clusters. This article describes how to use ACK One GitOps to achieve application distribution in multiple clusters, version management, automatic update, permission control, and CI pipeline integration. This article will help you to get started with GitOps quickly.

GitOps Overview

The core of GitOps application distribution is to use a Git repository to manage the deployment templates of applications, deploy applications to a specified Kubernetes cluster, and use the Git repository as the only source for application deployment. The application status on the Kubernetes cluster is continuously adjusted to be consistent with the expected status in the Git repository.

Advantages of GitOps

High Acceptability

Git tends to be accepted by developers and is easy to integrate, with no additional learning costs.

Strong Reliability

As the sole source for application deployment, the Git repository provides versioning, fast rollback, and auditing capabilities.

High Security

Developers do not need any Kubernetes cluster permissions to use GitOps. They only need Git repository permissions.

Continuous Application Deployment

The application status in a Kubernetes cluster and Git repository is automatically synchronized.

In its prediction of 2023 Cloud Native, CNCF pointed out that GitOps has matured and entered a period of stable productivity. Argo (the CNCF GitOps open-source project) graduated from the CNCF Incubator in December 2022 [1], marking the stability and maturity of the Argo project. More users are using the Argo project to realize GitOps application distribution.

ArgoCD is a subproject of Argo. It is a continuous delivery tool that follows the declarative GitOps concept. It connects Git and Helm repositories to provide powerful visual pages. ArgoCD runs in the Kubernetes cluster as a controller, continuously monitoring the actual state of the application and keeping it in sync with the desired state declared in the Git repository.

ACK One GitOps

The multi-cluster management master instance of ACK One manages the open-source ArgoCD project to implement GitOps’ continuous delivery of applications. It also integrates the multi-cluster capabilities of ACK One to implement GitOps’ continuous delivery to meet the requirements of high-availability deployment of applications and multi-cluster distribution of system components.

Advantages of ACK One GitOps

Manages Open-Source ArgoCD: It provides an ArgoCD native CLI and UI experience.
Dominates the Domain Name of the ArgoCD Console: It integrates Alibaba Cloud account SSO and supports multi-user permissions on ArgoCD.
Provides Multi-Cluster Distribution: The sub-cluster associated with ACK One is automatically added to ArgoCD and becomes the target cluster for GitOps application distribution.
Supports ArgoCD ApplicationSet to improve the experience of multi-cluster application distribution
Out-of-the-box and O&M-free

You can use ACK One GitOps to obtain the GitOps capabilities of ArgoCD, which is graduated in CNCF. The GitOps capabilities include the native ArgoCD console and CLI integrated with your Alibaba Cloud account, multi-user permission management, and multi-cluster distribution. Therefore, ACK One GitOps can help you quickly build a multi-cluster distribution pipeline.

Prerequisites

The multi-cluster management feature is enabled. Please see enable multi-cluster management [2] for more actions.
Multiple associated clusters have been added to the master instance. In this example, ackpro-cluster1 and ackpro-cluster2 are used. Please see add an associated cluster [3] for more actions.
The kubeconfig file of the master instance is obtained from the ACK One console and is connected to the master instance using KubeConfig.
Install the latest version of Alibaba Cloud CLI [4] and configure credentials [5]. If you use a sub-account, grant AliyunAdcpFullAccess permission to the sub-account in RAM.
Visit the ArgoCD [6] to download and install the latest ArgoCD CLI
The AMC command-line tool is installed. Please see the AMC command line help [7] for more actions.

Deploy an Application Using the ACK One GitOps Console

Step 1: Turn on ACK One GitOps with One Click (ArgoCD)

Run the following command to enable GitOps and replace clusterid with the ID of your master instance:

aliyun adcp UpdateHubClusterFeature --ClusterId <clusterid> --ArgoCDEnabled true

After you run the preceding command, ACK One will create an ECI in your account to run the ArgoCD service, create an internal SLB instance to expose the ArgoCD server service, generate an ArgoCD server domain name, and resolve the domain name to the internal IP address of the SLB instance.

ArgoCD Server domain name format: https://argocd.<ackone cluster id>.<region>.alicontainer.com

You can run the following command to view the opening result of GitOps:

# View the ArgoCD pod
kubectl get pod -n argocd
NAME                             READY   STATUS    RESTARTS   AGE
argocd-server-76c9b99f47-c8hwn   8/8     Running   0          41s

# View the domain name of the ArgoCD server
kubectl get cm argocd-cm -n argocd -o json | jq .data.url
"https://argocd.xxx.cn-xxx.alicontainer.com"

Step 2: Log on to the Console of ACK One GitOps (ArgoCD) and Deploy the Application

(1) You can use a browser to access the ArgoCD Server domain name obtained in Step 1 and click LOGIN VIA ALIYUN to log on to the ArgoCD console:

(2) Add the release source Git Repo:

1) On the Settings > Repositories page, click + CONNECT REPO

2) Enter the Git Repo URL: https://github.com/AliyunContainerService/gitops-demo.git

3) Click CONNECT

4) After the Git Repo is added, the CONNECTION STATUS of Git will display Successful:

(3) Create an ArgoCD Application and publish the application to the target cluster

1) On the Applications page, click + NEW APP. The configuration is listed below. The Application pulls the Helm Chart from GitRepo and deploys it to the default namespace of the target cluster.

2) After the creation, you can check the status of the echo-server-app application on the ArgoCD Applications page.

3) Click SYNC below the target application to synchronize the application to the target cluster.

If Healthy and Synced are displayed on the right side of Status, the echo-server-app application is synchronized. Click echo-server to view the application details. ArgoCD will display the topology and status of the application-related Kubernetes resources.

Step 3: View the Actual Deployment of the Application

Connect to the Kubeconfig of the master instance and run the following command. The master instance will forward the request to a specific cluster.

// View the sub-clusters associated with the master instance.
kubectl amc get managedcluster
Name       Alias             HubAccepted
cdxxx      ackpro-cluster2   true
cexxx ackpro-cluster1 true // Deploy the target cluster on GitOps.

// Use the-m parameter to specify the cluster and view the Deployment, Service, and Pod resources in the default command space. 
kubectl amc get all -n default -m cexxx
Run on ManagedCluster cexxx (ackpro-cluster1)
NAME                               READY   STATUS    RESTARTS   AGE
pod/echo-server-5cf54bdbcb-jv58k   1/1     Running   0          59m

NAME                  TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)          AGE
service/echo-server   LoadBalancer   192.xxx.xxx.xxx   39.xxx.xxx.xxx   8080:31769/TCP   59m        21d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echo-server   1/1     1            1           59m

// Access the External-IP of Loadbalance Service to verify the application execution. The displayed version is v1.0.
curl http://39.xxx.xxx.xxx:8080/version
"Hello Echo Server v1.0"

Step 4: Upgrade the Application Version

(1) Update the image tag version in Git from v1.0 to v2.0:

(2) The ArgoCD Application displays OutOfSync because GitRepo has changed.

Click the yellow OutOfSync icon to see the detailed differences:

(3) Click Sync to initiate synchronization. After the synchronization is completed, the Pod Image Tag becomes v2.0, which is consistent with expectations.

Access the Service to view the actual effect. The version is changed to v2.0.

curl http://39.98.55.113:8080/version
"Hello Echo Server v2.0"

Step 5: Roll Back the Application Version

You can find that after version v2.0 is released, ArgoCD retains the previous replicaset for a quick rollback of the previous version.

// On the master instance of ACK One, use the amc -m parameter to view the resource status of the specified sub-cluster.
kubectl amc get replicaset  -n default -m cexxx
Run on ManagedCluster cexxx (ackpro-cluster1)
NAME                     DESIRED   CURRENT   READY   AGE
echo-server-55664c4677 1 1 1 26m // Version v2.0
echo-server-5cf54bdbcb 0 0 0 109m // version v1.0

Now, let's assume that v2.0 has a problem and wants to roll back to v1.0. Click HISTORY AND ROLLBACK of the application to view all historical versions. You can select the corresponding version to roll back. This example selects v1.0 and clicks Rollback.

Check the rollback status. The Pod Image Tag has been rolled back to v1.0. Compared with GitRepo, the application status changes to OutOfSync. After the problem is fixed, click Sync to synchronize. ArgoCD also supports automatic synchronization.

Deploy an Application Using ArgoCD CLI

After ACK One GitOps is enabled, you can use ArgoCD CLI to achieve GitOps application management.

1. Log on to ArgoCD Server. The CLI will open a browser to complete the SSO login.

// Log on to the argocd server.
argocd login argocd.<ackone cluster id>.<region>.alicontainer.com --sso
Opening browser for authentication
Performing authorization_code flow login: https://signin.aliyun.com/oauth2/v1/auth?xxx
Authentication successful
'root' logged in successfully
Context 'argocd.<ackone cluster id>.<region>.alicontainer.com' updated

2. Add and view Git Repo:

// Add git repo.
argocd repo add https://github.com/AliyunContainerService/gitops-demo.git --name echo-server
Repository 'https://github.com/AliyunContainerService/gitops-demo.git' added

// View git repo.
argocd repo list
TYPE  NAME  REPO                                                       INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git         https://github.com/AliyunContainerService/gitops-demo.git  false     false  false  false  Successful           default

3. View the cluster:

argocd cluster list
SERVER                          NAME                    VERSION  STATUS   MESSAGE                                                  PROJECT
https://10.xxx.xxx.xxx:xxx      cexxx-ackpro-cluster1            Unknown  Cluster has no applications and is not being monitored.
https://10.xxx.xxx.xxx:xxx      cdxxx-ackpro-cluster2            Unknown  Cluster has no applications and is not being monitored.
https://kubernetes.default.svc  in-cluster                       Unknown  Cluster has no applications and is not being monitored.

You can manage the master instance through ACK One multiple clusters and associate new clusters. ACK One will automatically synchronize the clusters to the ArgoCD.

4. Create and synchronize an application:

// Create an application.
argocd app create echo-server --repo https://github.com/AliyunContainerService/gitops-demo.git --path manifests/helm --revision one-demo --dest-namespace default --dest-server https://10.0.47.122:6443
application 'echo-server' created

// Synchronize the application, pull the application from Git, and deploy it to the target cluster.
argocd app sync echo-server

Manage ACK One GitOps User Permissions

ArgoCD provides powerful user rights management. Based on the following policy templates, you can flexibly configure a user to have certain permissions on a resource. Supported resource types include applications, clusters, and repositories. You can specify the project to which the resource belongs. Permission actions include create, update, delete, and get.

p, <user/group>, <resource>, <action>, <project>/<object>

Let's use a sample to demonstrate user permission settings.

Define an ArgoCD administrator with all permissions on all resources, so an ArgoCD Application can be created to deploy the application. Define a developer user that can submit changes to GitRepo but can only view ArgoCD resources. For example, the developer user can view the deployment results of ArgoCD applications.

The corresponding configuration is listed below. The administrator edits the ArgoCD configmap argocd-rbac-cm with the kubectl edit cm argocd-rbac-cm -n argocd command.

data:
  policy.csv: |
    g, "26xxx", role:admin     //The Alibaba Cloud sub-account, which is authorized as the administrator of ArgoCD.
    g, "27xxx", role:readonly     // Alibaba Cloud sub-account, which is authorized as a read-only ArgoCD user.

role:admin and role:readonly are the names of the permissions embedded in ArgoCD. They correspond to the administrator permission and the read-only permission, respectively.

You can refer to the following specific definitions: https://github.com/argoproj/argo-cd/blob/master/assets/builtin-policy.csv

You can also create permissions suitable for specific business scenarios and authorize specific users. In addition, we recommend reading the ACK One GitOps security model for detailed user permission examples and descriptions.

Use ArgoCD Applicationset to Implement Multi-Cluster GitOps Deployment

Applicationset is built by ArgoCD for multi-cluster scenarios. You can use Applicationset to produce multiple Application instances to support different GitRepo and different target clusters. Following the example below, we will read the deployment Yaml from different GitRepo directories by defining Applicationset and completing the application deployment in two clusters.

1. As different clusters contain different image versions and service configurations, we use two directories in GitRepo to store two cluster deployment Yaml files corresponding to the production and staging environments.

└── manifests
    └── directory
        ├── production
        │   ├── deployment.yaml  //The image tag is v1.0.
        │   └── service.yaml
        └── staging
            ├── deployment.yaml  //The image tag is v2.0.
            └── service.yaml

2. Create an Applicationset using the following command:

// Create a namespace in the master instance. The namespace is automatically synchronized to the subcluster.
kubectl create ns one-demo

// Create an Applicationset.
cat << EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: echo-server
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: production            //Cluster 1
        url: https://10.xxx.xxx.xxx:6xxx
      - cluster: staging               //Cluster 2
        url: https://10.xxx.xxx.xxx:6xx
  template:
    metadata:
      name: '{{cluster}}-gitops-demo'
    spec:
      project: default
      source:
        repoURL: https://github.com/AliyunContainerService/gitops-demo.git
        targetRevision: one-demo
        path: manifests/directory/{{cluster}}  // The name of the corresponding cluster directory
      destination:
        server: '{{url}}'                      // Reference the cluster url
        namespace: one-demo
EOF

3. View the application deployment results. You can see that two applications have been created. The deployment Yaml is obtained from different GitRepo directories and deployed to different target clusters.

Click Sync. You can obtain the actual running status of Kubernetes resources by running kubectl in the master instance of ACK One.

kubectl amc get deployment -m all -n one-demo //amc -m all will get the resource status of all associated clusters.
Run on ManagedCluster cdcxxx (ackpro-cluster2)
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
echo-server   1/1     1            1           18m
Run on ManagedCluster cedxxx (ackpro-cluster1)
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
echo-server   1/1     1            1           18m

kubectl amc get svc -m all -n one-demo //amc -m all will get the resource status of all associated clusters.
Run on ManagedCluster cdcxxx (ackpro-cluster2)
NAME          TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)    AGE
echo-server   ClusterIP      192.xxx.xxx.xxx   <none>         80/TCP     18m
Run on ManagedCluster cedxxx (ackpro-cluster1)
NAME          TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)    AGE
echo-server   ClusterIP      192.xxx.xxx.xxx   <none>         80/TCP     18m

Use Image Updater to Implement CI Pipeline Integration

ACK One GitOps solves the problem of application deployment, but the development team needs E2E, including code submission, image building, and automatic deployment of the whole process fully automatic solution. The automatic solution is particularly important for the development and test environment, which can help improve the efficiency of research and development.

ACK One GitOps integrates open-source ArgoCD Image Updater projects and extends support for the ACR image repository. After the CI pipeline builds an image and publishes it to the ACK image repository, the Image Updater can monitor the image changes and automatically update the image tags in the deployment Git Repo. Then, the ArgoCD Application initiates Sync to deploy the application after monitoring changes in Git Repo. Through Image Updater, ACK One GitOps can be linked with CI pipelines to provide CI + CD automation solutions for E2E whole process.

Please visit this article [link to other article in this batch](for more information about the description and usage of Image Updater.