Clone a private Git repository in a CI pipeline - Container Service for Kubernetes

Continuous Integration (CI) pipelines that are built in Distributed Cloud Container Platform for Kubernetes (ACK One) workflow clusters use BuildKit Cache and Apsara File Storage NAS (NAS) to store the Go mode cache, which greatly accelerates the pipelines. If you use a private Git repository to build a CI pipeline for a Golang project in a workflow cluster, you need to first clone the private repository in the pipeline. This topic describes how to clone a private Git repository in a CI pipeline.

Background information

For more information about the best practices for using a public Git repository to build a CI pipeline, see Create CI pipelines for Golang projects in workflow clusters.

To use a private Git repository, you need to first clone the private repository in the pipeline.

You can use the following methods to clone a private Git repository:

Method 1: Use Argo Workflows Git artifacts and the username and password
Method 2: Use Argo Workflows Git artifacts and an SSH private key
Method 3: Use the git clone command and the username and password

Store the repository credentials in the workflow cluster

Before you clone a private Git repository, you need to first run the following command to create a Secret that stores the username, password, and SSH private key of the repository in the workflow cluster:

Replace username, password, and ssh-private-key in the command with the actual values.

kubectl create secret generic git-creds --from-literal="username=${username}" --from-literal="password=${password or token}" --from-file=ssh-private-key=${ssh private key path}

# example
# kubectl create secret generic git-creds --from-literal="username=demo" --from-literal="password=ghp_GePB****************d407" --from-file=ssh-private-key=$HOME/.ssh/id_rsa

Method 1: Use Argo Workflows Git artifacts and the username and password

Clone a private Git repository and then check out the repository.

Compared with the predefined workflow template used in the Create CI pipelines for Golang projects topic, the workflow templates used by the three methods retain only the git-checkout-pr task. In the workflow template used by Method 1, the git-clone task is added and the git-clone task is configured as the dependency of the git-checkout-pr task.

You do not need to modify the Shell script in the command parameter of the git-checkout-pr task.
The artifacts parameter of the git-clone task references the username and password of the repository from the git-creds Secret.

Sample template

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: ci-git-artifact
spec:
  entrypoint: main
  volumes:
  - name: run-test
    emptyDir: {}
  - name: workdir
    persistentVolumeClaim:
      claimName: pvc-nas
  - name: docker-config
    secret:
      secretName: docker-config
  arguments:
    parameters:
    - name: repo_url
      value: ""
    - name: repo_name
      value: ""
    - name: target_branch
      value: "main"
  templates:
    - name: main
      dag:
        tasks:
          - name: git-clone
            arguments:
              artifacts:
              - name: git-repo
                path: /workdir
                git:
                  repo: "{{arguments.parameters.repo_url}}"
                  revision: main
                  usernameSecret:
                    name: git-creds
                    key: username
                  passwordSecret:
                    name: git-creds
                    key: password
                  sshPrivateKeySecret:
                    name: git-creds
                    key: ssh-private-key
            inline:
              container:
                image: golang:1.10
                command: 
                - sh
                - -c
                - |
                  cd {{workflow.parameters.repo_name}}
                  git status && ls
                workingDir: /workdir
                volumeMounts:
                - name: "workdir"
                  mountPath: /workdir
          - name: git-checkout-pr
            inline:
              container:
                image: alpine:latest
                command:
                  - sh
                  - -c
                  - |
                    set -eu
                    
                    apk --update add git
          
                    cd /workdir
                    echo "Start to Clone "{{workflow.parameters.repo_url}}
                    git -C "{{workflow.parameters.repo_name}}" pull || git clone {{workflow.parameters.repo_url}} 
                    cd {{workflow.parameters.repo_name}}
          
                    echo "Start to Checkout target branch" {{workflow.parameters.target_branch}}
                    git checkout {{workflow.parameters.target_branch}}
                    
                    echo "Get commit id" 
                    git rev-parse --short origin/{{workflow.parameters.target_branch}} > /workdir/{{workflow.parameters.repo_name}}-commitid.txt
                    commitId=$(cat /workdir/{{workflow.parameters.repo_name}}-commitid.txt)
                    echo "Commit id is got: "$commitId
                                        
                    echo "Git Clone and Checkout Complete."
                volumeMounts:
                - name: "workdir"
                  mountPath: /workdir
                resources:
                  requests:
                    memory: 1Gi
                    cpu: 1
                activeDeadlineSeconds: 1200
            depends: git-clone

Parameters for submitting the workflow

When you submit the workflow, set the workflow parameters in consistency with the CI pipeline. The following figure is an example.

Method 2: Use Argo Workflows Git artifacts and an SSH private key

Compared with Method 1, Method 2 has the following differences:

The artifacts parameter of the git-clone task references the SSH private key from the git-creds Secret.

When you submit the workflow, specify the repo_url parameter in SSH format. Example: git@github.com:ivan-cai/gitops-demo-private.git.

Sample template

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: ci-git-artifact-sshkey
spec:
  entrypoint: main
  volumes:
  - name: run-test
    emptyDir: {}
  - name: workdir
    persistentVolumeClaim:
      claimName: pvc-nas
  - name: docker-config
    secret:
      secretName: docker-config
  arguments:
    parameters:
    - name: repo_url
      value: ""
    - name: repo_name
      value: ""
    - name: target_branch
      value: "main"
  templates:
    - name: main
      dag:
        tasks:
          - name: git-clone
            arguments:
              artifacts:
              - name: git-repo
                path: /workdir
                git:
                  repo: "{{arguments.parameters.repo_url}}"
                  revision: main
                  sshPrivateKeySecret:
                    name: git-creds
                    key: ssh-private-key
            inline:
              container:
                image: golang:1.10
                command: 
                - sh
                - -c
                - |
                  cd {{workflow.parameters.repo_name}}
                  git status && ls
                workingDir: /workdir
                volumeMounts:
                - name: "workdir"
                  mountPath: /workdir
          - name: git-checkout-pr
            inline:
              container:
                image: alpine:latest
                command:
                  - sh
                  - -c
                  - |
                    set -eu
                    
                    apk --update add git
          
                    cd /workdir
                    echo "Start to Clone "{{workflow.parameters.repo_url}}
                    git -C "{{workflow.parameters.repo_name}}" pull || git clone {{workflow.parameters.repo_url}} 
                    cd {{workflow.parameters.repo_name}}
          
                    echo "Start to Checkout target branch" {{workflow.parameters.target_branch}}
                    git checkout {{workflow.parameters.target_branch}}
                    
                    echo "Get commit id" 
                    git rev-parse --short origin/{{workflow.parameters.target_branch}} > /workdir/{{workflow.parameters.repo_name}}-commitid.txt
                    commitId=$(cat /workdir/{{workflow.parameters.repo_name}}-commitid.txt)
                    echo "Commit id is got: "$commitId
                                        
                    echo "Git Clone and Checkout Complete."
                volumeMounts:
                - name: "workdir"
                  mountPath: /workdir
                resources:
                  requests:
                    memory: 1Gi
                    cpu: 1
                activeDeadlineSeconds: 1200
            depends: git-clone

Parameters for submitting the workflow

Parameters:

Note

Specify the repo_url parameter in SSH format.

Method 3: Use the git clone command and the username and password

Compared with Method 1 and 2, Method 3 does not require a directed acyclic graph (DAG) task. In addition, this method modifies the git clone command in the git-checkout-pr task and uses environment variables to reference the username and password from the git-creds Secret. Command:

git clone https://${GIT_USER}:${GIT_TOKEN}@github.com/${GITHUB_REPOSITORY}

Sample template

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: ci-git
spec:
  entrypoint: main
  volumes:
  - name: run-test
    emptyDir: {}
  - name: workdir
    persistentVolumeClaim:
      claimName: pvc-nas
  - name: docker-config
    secret:
      secretName: docker-config
  arguments:
    parameters:
    - name: repo_url
      value: ""
    - name: repo_name
      value: ""
    - name: target_branch
      value: "main"
  templates:
    - name: main
      dag:
        tasks:
          - name: git-checkout-pr
            inline:
              container:
                image: alpine:latest
                env:
                - name: GIT_USER
                  valueFrom:
                    secretKeyRef:
                      name: git-creds
                      key: username
                - name: GIT_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: git-creds
                      key: password
                command:
                  - sh
                  - -c
                  - |
                    set -eu
                    
                    apk --update add git
          
                    cd /workdir
                    echo "Start to Clone "{{workflow.parameters.repo_url}}
                    git -C "{{workflow.parameters.repo_name}}" pull || git clone https://$GIT_USER:$GIT_TOKEN@{{workflow.parameters.repo_url}} 
                    cd {{workflow.parameters.repo_name}}
          
                    echo "Start to Checkout target branch" {{workflow.parameters.target_branch}}
                    git checkout {{workflow.parameters.target_branch}}
                    
                    echo "Get commit id" 
                    git rev-parse --short origin/{{workflow.parameters.target_branch}} > /workdir/{{workflow.parameters.repo_name}}-commitid.txt
                    commitId=$(cat /workdir/{{workflow.parameters.repo_name}}-commitid.txt)
                    echo "Commit id is got: "$commitId
                                        
                    echo "Git Clone and Checkout Complete."
                volumeMounts:
                - name: "workdir"
                  mountPath: /workdir
                resources:
                  requests:
                    memory: 1Gi
                    cpu: 1
                activeDeadlineSeconds: 1200

Parameters for submitting the workflow

Parameters:

Note

The value of the repo_url parameter cannot be prefixed with repo_url.

References

For more information about the best practices for using a public Git repository to build a CI pipeline, see Create CI pipelines for Golang projects in workflow clusters.