Persist workflows to a database - Container Service for Kubernetes

Workflow-related resources, such as the running status, are periodically deleted. To analyze and trace the history of workflows, you can configure a persistence policy for workload persistence in a database. This ensures that workflow logs remain accessible even after the workflow or its running pod is deleted. This topic takes ApsaraDB RDS for MySQL database as an example to demonstrate how to set up a persistence policy.

Step 1: Configure an ApsaraDB RDS for MySQL instance and create a database

Create an ApsaraDB RDS for MySQL instance.
Set up a new database in the instance created in the previous step, and configure an account.
Configure a virtual private cloud (VPC) for the instance. Ensure that the VPC matches the one used by your cluster.
Important
When configuring the IP whitelist, include the CIDR block of this VPC.

For detailed steps, see Step 1: Quickly create an RDS MySQL instance and configure the database.

After configuration, record your database username and password for subsequent use.

Note

For information about how RDS is billed, see Billable items.

Step 2: Add persistence configuration

You can add persistence-related parameters to the Argo Workflows ConfigMap to persist workflow data in a database. Data stored in the database is persistent. By default, Argo core components cannot automatically detect changes in the ConfigMap database configuration. After configuration, you must manually restart the Argo Workflow controller and Argo Server to apply the changes.

In the argo namespace of the cluster, create a Secret named argo-mysql-config to store the database username and password.

apiVersion: v1
stringData:
  username: database-username # Replace with your RDS database account.
  password: database-password # Replace with your RDS database password.
kind: Secret
metadata:
  name: argo-mysql-config
  namespace: argo
type: Opaque

Modify workflow-controller-configmap in the argo namespace. Add persistence configuration in the data field.

data:  
  persistence: |
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
      connMaxLifetime: 0s     # 0 means connections don't have a max lifetime.
    archiveTTL: 30d
    archive: true
    mysql:
      host: rm-xxx.mysql.cn-beijing.rds.aliyuncs.com
      port: 3306
      database: argo-workflow
      tableName: argo_workflows
      userNameSecret:
        name: argo-mysql-config
        key: username
      passwordSecret:
        name: argo-mysql-config
        key: password

The following table describes the key parameters used in the configuration file:

Parameter	Description
`host`	The endpoint of the ApsaraDB RDS for MySQL instance.
`database`	The name of the database that you created in the ApsaraDB RDS for MySQL instance in the preceding step.
`archive`	Specifies whether to enable workflow persistence. Set this parameter to `true`.
`archiveTTL`	Specifies the retention period for the workflows. There is no maximum value. In this example, the parameter is set to `30d`, which means that workflows are persisted in the database for 30 days.

Restart both the Argo Workflow controller and Argo Server to apply the changes.