Use KMS to encrypt secrets in an ACK Edge cluster - Container Service for Kubernetes

In ACK Edge Pro clusters, you can use the keys created in Alibaba Cloud Key Management Service (KMS) to encrypt Kubernetes secrets. This enhances the protection of encrypted data. This topic describes how to use keys that are managed in KMS to encrypt Kubernetes secrets in an ACK Edge Pro cluster.

Prerequisites

Item

Description

KMS key

A KMS key is created in the KMS console. The key belongs to the region where your ACK Pro cluster resides.

ACK Pro clusters support default keys, software-protected keys, and hardware-protected keys. For more information about the key management feature of KMS, see Getting started with keys. For more information about KMS billing, see Billing.

Important

After you enable Secret encryption, do not use the KMS API or the KMS console to disable or delete the key that is used to encrypt and decrypt Secrets. Otherwise, the API server becomes unavailable and cannot retrieve Secrets or service account objects. As a result, service interruptions occur.

Authorization

The following requirements must be met:

If you use an Alibaba Cloud account, the account must be authorized to assume the AliyunCSManagedSecurityRole role. Otherwise, the ACK console prompts you to perform the authorization when you enable Secret encryption. You can follow the instructions in the console to complete the authorization or go to the Cloud Resource Access Authorization page and complete the authorization.
If you use a Resource Access Management (RAM) user or RAM role:
- The RAM user or RAM role must be granted administrator or O&M engineer permissions on the ACK Pro cluster. For more information, see Grant RBAC permissions to RAM users or RAM roles.
- The RAM user or RAM role must be granted the AliyunKMSCryptoAdminAccess permission. For more information, see Grant permissions to a RAM user or RAM role.

Overview of secret encryption

Kubernetes Secrets are used to store and manage sensitive data, such as the passwords of applications, Transport Layer Security (TLS) certificates, and credentials to download Docker images. Kubernetes stores Secrets in etcd of a cluster. For more information, see Secrets.

You can use the keys created in KMS to encrypt secrets in an ACK Edge Pro cluster. The KMS provider mechanism of Kubernetes is used during encryption. A KMS provider uses envelope encryption to encrypt and decrypt secrets that are stored in etcd. Procedures for secret encryption and decryption:

When you use a Kubernetes secret to store a password, the API server generates a random data encryption key (DEK) to encrypt the secret. Then, the API server sends the DEK to KMS. KMS uses the specified key to encrypt the DEK and returns the encrypted DEK to the API server. The API server then stores the encrypted secret and DEK in etcd.
When you decrypt the Kubernetes secret, the system calls the Decrypt operation of KMS to decrypt the DEK first. Then, the system uses the plaintext DEK to decrypt the Kubernetes secret and returns the decrypted Secret.

For more information, see The KMS provider and Use envelope encryption.

Enable the secret encryption feature for an ACK Edge Pro cluster

Enable the secret encryption feature when you create a new cluster

Log on to the ACK console. In the left-side navigation pane, click Cluster.
In the upper-right corner of the Clusters page, click Create Kubernetes Cluster.
On the Create Cluster page, click the ACK Edge tab. In the lower part of the tab, click Show Advanced Options. Find the Secret Encryption parameter and select Select Key. Then, select the key ID from the drop-down list.
If you have not created a key in KMS, click create keys to go to the KMS console to create one. For more information, see Create a CMK.
For more information about other configurations of ACK Edge Pro cluster, see Create an ACK Edge cluster.
Log on to the ActionTrail console. In the left-side navigation pane, choose Events > Event Query. On the Event Query page, if the aliyuncsmanagedsecurityrole system role is used to encrypt and decrypt event logs, the secret encryption feature is enabled for the cluster.

Enable the secret encryption feature for an existing cluster

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the ACK Pro cluster that you want to manage. On the details page of the cluster, click the Basic Information tab. In the Basic Information section, turn on Secret Encryption.
If this is the first time that you enable Secret encryption, follow the instructions and click Go to RAM console to navigate to the Cloud Resource Access Authorization page. Then, click Confirm Authorization Policy.
Note
- If you want to enable Secret encryption, make sure that the RAM user or RAM role that you use is assigned one of the following RBAC roles: the administrator role and the O&M engineer role. For more information, see Grant RBAC permissions to RAM users or RAM roles.
- If you want to assign the aliyuncsmanagedsecurityrole role, make sure that you log on to the ACK console with an Alibaba Cloud account or a RAM user or RAM role that has the RAM management permissions.
In the Secret Encryption dialog box, select an existing key and click OK.
If no key is available, click create keys to create a key in the KMS console. For more information, see Create a CMK.
If the status of the cluster changes from Updating to Running, the Secret encryption feature is enabled for the cluster.
If you no longer need the Secret encryption feature, you can turn off Secret Encryption in the Basic Information section.

Use automatic key rotation to encrypt Secrets

You can use the automatic key rotation feature provided by KMS to encrypt Secrets. During a key rotation, the system uses the original key to encrypt existing Secrets and uses the new key to encrypt new Secrets. For more information about automatic key rotation, see Configure key rotation.

To force the system to use the new key to encrypt existing Secrets, run the following command after the key is rotated:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

FAQ

After Secret encryption is enabled, is ciphertext returned if I use kubectl to query a Secret?

No. After Secret encryption is enabled, plaintext is returned if you use kubectl to query a Secret. The Secret encryption feature encrypts the Secrets that are stored in etcd. After you enable Secret encryption, Secrets are stored in etcd as ciphertext. However, if you use a kubectl client to query a Secret by calling the Secret API provided by the API server of the cluster, plaintext is returned for the Secret.

How do I prohibit RAM users or RAM roles from enabling or disabling the Secret encryption feature for existing ACK Edge clusters?

To prohibit RAM users or RAM roles from enabling or disabling the Secret encryption feature for existing ACK Edge clusters, attach the following policy to the RAM users or RAM roles. For more information, see Create a custom RAM policy.

  {
      "Action": [
          "cs:UpdateKMSEncryption"
      ],
      "Effect": "Deny",
      "Resource": [
          "*"
      ]
  }