Container Service for Kubernetes:Data encryption and key management

Data encryption

Suggestions on disk encryption

• Enable disk encryption

  Use static encryption as a security best practice. For more information, see Encrypt disk volumes.

• Periodically rotate CMKs

  You can periodically rotate keys and configure key versioning to enhance the security of customer master keys (CMKs). For more information, see Automatic key rotation.

Use keys stored in KMS to encrypt disk volumes

Container Service for Kubernetes (ACK) clusters are suitable for scenarios that require high security or have compliance requirements. ACK allows you to use storage encryption to ensure the privacy and autonomy of data stored in ECS instances without the need to develop and maintain an underlying key management system. For more information, see Encrypt disk volumes.

The example in this section describes how to use a key stored in Key Management Service (KMS) to encrypt a disk volume when you create the disk volume.

1. Create a StorageClass.

   1. Create a file named sc-kms.yaml and copy the following content to the file:

      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        name: csi-disk
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          fsType: ext4
          type: cloud_ssd
          encrypted: "true"
          kmsKeyId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      reclaimPolicy: Delete

   2. Run the following command to create a StorageClass:

      kubectl create -f sc-kms.yaml

2. Create a persistent volume claim (PVC).

   1. Create a file named sc-pvc.yaml and add the following content to the file:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: disk-pvc
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: csi-disk

   2. Run the following command to create a PVC:

      kubectl create -f sc-pvc.yaml

Key management

Kubernetes allows application developers to store sensitive information, such as database passwords, application certificates, and tokens, in Secrets that can be used by pods. The following bullet point list describes the terms related to Secrets:

• Secrets are scoped to namespaces. You can use Secrets together with Kubernetes role-based access control (RBAC) to isolate reads and writes by namespace.

• You can mount Secrets as files or environment variables to containers in pods.

• The sensitive information in Secrets is stored in the temporary file system (tmpfs) of nodes.

• The API server saves Secrets as Base64-encoded plaintext in the etcd of the Kubernetes cluster.

• A Secret is less than 1 MB in size.

As the vessel that stores the sensitive information of applications in Kubernetes, Secrets cannot ensure the security of the sensitive information. Developers and security engineers may encounter the following issues when using Secrets:

• Developers and security engineers need to decide a security solution to manage key information and store keys before applications are deployed.

• How do developers and security engineers keep reading, writing, and transmission of keys safe without compromising the loading and use of keys?

• What security solutions do cloud service providers offer to ensure the security of keys?

• What security approaches should I choose to manage and protect keys in Kubernetes?

To resolve the preceding issues, we recommend that you choose the key management security solutions provided by cloud service providers and key management security best practices intended for developers.

Security solutions

Based on the shared responsibility model for cloud security, cloud service providers are responsible for ensuring the security of configurations and sensitive information on the control plane side and providing key management security solutions. This helps you enhance the security of keys. The following security solutions are recommended:

• Use key management services provided by cloud service providers

  Cloud service providers usually provide key management services. For example, Alibaba Cloud KMS provides professional key lifecycle management and data encryption/decryption services to help you simplify the procedure of application system integration. For more information about KMS, see What is KMS?

  Data breaches may occur during sensitive information hard-coding across the pipeline of application system development and deployment. To avoid hard-coding, you can use key management services in the cloud to read and write keys throughout the pipeline of application development, testing, and building. Key management services also support automatic key rotation, which reduces the risk of sensitive data breaches and helps enterprises meet security compliance requirements.

• Use the Secret Store CSI driver

  In most cases, applications need to retrieve keys from the specified file system path or environment variable. The Secret Store CSI driver helps you import keys stored in a key management service to the specified file system path of a container.

  The Secret Store CSI driver secrets-store-csi-driver is developed by the Kubernetes community based on the CSI standard. secrets-store-csi-driver allows you to mount keys stored in an external key management service to pods by using volumes. After you mount keys by using a volume, the keys can be used in a container through a file system. You do not need to create a Secret in Kubernetes. This avoids storing plaintext Secrets in the etcd or maintaining a large number of Secrets in large Kubernetes clusters. In addition, applications can retrieve keys from the original file path instead of calling the key management service. This avoids incurring additional costs.

  With the help of secrets-store-csi-driver, cloud service providers are able to interface with different key management services in the backend. For example, you can deploy Alibaba Cloud secrets-store-csi-driver-provider-alibabacloud in your cluster to synchronize keys stored in KMS as files or Kubernetes Secrets to containers. In addition, changes in keys can be automatically synchronized from KMS to containers in real time to ensure the validity of the keys. To resolve The Last Key issue, you can use the RAM Roles for Service Accounts (RRSA) solution because secrets-store-csi-driver requires permissions to access keys in KMS. The RRSA solution allows you to authorize the service account of the Secret Store CSI driver to access keys in KMS instead of granting the permissions to pods.

• Enable etcd encryption

  The etcd is the default storage system used by Kubernetes clusters. Secrets are stored as Base64-encoded plaintext in the etcd. This poses security risks. In managed clusters provided by cloud service providers, the etcd is maintained by the cloud service providers. Due to the zero trust principle and security compliance requirements in different scenarios, we recommend that you use etcd encryption provided by Kubernetes. Then, you can use envelope encryption to automatically encrypt keys when synchronizing them from KMS to Secrets and decrypt keys when retrieving them from the Secrets. Etcd encryption can be used together with automatic key rotation to further enhance data security.

• Use confidential containers

  In scenarios that require enhanced data security, such as financial payment, privacy authentication, and data computing related to intellectual property protection, you need to keep the reading, writing, and transmission of sensitive information confidential, and further protect the in-memory computing and storage of the sensitive information in the cloud. In the preceding scenarios, we recommend that you use ACK clusters for trusted execution environment (TEE)-based confidential computing. ACK for TEE-based confidential computing provides a trusted, encrypted execution environment based on the underlying hardware encryption technology. It ensures the integrity of the sensitive information used across code snippets in the cloud and guarantees the protection and confidentiality of data throughout its lifecycle. In addition, the keys used by application systems can be stored in trusted, isolated environments called enclaves to reduce key transmission. Enclaves are similar to the hardware security module (HSM) of KMS.

Security best practices

Based on the key management security solutions provided by cloud service providers, developers and O&M engineers need to guarantee the security of the keys on their side. The following sections describe the recommended key management security best practices on the business side.

• RBAC

  Secret is a basic model of Kubernetes. RBAC on Secrets is fundamental but essential to data security. You need to follow the least privilege principle during daily cluster development and maintenance to avoid issuing credentials that provide read and write permissions on global Secrets. You also need to revoke cluster credentials that may be disclosed.

• Pod security hardening

  Container escapes are common attacks that target Kubernetes clusters. For attackers that have escaped from a container to the underlying host, they can easily obtain the plaintext data stored in the Secrets on the node, and then escalate their privilege to access the entire cluster and steal the sensitive data in the cluster.

  Kubernetes provides multiple security configurations to help developers reinforce the security of containers. Developers need to minimize the admission of pods with capabilities based on business requirements to avoid using privileged configurations, shared host networks, or shared file systems. Developers can use security policies to reject privilege assignment that does not comply with the security policies during application deployment. In addition, network policies can be used to regulate east-west access among pods and further reduce the risk of lateral movement attacks.

• Node security hardening

  Cluster infrastructure security is the foundation of application security. You need to choose private networks when possible and add access control list (ACL) rules to security groups to control inbound and outbound traffic. To ensure the security of cluster nodes, we recommend that you define security compliance standards based on Multi-Level Protection Scheme (MLPS) or Alibaba Cloud Linux Security Hardening. This enhances infrastructure security in terms of identity verification, access control, security audit, and intrusion prevention and reduces the chances of sensitive data breaches on the host side. For more information about MPLS Security Hardening, see ACK security hardening based on MLPS.

  The configurations of the components in Kubernetes clusters play an important role in application security. Baseline inspection can help you identify high-risk configurations and patch vulnerabilities at the earliest opportunity.

• Supply chain security hardening

  Keys are retrieved and used throughout the lifecycle of application artifacts. Therefore, enterprises must improve their awareness of key security and further regulate the use of keys. Sensitive information hard-coding must be prohibited in application templates, code repositories, and configuration files. Key management services are required to centrally manage keys throughout the artifact supply chain. In addition, the internal security management and O&M teams of enterprises need to develop an automatic security inspection mechanism to prevent data breaches in each component of the supply chain.

• Auditing and monitoring

  In enterprise application systems, all operations related to the reading, writing, use, and lifecycle management of keys must be audited and logged to ensure that operations on sensitive data are traceable. In addition, a runtime monitoring mechanism is required, such as alert rules for suspicious read and write operations on sensitive data and alert rules for breaches of the AccessKey pairs of Alibaba Cloud accounts. This way, logs and alerts can be generated to help O&M engineers quickly handle key breaches, assess impacts, and minimize financial losses.

• Use temporary tokens, periodically rotate keys, or configure automatic key rotation

  Do not use static keys such as AccessKey pairs in application systems. We recommend that you use temporary tokens. When an attacker breaches a temporary token, the attacker can exploit the token only within a limited time. This minimizes the chances of the attacker to expand their attack and provides an opportunity for the system O&M engineer to patch the system vulnerability. The vulnerability patching work would be more complicated if long-term keys are disclosed. We recommend that you periodically rotate keys or enable the automatic key rotation feature when managing the keys stored in KMS to enhance application security.

• Use envelope encryption to protect the last key

  Unlike directly encrypting plaintext by using the keys stored in a key management service in the cloud, envelope encryption allows you to use a separate key and then pass the encrypted key through an envelope. With the help of envelope encryption, sensitive data can be encrypted and decrypted offline on the business side. This avoids uploading your keys to the cloud and resolves the trust issue arising from cloud computing. In offline computing scenarios where large amounts of data need to be encrypted and decrypted, envelope encryption also helps save the IT spending on data transmission to the cloud and cloud computing, and improves the overall performance. For more information about envelope encryption, see Use envelope encryption to encrypt and decrypt local data.

  The Last Key is a common issue in key management service-based encryption and decryption scenarios. Envelope encryption requires you to store a Key Encryption Key (KEK) in the cloud to encrypt and decrypt the Content Encryption Key (CEK). To protect the KEK based on least privilege in most scenarios, you can use an access control service such as Alibaba Cloud Resource Access Management (RAM). On the application side, you need to limit access to the RAM credentials that are used to retrieve the KEK from the cloud. You may also need to use automatically rotated temporary tokens to protect the RAM credentials. We recommend that you use an isolation mechanism similar to RRSA to limit applications from accessing the RAM credentials. For more information, see RRSA.