Container Service for Kubernetes:Use RRSA to authorize different pods to access different cloud services

Background information

Elastic container instances run on Elastic Compute Service (ECS) instances. ECS instance metadata contains information about ECS instances on Alibaba Cloud. You can log on to a running ECS instance and view its metadata, and then configure or manage the ECS instance based on the metadata. Applications deployed in an ACK cluster can use ECS instance metadata to obtain Security Token Service (STS) tokens that are used to assume different Resource Access Management (RAM) roles. This way, the applications can call the APIs of different cloud services. For more information, see Obtain instance metadata.

If you want to control the RAM permissions of different applications in a cluster, you can prohibit these applications from using the metadata of ECS instances or elastic container instances to obtain the STS tokens that are used to assume the RAM roles assigned to the instances. You can also choose not to attach RAM policies to the RAM roles that are assigned to the ECS instances or elastic container instances. However, these applications still need to obtain STS tokens for accessing cloud resources by using a secure method. To address these issues, ACK releases the RRSA feature which is developed based on RAM.

You can use the RRSA feature to allow different applications in an ACK cluster to assume different RAM roles. Applications can obtain STS tokens, use the tokens to assume specific RAM roles, and then access relevant cloud services. This enforces the principle of least privilege and allows applications to call API operations without the need to use AccessKey pairs, which prevents AccessKey pair leaks.

Limits

The RRSA feature supports only ACK clusters that run Kubernetes 1.22 and later. ACK clusters that support the RRSA feature include ACK Basic clusters, ACK Pro clusters, ACK Serverless Basic clusters, ACK Serverless Pro clusters, and ACK Edge Pro clusters.

Enable RRSA

If you do not have an existing cluster, enable RRSA when you create an ACK managed cluster or ACK Edge cluster. Otherwise, enable it in the Security and Auditing section of the cluster details page.

Obtain the URL and ARN information

After the RRSA feature is enabled for the cluster, navigate to the Security and Auditing section under the Basic Information tab. If you hover your mouse over the Enabled label next to RRSA OIDC, the URL and Alibaba Cloud Resource Name (ARN) of the OIDC provider are displayed.

After RRSA is enabled, ACK performs the following operations in the background:

• Automatically creates an OIDC issuer that is dedicated to the cluster. The OIDC issuer is managed by ACK. For more information, see OIDC Issuer.

• Enables service account token volume projection for the cluster and merges configurations of the OIDC issuer with the value of the existing service-account-issuer parameter of the cluster. For more information, see Use ServiceAccount token volume projection.

• Creates a RAM role identity provider within your account. The identity provider uses the OIDC issuer for single sign-on (SSO). The identity provider is named ack-rrsa-<cluster_id>. <cluster_id> indicates the ID of your cluster. For more information, see Manage an OIDC IdP.

Work with RRSA

After you enable RRSA for your cluster, perform the following steps to enable the applications in the cluster to obtain STS tokens through RRSA. The STS tokens are used to call the APIs of specific cloud services.

Example

In this example, an application is created and RRSA is enabled for the cluster to allow the application to perform the following operations: assuming a specified RAM role and then calling an API operation to query clusters that belong to the current Alibaba Cloud account.

Sample configurations

• Namespace: rrsa-demo

• Service account: demo-sa

• RAM role: demo-role-for-rrsa

Procedure

• If you want to enable RRSA without installing the ack-pod-identity-webhook component, you can manually specify the required OIDC token file as environment variables in the application template. For more information, see Manually modify the application template to enable RRSA.

• If you want to use an existing RAM role, you must grant the required permissions to the RAM role. For more information, see Use an existing RAM role and grant the required permissions to the RAM role.

1. Install the ack-pod-identity-webhook component.

   1. On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Operations > Add-ons.

   2. On the Add-ons page, click the Security tab, find ack-pod-identity-webhook, and then click Install in the lower-right part of the card.

   3. In the message that appears, confirm the information and click OK.

2. Create a RAM role named demo-role-for-rrsa.

   1. Log on to the RAM console with your Alibaba Cloud account.

   2. In the left-side navigation pane, choose Identities > Roles. On the Roles page, click Create Role.

   3. In the Create Role panel, select IdP for Select Trusted Entity and click Next.

   4. On the Configure Role wizard page, set the following parameters and click OK.

      The following table describes the parameters.

      Parameter	Description
      RAM Role Name	Set the value to demo-role-for-rrsa.
      Note	Enter the description of the RAM role. This parameter is optional.
      IdP Type	Select OIDC.
      Select IdP	Select an IdP. The IdP is named in the ack-rrsa-<cluster_id> format. <cluster_id> indicates the ID of your cluster.
      Conditions	oidc:iss: Use the default value. oidc:aud: Select sts.aliyuncs.com. oidc:sub: Set the condition operator to StringEquals and the value to system:serviceaccount:<namespace>:<serviceAccountName>. <namespace>: Specify the namespace of the application. <serviceAccountName>: Specify the name of the service account. In this example, enter system:serviceaccount:rrsa-demo:demo-sa.

3. Attach the AliyunCSReadOnlyAccess policy to the RAM role created in Step 2 to grant the required permissions to the application. For more information, see Grant permissions to a RAM role.

4. Deploy an application. For more information about the SDK demos, see Demos of Alibaba Cloud SDKs that support OIDC token authentication of RRSA.

   1. Create a file named demo.yaml based on the following requirements:

      In the following sample YAML template, the pod-identity.alibabacloud.com/injection: 'on' label is added to the namespace, the pod-identity.alibabacloud.com/role-name: demo-role-for-rrsa annotation is added to the service account, and the auto injection feature of ack-pod-identity-webhook is enabled. For more information about how to configure ack-pod-identity-webhook, see ack-pod-identity-webhook.

      View sample code

      ---
      apiVersion: v1
      kind: Namespace
      metadata:
        name: rrsa-demo
        labels:
          pod-identity.alibabacloud.com/injection: 'on'
      
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: demo-sa
        namespace: rrsa-demo
        annotations:
          pod-identity.alibabacloud.com/role-name: demo-role-for-rrsa
      
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: demo
        namespace: rrsa-demo
      spec:
        serviceAccountName: demo-sa
        containers:
          - image: registry.cn-hangzhou.aliyuncs.com/acs/ack-ram-tool:1.0.0
            imagePullPolicy: "Always"
            args:
              - rrsa
              - demo
            name: demo
        restartPolicy: OnFailure

   2. Run the following command to deploy the application:

      kubectl apply -f demo.yaml

5. Run the following command to check whether ack-pod-identity-webhook has injected the required configurations into the pod that is created for the application:

   kubectl -n rrsa-demo get pod demo -o yaml

   Expected output

   apiVersion: v1
   kind: Pod
   metadata:
     name: demo
     namespace: rrsa-demo
   spec:
     containers:
     - args:
       - rrsa
       - demo
       env:
       - name: ALIBABA_CLOUD_ROLE_ARN
         value: acs:ram::1***:role/demo-role-for-rrsa
       - name: ALIBABA_CLOUD_OIDC_PROVIDER_ARN
         value: acs:ram::1***:oidc-provider/ack-rrsa-c***
       - name: ALIBABA_CLOUD_OIDC_TOKEN_FILE
         value: /var/run/secrets/ack.alibabacloud.com/rrsa-tokens/token
       image: registry.cn-hangzhou.aliyuncs.com/acs/ack-ram-tool:1.0.0
       imagePullPolicy: Always
       name: demo
       volumeMounts:
       - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
         name: kube-api-access-4bwdg
         readOnly: true
       - mountPath: /var/run/secrets/ack.alibabacloud.com/rrsa-tokens
         name: rrsa-oidc-token
         readOnly: true
     restartPolicy: OnFailure
     serviceAccount: demo-sa
     serviceAccountName: demo-sa
     volumes:
     - name: kube-api-access-4bwdg
       projected:
         defaultMode: 420
         sources:
         - serviceAccountToken:
             expirationSeconds: 3607
             path: token
         - configMap:
             items:
             - key: ca.crt
               path: ca.crt
             name: kube-root-ca.crt
         - downwardAPI:
             items:
             - fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.namespace
               path: namespace
     - name: rrsa-oidc-token
       projected:
         defaultMode: 420
         sources:
         - serviceAccountToken:
             audience: sts.aliyuncs.com
             expirationSeconds: 3600
             path: token

   The output indicates that ack-pod-identity-webhook has injected the following configurations into the pod.

   Category	Configuration item	Description
   The environment variables.	ALIBABA_CLOUD_ROLE_ARN	The ARN of the RAM role to be assumed.
   	ALIBABA_CLOUD_OIDC_PROVIDER_ARN	The ARN of the OIDC IdP.
   	ALIBABA_CLOUD_OIDC_TOKEN_FILE	The path of the OIDC token file.
   VolumeMount	rrsa-oidc-token	The configuration for mounting the OIDC token.
   Volume	rrsa-oidc-token	The configuration for mounting the OIDC token.

6. Run the following command to print the log of the application:

   kubectl -n rrsa-demo logs demo

   A list of clusters are displayed in the output:

   cluster id: cf***, cluster name: foo*
   cluster id: c8***, cluster name: bar*
   cluster id: c4***, cluster name: foob*

7. Optional: Detach the AliyunCSReadOnlyAccess system policy from the RAM role. For more information, see Remove permissions from a RAM role.

   Wait 30 seconds and run the following command to print the log of the application again:

   kubectl -n rrsa-demo logs demo

   The following error message indicates that the application does not have the required permissions:

      StatusCode: 403
      Code: StatusForbidden
      Message: code: 403, STSToken policy Forbidden for action cs:DescribeClusters request id: E78A2E2D-***
      Data: {"accessDeniedDetail":{"AuthAction":"cs:DescribeClusters","AuthPrincipalDisplayName":"demo-role-for-rrsa:ack-ram-tool","AuthPrincipalOwnerId":"11***","AuthPrincipalType":"AssumedRoleUser","NoPermissionType":"ImplicitDeny","PolicyType":"ResourceGroupLevelIdentityBasedPolicy"},"code":"StatusForbidden","message":"STSToken policy Forbidden for action cs:DescribeClusters","requestId":"E78A2E2D-***","status":403,"statusCode":403}

Manually modify the application template to enable RRSA

You can manually specify the required OIDC token file as environment variables in the application template. This way, you can enable RRSA without installing ack-pod-identity-webhook.

The following template provides an example.

apiVersion: v1
kind: Pod
metadata:
  name: demo
  namespace: rrsa-demo
spec:
  containers:
  - args:
    - rrsa
    - demo
    env:
    - name: ALIBABA_CLOUD_ROLE_ARN
      value: <role_arn>
    - name: ALIBABA_CLOUD_OIDC_PROVIDER_ARN
      value: <oid_provider_arn>
    - name: ALIBABA_CLOUD_OIDC_TOKEN_FILE
      value: /var/run/secrets/ack.alibabacloud.com/rrsa-tokens/token
    image: registry.cn-hangzhou.aliyuncs.com/acs/ack-ram-tool:1.0.0
    imagePullPolicy: Always
    name: demo
    volumeMounts:
    - mountPath: /var/run/secrets/ack.alibabacloud.com/rrsa-tokens
      name: rrsa-oidc-token
      readOnly: true
  restartPolicy: OnFailure
  serviceAccount: demo-sa
  serviceAccountName: demo-sa
  volumes:
  - name: rrsa-oidc-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.aliyuncs.com
          expirationSeconds: 3600
          path: token

After you redeploy the application based on the modified template, the application can use the OIDC token file, the RAM role ARN, and the OIDC provider ARN that you specified in the template to call the AssumeRoleWithOIDC operation of STS and obtain the STS token used to assume a specific RAM role. This way, the application can call the API operations of different cloud services. The OIDC token file is specified in the ALIBABA_CLOUD_OIDC_TOKEN_FILE environment variable, the RAM role ARN is specified in the ALIBABA_CLOUD_ROLE_ARN environment variable, and the OIDC provider ARN is specified in the ALIBABA_CLOUD_OIDC_PROVIDER_ARN environment variable. The application reads the latest OIDC token from the OIDC token file when the application uses the file. For more information about the SDK demos, see Demos of Alibaba Cloud SDKs that support OIDC token authentication of RRSA. For more information, see AssumeRoleWithOIDC.

Use an existing RAM role and grant the required permissions to the RAM role

If you want the application to use an existing RAM role, you must modify the trust policy of the RAM role based on the following template. This way, the application can use the service account to assume the RAM role and obtain an STS token. For more information, see Edit the trust policy of a RAM role.

Example of the Statement configurations:

{
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": {
      "oidc:aud": "sts.aliyuncs.com",
      "oidc:iss": "<oidc_issuer_url>",
      "oidc:sub": "system:serviceaccount:<namespace>:<service_account>"
    }
  },
  "Effect": "Allow",
  "Principal": {
    "Federated": [
      "<oidc_provider_arn>"
    ]
  }
}

You can also use the ack-ram-tool tool to automate the modification of the trust policy. Sample commands:

ack-ram-tool rrsa associate-role --cluster-id <cluster_id> \
    --namespace <namespace> --service-account <service_account> \
    --role-name <role_name> --create-role-if-not-exist

Demos of Alibaba Cloud SDKs that support OIDC token authentication of RRSA

SDK demos

Alibaba Cloud SDK V2.0 supports OIDC token authentication of RRSA. By default, all cloud service SDKs that support STS token authentication and are developed based on Alibaba Cloud SDK V2.0 support RRSA OIDC token authentication. The following table describes the supported SDK versions and demos.

For some cloud service SDKs, you can refer to the preceding methods to use OIDC tokens of RRSA for authentication when you call API operations. The following table describes the SDK demos.

Solutions to SDK errors

The following table describes the solutions to different errors.

{
 "Code": "AuthenticationFail.OIDCToken.Expired",
 "Message": "This JsonWebToken is expired."
}

{
 "Code": "Throttling.User",
 "Message": "Request was denied due to user flow control."
}

{
 "Code": "AuthenticationFail.OIDCToken.AudienceNotMatch",
 "Message": "Invalid audience."
}

{
 "Code": "AuthenticationFail.OIDCToken.IssuerConfigurationBroken",
 "Message": "Get public keys from OIDC Provider failed, the issuer is https://kubernetes.default.svc."
}

{
 "Code": "AuthenticationFail.OIDCToken.IssuerNotMatch",
 "Message": "The issuer in the OIDC Token doesn't match the OIDC Provider registered."
}

{
 "Code": "AuthenticationFail.NoPermission",
 "Message": "No such OIDC Provider registered."
}

{
 "Code": "EntityNotExist.Role",
 "Message": "The role not exists: acs:ram::19981***:role/***. "
}

{
  "Code": "AuthenticationFail.NoPermission",
  "Message": "There is no permission"
}

Enable commonly used CLIs to support OIDC token authentication of RRSA

You can use ack-ram-tool to enable commonly used CLIs to support OIDC token authentication of RRSA in pods. The following table describes the configurations and examples.

{
  "current": "rrsa",
  "profiles": [
    {
      "name": "rrsa",
      "mode": "OIDC",
      "region_id": "cn-hangzhou",
      "ram_session_name": "test-rrsa"
    }
  ],
  "meta_path": ""
}

$ aliyun sts GetCallerIdentity
{
    "AccountId": "11380***",
    "Arn": "acs:ram::1138***:assumed-role/test-rrsa-***/test-rrsa",
    "IdentityType": "AssumedRoleUser",
    "PrincipalId": "33300***:test-rrsa",
    "RequestId": "20F78881-F47E-5771-90D6-***",
    "RoleId": "33300***"
}

$ aliyun sts GetCallerIdentity --region cn-hangzhou --role-session-name=test-rrsa
{
	"AccountId": "11380***",
	"Arn": "acs:ram::1138***:assumed-role/test-rrsa-***/test-rrsa",
	"IdentityType": "AssumedRoleUser",
	"PrincipalId": "33300***:test-rrsa",
	"RequestId": "20F78881-F47E-5771-90D6-***",
	"RoleId": "33300***"
}

$ ack-ram-tool export-credentials -f environment-variables -- aliyunlog log list_project --region-endpoint=cn-hangzhou.log.aliyuncs.com

{"count": 1, "projects": [
{"createTime": "1676282996", "description": "k8s log project, ***",
 "lastModifyTime": "1676282996", "owner": "", "projectName": "k8s-log-c0edc***", 
 "region": "cn-hangzhou", "resourceGroupId": "rg-***", "status": "Normal"}],
  "total": 24}

provider "alicloud" {
  assume_role_with_oidc {
    role_session_name = "terraform-with-rrsa-auth-example"
  }
  region = "cn-hangzhou"
}

References

• ack-pod-identity-webhook

• ServiceAccount token volume projection

• OIDC introduction

• Overview of OIDC-based SSO

• AssumeRoleWithOIDC - Obtains an STS token to assume a RAM role during SSO by using OIDC

• ack-ram-tool