Container Service for Kubernetes:Use Terraform to manage add-ons

Types of add-ons

ACK manages the following types of cluster add-ons: system add-ons and optional add-ons. For more information, see Component overview.

System add-ons

Optional add-ons

You can deploy optional add-ons to extend the capabilities of your clusters on demand. Optional add-ons are classified into application management add-ons, logging and monitoring add-ons, volume add-ons, network add-ons, and security add-ons.

Best practices for add-on management

You can use Terraform to specify the add-ons to be installed when you create a cluster. After the cluster is created, you can manage the lifecycle of the add-ons in the cluster. The following sections describe how to manage the lifecycle of add-ons installed in a cluster and provide best practices that apply to different scenarios.

Specify the add-ons to be installed when you create a cluster

You can specify the add-ons to be installed when you create a cluster. The following Resources are involved when you create different types of clusters:

• ACK managed cluster: alicloud_cs_managed_kubernetes

• ACK dedicated cluster: alicloud_cs_kubernetes

• ACK Edge cluster: alicloud_cs_edge_kubernetes

• ACK Serverless cluster: alicloud_cs_serverless_kubernetes

You can set the addons object to specify the add-ons to be installed when you create one of the preceding Resources. The following code block shows the attributes of the addons object:

# This example uses an ACK managed cluster. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  # The addons object is a list. You can set the addons object in a Resource to specify the add-ons to be installed when the system creates the cluster. 
  addons {
    # The name of the add-on. You can query the name of an add-on by using alicloud_cs_kubernetes_addons of Data Source. 
    # The add-ons that are already installed and can be installed in the cluster and the versions of the add-ons. 
    name = "XXX"

    # Custom add-on parameters. You can set this attribute for cluster add-ons that support custom parameters. For more information, see the Modify the custom parameters of a cluster add-on section. 
    config = jsonencode(
      {
        ....
      }
    )

    # The value is of Boolean type and the default is false. By default, ACK automatically installs specific add-ons for you to manage the cluster. If you do not want ACK to install add-ons when ACK creates the cluster, set disabled=true. 
    disabled = XXX
  }
}

The following table describes how to configure add-ons by using Terraform in ACK clusters.

Manage the lifecycle of add-ons after the cluster is created

To manage the lifecycle of add-ons, make sure that you have an ACK cluster. If you do not have an ACK cluster, create one first.

You can set the alicloud_cs_kubernetes_addon object in the cluster Resource to manage the lifecycle of the add-ons installed in the cluster. You can install, update, and uninstall add-ons and customize the configuration of the add-ons. The following code block shows the attributes of the alicloud_cs_kubernetes_addon object:

resource "alicloud_cs_kubernetes_addon" "addon-example" {
  # The ID of the cluster. 
  cluster_id = "XXXX"

  # The name of the add-on. You can query the add-ons that are already installed and can be installed and their versions by using alicloud_cs_kubernetes_addons of Data Source. 
  name = "XXXX"

  # The version of the add-on. 
  version = "XXXX"

  # Custom add-on parameters in a JSON string. You can use the jsonencode method of Terraform to specify the parameters or directly specify the parameters in a JSON string. Pay attention to character escaping if you directly specify the parameters in a JSON string. You can set this attribute for cluster add-ons that support custom parameters. For more information, see the Modify the custom parameters of a cluster add-on section. 
  config = jsonencode(
    {
      ....
    }
  )
}

You can directly specify custom parameters in a JSON string. Pay attention to character escaping when you use this method. For example, you can use one of the following methods to configure nginx-ingress-controller:

• Use jsonencode to configure custom parameters:

  config = jsonencode(   
    {       
      IngressSlbNetworkType="internet"       
      IngressSlbSpec="slb.s2.small"     
    }  
  )

• Directly specify custom parameters in a JSON string:

  config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"

Import the installed add-ons to Terraform for management

You can use the terraform import method to import the add-ons that are installed in the cluster to Terraform and then manage these add-ons by using Terraform. This section uses nginx-ingress-controller as an example to demonstrate how to import the installed add-ons to Terraform.

1. Create a file whose suffix is .tf and define a Resource. If you already have a file whose suffix is .tf, define a Resource in the file.

   The alicloud_cs_kubernetes_addon object in the Resource is used to manage the add-ons in the cluster. You do not need to add content to the object.

   resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" {
   }

2. Run the following command to import nginx-ingress-controller:

   Terraform automatically pulls the configuration of nginx-ingress-controller from the cluster and adds the configuration to the file whose suffix is .state.

   terraform import alicloud_cs_kubernetes_addon.nginx-ingress-controller <cluster_id>:nginx-ingress-controller

3. Run the terraform plan command. The command output shows the difference between the configurations of nginx-ingress-controller and Resource.

   Modify the Resource defined in Step 1 based on the configuration difference and the content of the file whose suffix is .state. If the output of the terraform plan command shows no difference between the configurations of nginx-ingress-controller and Resource, the add-on is imported to Terraform.

   resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" {
     cluster_id = "XXXXX"
     name = "nginx-ingress-controller"
     version = "v1.2.1-aliyun.1"
     config = jsonencode(
       {
         IngressSlbNetworkType = "internet"
         IngressSlbSpec        = "slb.s2.small"
       }
     )
   }

Install cluster add-ons

You can use the alicloud_cs_kubernetes_addon object in the Resource to install add-ons in the cluster. This section uses the Gatekeeper add-on as an example.

1. Specify the following information about the add-on to be installed in the file whose suffix is .tf:

   • The ID of the cluster.

   • The name and version of the add-on:

     You can query the names and versions of the add-ons that can be installed by using alicloud_cs_kubernetes_addons of Data Source. The result displays only the latest version of each add-on available for installation. If you want to install an earlier version, check the release notes of the add-on and specify the corresponding version number.

   • (Optional) Custom add-on configuration:

     You can modify the config field to customize the add-on configuration by using the jsonencode method of Terraform. You can query the custom parameters of an add-on by using alicloud_cs_kubernetes_addon_metadata of Data Source. For more information, see Modify the custom parameters of a cluster add-on.

   Click to view details

   resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
     cluster_id = "ce36b7c61e126430b8b245730ca6d****"
     name = "gatekeeper"
     version = "v3.8.1.113-geb7947ef-aliyun"
     config = jsonencode(
       {
         AdmissionPodCpuLimit      = "1000m"
         AdmissionPodCpuRequest    = "100m"
         AdmissionPodMemoryLimit   = "512Mi"
         AdmissionPodMemoryRequest = "256Mi"
         AdmissionPodNumber        = 3
         AuditInterval             = 1800
         AuditPodCpuLimit          = "1000m"
         AuditPodCpuRequest        = "100m"
         AuditPodMemoryLimit       = "512Mi"
         AuditPodMemoryRequest     = "256Mi"
         EnableAuditPod            = false
         EnableMutatingWebhook     = false
       }
     )
   }

2. un the following command to install the add-on in the cluster:

   terraform apply

   Expected output:

   Plan: 1 to add, 0 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   
     Enter a value: yes
   
   alicloud_cs_kubernetes_addon.gatekeeper: Creating...
   alicloud_cs_kubernetes_addon.gatekeeper: Still creating... [10s elapsed]
   alicloud_cs_kubernetes_addon.gatekeeper: Creation complete after 16s [id=XXXXX:gatekeeper]
   
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

   If Apply complete! is displayed, the add-on is installed.

Update cluster add-ons

You can query the versions of an add-on that are available for updating by using alicloud_cs_kubernetes_addons of Data Source. If a new version is available, you can change the version number to update the add-on. This section uses the Gatekeeper add-on as an example.

resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name = "gatekeeper"

  # Change version to the version number that you want to use. 
  version = "XXXXXXXXX"
  config = jsonencode(
    {
      AdmissionPodCpuLimit      = "1000m"
      AdmissionPodCpuRequest    = "100m"
      AdmissionPodMemoryLimit   = "512Mi"
      AdmissionPodMemoryRequest = "256Mi"
      AdmissionPodNumber        = 3
      AuditInterval             = 1800
      AuditPodCpuLimit          = "1000m"
      AuditPodCpuRequest        = "100m"
      AuditPodMemoryLimit       = "512Mi"
      AuditPodMemoryRequest     = "256Mi"
      EnableAuditPod            = false
      EnableMutatingWebhook     = false
    }
  )
}

Run the terraform apply command to update the add-on. If Apply complete! is displayed, the add-on is updated.

Modify the custom parameters of a cluster add-on

You can use alicloud_cs_kubernetes_addons in the Resource to modify the configuration of an add-on if the add-on supports custom parameters. This section uses the Gatekeeper add-on as an example to demonstrate how to modify the config field to customize the add-on configuration.

resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name = "gatekeeper"
  version = "v3.8.1.113-geb7947ef-aliyun"

  # You can modify and apply the attributes in Config to customize the configuration of a cluster add-on. 

}

To query the custom parameters supported by an add-on, use alicloud_cs_kubernetes_addon_metadata of Data Source. The result is returned in a JSON schema. For example, to customize the configuration of the Gatekeeper add-on, add the following content to the file whose suffix is .tf.

# Define Data Source to obtain the schema that includes the custom parameters supported by the Gatekeeper add-on. 
data "alicloud_cs_kubernetes_addon_metadata" "default" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name       = "gatekeeper"
  version    = "v3.8.1.113-geb7947ef-aliyun"
}

# Output the schema. 
output "addon_config_schema" {
  value = data.alicloud_cs_kubernetes_addons.default.config_schema
}

addon_config_schema = <<EOT
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "AdmissionPodCpuLimit": {
      "default": "1000m",
      "description": "cpu limit for gatekeeper",
      "pattern": "^(|[1-9][0-9]*(m|\\.\\d+)?)$",
      "type": "string"
    },
    "AdmissionPodCpuRequest": {
      "default": "100m",
      "description": "cpu request for gatekeeper",
      "pattern": "^[1-9][0-9]*(m|\\.\\d+)?$",
      "type": "string"
    },
    "AdmissionPodMemoryLimit": {
      "default": "512Mi",
      "description": "memory limit for gatekeeper",
      "pattern": "^(|[1-9][0-9]*(\\.\\d+)?(K|Ki|M|Mi|G|Gi|T|Ti)?)$",
      "type": "string"
    },

    ......
  },
  "title": "Config",
  "type": "object"
}
EOT

Configure network add-ons

In ACK clusters, you can use Terway to implement the preceding container network features. For more information, see Overview.

# Configure the Terway network add-on and enable the Assign One ENI to Each Pod mode. This is the default mode. 
# In this mode, the number of pods on a node is limited by the elastic network interface (ENI) quota of Elastic Compute Service (ECS) instances. 

resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "terway-eni"
  }
}

# Configure the Terway network add-on and enable the IPVLAN mode. 
# You can use only the Alibaba Cloud Linux 2 operating system because the One ENI for Multi-Pod mode uses the IPVLAN + eBPF virtualization technology. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"false\"}"
  }
}

# Configure the Terway network add-on and enable Kubernetes network policies in IPVLAN mode. 
# You can use only the Alibaba Cloud Linux 2 operating system because the One ENI for Multi-Pod mode uses the IPVLAN + eBPF virtualization technology. 
# The IPVLAN mode provides network access control based on Kubernetes network policies. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"true\"}"
  }
}

Configure volume add-ons

# The CSI volume add-on consists of csi-plugin and csi-provisioner. If you use the CSI add-on and want ACK to create a default NAS file system and CNFS file system for dynamically provisioned volumes, you must also install the storage-operator add-on. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name   = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
}

Configure logging add-ons

# Use the automatically created Simple Log Service project. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "logtail-ds"
  }
}

# Use the automatically created Simple Log Service project. Enable the automatically created Ingress dashboard. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\"}
  }
}


# Use an existing Simple Log Service project. Enable the automatically created Ingress dashboard. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\",\"sls_project_name\":\"k8s-log-c55c35ff493df47b88783bea48827****\"}"
  }
}

# Install and configure node-problem-detector. 
# Configure node-problem-detector to use the automatically created Simple Log Service project to store events. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
}

# Install and configure node-problem-detector. 
# Configure node-problem-detector to use an existing Simple Log Service project to store events. node-problem-detector can share the Logstore that is used by logtail-ds. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"k8s-log-c55c35ff493df47b88783bea48827****\"}"
  }
}

Configure monitoring add-ons

# Install the CloudMonitor agent on ECS nodes. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...
  install_cloud_monitor = true
}

# Install Managed Service for Prometheus add-on. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "arms-prometheus"
  }
}

# Install the CloudMonitor agent and Managed Service for Prometheus add-on at the same time. 

# Install the Managed Service for Prometheus add-on. We recommend that you install the add-on and then enable Managed Service for Prometheus. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...
  install_cloud_monitor = true
  addons {
    name = "arms-prometheus"
  }
}

Configure Ingress add-ons

# Configure nginx-ingress-controller. 
# If you use an Internet-facing Server Load Balancer (SLB) instance, you must set the IngressSlbNetworkType parameter to internet in the Config. 
# If you use an internal-facing SLB instance, you must set the IngressSlbNetworkType parameter to intranet in the Config. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "nginx-ingress-controller",
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
}

# Configure alb-ingress-controller. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "alb-ingress-controller",
  }
}

Disable the system to install default add-ons

# Disable the system to install the nginx-ingress-controller add-on. 

resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name     =  "nginx-ingress-controller",
    disabled = true
  }
}

Add-ons that are automatically installed if no add-on is specified in the addons object

The system automatically installs the following add-ons if you do not specify an add-on in the addons object:

Examples of commonly used configurations

# Do not install any add-ons when you create the cluster. In this scenario, only the default add-ons are installed. 
# A simple template. Replace the variables with the desired values. 
resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  pod_cidr                     = "172.20.0.0/16"
  service_cidr                 = "172.21.0.0/20"
  worker_vswitch_ids           = ["var.vswitch_id"]
}

# Create a cluster that has Terway installed. 
# Enable the Assign One ENI to Each Pod mode. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  pod_vswitch_ids              = [var.vswitch_id]
  service_cidr                 = "172.21.0.0/20"
  worker_vswitch_ids           = ["var.vswitch_id"]

  addons {
    name = "terway-eni"
  }
}

# Create a cluster that has Terway installed. 
# Enable the IPVLAN mode and enable Kubernetes network policies. 
resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = ["var.vswitch_id"]
  worker_vswitch_ids           = ["var.vswitch_id"]

  addons {
    name   =  "terway-eniip"
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"true\"}"
  }
}

# A template for installing the Terway, CSI, and nginx-ingress-controller add-ons. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = ["var.vswitch_id"]
  worker_vswitch_ids           = ["var.vswitch_id"]

  addons {
    name = "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"false\"}"
  }
  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\"}"
  }
  addons {
    name = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
  addons {
    name = "nginx-ingress-controller"
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
  addons {
    name = "ack-node-local-dns"
  }
  addons {
    name = "arms-prometheus"
  }
  addons {
    name = "alicloud-monitor-controller"
    config = "{\"group_contact_ids\":\"[10619]\"}"
  }
}