全部產品
Search
文件中心

Elastic Container Instance:部署CCM

更新時間:Jul 06, 2024

CCM提供Kubernetes與阿里雲基礎產品(例如CLB、VPC等)對接的能力,支援在同一個CLB後端掛載叢集內節點和叢集外伺服器,可以解決遷移過程中流量中斷的難題,同時還支援將業務流量轉寄至多個Kubernetes叢集,實現備份、容災等需求,從而保障業務的高可用。本文介紹如何在自建的Kubernetes叢集中部署CCM。

前提條件

  • 自建Kubernetes叢集中已部署VNode。

  • 如果您的Kubernetes叢集部署線上下IDC,請確保已打通IDC與阿里雲的網路。

背景資訊

CCM(Cloud Controller Manager)是阿里雲提供的一個用於Kubernetes與阿里雲基礎產品進行對接的組件,目前包括以下功能:

  • 管理負載平衡

    當Service的類型設定為LoadBalancer時,CCM組件會為該Service建立並配置阿里雲負載平衡CLB,包括CLB執行個體、監聽、後端伺服器組等資源。當Service對應的後端Endpoint或者叢集節點發生變化時,CCM會自動更新CLB的後端伺服器組。

  • 實現跨節點通訊

    當叢集網路組件為Flannel時,CCM組件負責打通容器與節點間網路,將節點的Pod網段資訊寫入VPC的路由表中,從而實現容器的跨節點通訊。該功能無需配置,安裝即可使用。

更多資訊,請參見Cloud Controller Manager

說明

CCM已經開源,具體專案資訊請參見cloud-provider-alibaba-cloud

準備工作

如果您自建的Kubernetes叢集中沒有使用阿里雲ECS作為節點,可跳過準備工作。如果使用了ECS作為叢集節點,需要參考以下步驟配置ECS節點的providerID,使得CCM可以管理這些節點的路由。

  1. 部署OpenKurise以便使用BroadcastJob。

    命令參考如下:

    helm repo add openkruise https://openkruise.github.io/charts/
    helm repo update
    helm install kruise openkruise/kruise --version 1.3.0

    更多資訊,請參見OpenKruise文檔

  2. 通過BroadcastJob為ECS節點配置providerID。

    1. 將以下內容儲存為provider.yaml。

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: ecs-node-initor
      rules:
        - apiGroups:
            - ""
          resources:
            - nodes
          verbs:
            - get
            - patch
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: ecs-node-initor
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: ecs-node-initor
      subjects:
      - kind: ServiceAccount
        name: ecs-node-initor
        namespace: default
      roleRef:
        kind: ClusterRole
        name: ecs-node-initor
        apiGroup: rbac.authorization.k8s.io
      ---
      apiVersion: apps.kruise.io/v1alpha1
      kind: BroadcastJob
      metadata:
        name: create-ecs-node-provider-id
      spec:
        template:
          spec:
            serviceAccount: ecs-node-initor
            restartPolicy: OnFailure
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: type
                      operator: NotIn
                      values:
                      - virtual-kubelet
            tolerations:
            - operator: Exists
            containers:
              - name: create-ecs-node-provider-id
                image: registry.cn-beijing.aliyuncs.com/eci-release/provider-initor:v1
                command: [ "/usr/bin/init" ]
                env:
                - name: NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
        completionPolicy:
          type: Never
        failurePolicy:
          type: FailFast
          restartLimit: 3
    2. 部署BroadcastJob。

      kubectl apply -f provider.yaml
  3. 查看BroadcastJob執行結果。

    kubectl get pods -o wide

    如果create-ecs-node-provider-id相關的Pod均達到Completed狀態,則表示相應ECS節點的ProviderID已經配置成功。返回樣本如下:

    部署CCM1

  4. (可選)清理BroadcastJob。

    kubectl delete -f provider.yaml

操作步驟

  1. 建立ConfigMap。

    1. 將您的阿里雲帳號對應的AccessKey儲存到環境變數。

      export ACCESS_KEY_ID=LTAI********************
      export ACCESS_KEY_SECRET=HAeS**************************

      關於如何擷取AccessKey ID和AccessKey Secret,請參見擷取AccessKey

    2. 執行以下指令碼建立ConfigMap。

      將以下內容儲存為configmap-ccm.sh,並根據實際替換代碼中的region值,然後執行指令碼。

      #!/bin/bash
      
      ## create ConfigMap kube-system/cloud-config for CCM.
      accessKeyIDBase64=`echo -n "$ACCESS_KEY_ID" |base64 -w 0`
      accessKeySecretBase64=`echo -n "$ACCESS_KEY_SECRET"|base64 -w 0`
      
      cat <<EOF >cloud-config.yaml
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cloud-config
        namespace: kube-system
      data:
        cloud-config.conf: |-
          {
              "Global": {
                  "accessKeyID": "$accessKeyIDBase64",
                  "accessKeySecret": "$accessKeySecretBase64",
                  "region": "cn-hangzhou"
              }
          }
      EOF
      
      kubectl create -f cloud-config.yaml
      bash configmap-ccm.sh

      執行指令碼後,系統將在Kube-system下建立一個名為cloud-config的ConfigMap。

  2. 部署CCM。

    1. 修改${ImageVersion}{$ClusterCIDR},然後將以下內容儲存為ccm.yaml。

      • 您可以根據CCM的變更記錄擷取ImageVersion。具體請參見Cloud Controller Manager

      • 您可以通過kubectl cluster-info dump | grep -m1 cluster-cidr命令查看ClusterCIDR

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: system:cloud-controller-manager
      rules:
        - apiGroups:
            - coordination.k8s.io
          resources:
            - leases
          verbs:
            - get
            - list
            - update
            - create
        - apiGroups:
            - ""
          resources:
            - persistentvolumes
            - services
            - secrets
            - endpoints
            - serviceaccounts
          verbs:
            - get
            - list
            - watch
            - create
            - update
            - patch
        - apiGroups:
            - ""
          resources:
            - nodes
          verbs:
            - get
            - list
            - watch
            - delete
            - patch
            - update
        - apiGroups:
            - ""
          resources:
            - services/status
          verbs:
            - update
            - patch
        - apiGroups:
            - ""
          resources:
            - nodes/status
          verbs:
            - patch
            - update
        - apiGroups:
            - ""
          resources:
            - events
            - endpoints
          verbs:
            - create
            - patch
            - update
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: cloud-controller-manager
        namespace: kube-system
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: system:cloud-controller-manager
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:cloud-controller-manager
      subjects:
        - kind: ServiceAccount
          name: cloud-controller-manager
          namespace: kube-system
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: system:shared-informers
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:cloud-controller-manager
      subjects:
        - kind: ServiceAccount
          name: shared-informers
          namespace: kube-system
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: system:cloud-node-controller
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:cloud-controller-manager
      subjects:
        - kind: ServiceAccount
          name: cloud-node-controller
          namespace: kube-system
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: system:pvl-controller
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:cloud-controller-manager
      subjects:
        - kind: ServiceAccount
          name: pvl-controller
          namespace: kube-system
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: system:route-controller
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:cloud-controller-manager
      subjects:
        - kind: ServiceAccount
          name: route-controller
          namespace: kube-system
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        labels:
          app: cloud-controller-manager
          tier: control-plane
        name: cloud-controller-manager
        namespace: kube-system
      spec:
        selector:
          matchLabels:
            app: cloud-controller-manager
            tier: control-plane
        template:
          metadata:
            labels:
              app: cloud-controller-manager
              tier: control-plane
            annotations:
              scheduler.alpha.kubernetes.io/critical-pod: ''
          spec:
            serviceAccountName: cloud-controller-manager
            tolerations:
              - effect: NoSchedule
                operator: Exists
                key: node-role.kubernetes.io/master
              - effect: NoSchedule
                operator: Exists
                key: node.cloudprovider.kubernetes.io/uninitialized
            nodeSelector:
              node-role.kubernetes.io/master: ""
            containers:
              - command:
                -  /cloud-controller-manager
                - --leader-elect=true
                - --cloud-provider=alicloud
                - --use-service-account-credentials=true
                - --cloud-config=/etc/kubernetes/config/cloud-config.conf
                - --configure-cloud-routes=true
                - --route-reconciliation-period=3m
                - --leader-elect-resource-lock=endpoints
                # replace ${cluster-cidr} with your own cluster cidr
                # example: 172.16.0.0/16
                - --cluster-cidr=${ClusterCIDR}  
                # replace ${ImageVersion} with the latest release version
                # example: v2.1.0
                image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:${ImageVersion}
                livenessProbe:
                  failureThreshold: 8
                  httpGet:
                    host: 127.0.0.1
                    path: /healthz
                    port: 10258
                    scheme: HTTP
                  initialDelaySeconds: 15
                  timeoutSeconds: 15
                name: cloud-controller-manager
                resources:
                  requests:
                    cpu: 200m
                volumeMounts:
                  - mountPath: /etc/kubernetes/
                    name: k8s
                  - mountPath: /etc/ssl/certs
                    name: certs
                  - mountPath: /etc/pki
                    name: pki
                  - mountPath: /etc/kubernetes/config
                    name: cloud-config
            hostNetwork: true
            volumes:
              - hostPath:
                  path: /etc/kubernetes
                name: k8s
              - hostPath:
                  path: /etc/ssl/certs
                name: certs
              - hostPath:
                  path: /etc/pki
                name: pki
              - configMap:
                  defaultMode: 420
                  items:
                    - key: cloud-config.conf
                      path: cloud-config.conf
                  name: cloud-config
                name: cloud-config
    2. 執行以下命令部署CCM。

      kubectl create -f ccm.yaml

結果驗證

  1. 建立LoadBalancer類型的Service和一組後端Endpoint。

    1. 將以下內容儲存為ccm-test.yaml。

      image地址請替換為VNode所在地區的地址,以免鏡像下載失敗。

      apiVersion: v1
      kind: Service
      metadata:
        name: nginx
        namespace: default
        annotations:
          service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
      spec:
        ports:
        - port: 80
          protocol: TCP
          targetPort: 80
        selector:
          app: nginx
        type: LoadBalancer
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: test-nginx
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
            - name: nginx
              image: registry-vpc.cn-beijing.aliyuncs.com/eci_open/nginx:1.14.2
    2. 執行命令部署Service和Deployment。

      kubectl create -f ccm-test.yaml

      部署後,CCM組件會為Service建立並配置阿里雲負載平衡CLB,包括CLB執行個體、監聽、後端伺服器組等資源。

  2. 驗證Service是否能夠正常工作。

    執行curl命令訪問Service地址,可以看到已經能通過Service地址來訪問後端的Nginx服務。

    ccm

更多關於Service的使用方式,請參見CCM使用方式說明