CCM提供Kubernetes與阿里雲基礎產品(例如CLB、VPC等)對接的能力,支援在同一個CLB後端掛載叢集內節點和叢集外伺服器,可以解決遷移過程中流量中斷的難題,同時還支援將業務流量轉寄至多個Kubernetes叢集,實現備份、容災等需求,從而保障業務的高可用。本文介紹如何在自建的Kubernetes叢集中部署CCM。
前提條件
自建Kubernetes叢集中已部署VNode。
如果您的Kubernetes叢集部署線上下IDC,請確保已打通IDC與阿里雲的網路。
背景資訊
CCM(Cloud Controller Manager)是阿里雲提供的一個用於Kubernetes與阿里雲基礎產品進行對接的組件,目前包括以下功能:
管理負載平衡
當Service的類型設定為LoadBalancer時,CCM組件會為該Service建立並配置阿里雲負載平衡CLB,包括CLB執行個體、監聽、後端伺服器組等資源。當Service對應的後端Endpoint或者叢集節點發生變化時,CCM會自動更新CLB的後端伺服器組。
實現跨節點通訊
當叢集網路組件為Flannel時,CCM組件負責打通容器與節點間網路,將節點的Pod網段資訊寫入VPC的路由表中,從而實現容器的跨節點通訊。該功能無需配置,安裝即可使用。
更多資訊,請參見Cloud Controller Manager。
CCM已經開源,具體專案資訊請參見cloud-provider-alibaba-cloud。
準備工作
如果您自建的Kubernetes叢集中沒有使用阿里雲ECS作為節點,可跳過準備工作。如果使用了ECS作為叢集節點,需要參考以下步驟配置ECS節點的providerID,使得CCM可以管理這些節點的路由。
部署OpenKurise以便使用BroadcastJob。
命令參考如下:
helm repo add openkruise https://openkruise.github.io/charts/ helm repo update helm install kruise openkruise/kruise --version 1.3.0
更多資訊,請參見OpenKruise文檔。
通過BroadcastJob為ECS節點配置providerID。
將以下內容儲存為provider.yaml。
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ecs-node-initor rules: - apiGroups: - "" resources: - nodes verbs: - get - patch --- apiVersion: v1 kind: ServiceAccount metadata: name: ecs-node-initor --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: ecs-node-initor subjects: - kind: ServiceAccount name: ecs-node-initor namespace: default roleRef: kind: ClusterRole name: ecs-node-initor apiGroup: rbac.authorization.k8s.io --- apiVersion: apps.kruise.io/v1alpha1 kind: BroadcastJob metadata: name: create-ecs-node-provider-id spec: template: spec: serviceAccount: ecs-node-initor restartPolicy: OnFailure affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: type operator: NotIn values: - virtual-kubelet tolerations: - operator: Exists containers: - name: create-ecs-node-provider-id image: registry.cn-beijing.aliyuncs.com/eci-release/provider-initor:v1 command: [ "/usr/bin/init" ] env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName completionPolicy: type: Never failurePolicy: type: FailFast restartLimit: 3
部署BroadcastJob。
kubectl apply -f provider.yaml
查看BroadcastJob執行結果。
kubectl get pods -o wide
如果create-ecs-node-provider-id相關的Pod均達到Completed狀態,則表示相應ECS節點的ProviderID已經配置成功。返回樣本如下:
(可選)清理BroadcastJob。
kubectl delete -f provider.yaml
操作步驟
建立ConfigMap。
將您的阿里雲帳號對應的AccessKey儲存到環境變數。
export ACCESS_KEY_ID=LTAI******************** export ACCESS_KEY_SECRET=HAeS**************************
關於如何擷取AccessKey ID和AccessKey Secret,請參見擷取AccessKey。
執行以下指令碼建立ConfigMap。
將以下內容儲存為configmap-ccm.sh,並根據實際替換代碼中的region值,然後執行指令碼。
#!/bin/bash ## create ConfigMap kube-system/cloud-config for CCM. accessKeyIDBase64=`echo -n "$ACCESS_KEY_ID" |base64 -w 0` accessKeySecretBase64=`echo -n "$ACCESS_KEY_SECRET"|base64 -w 0` cat <<EOF >cloud-config.yaml apiVersion: v1 kind: ConfigMap metadata: name: cloud-config namespace: kube-system data: cloud-config.conf: |- { "Global": { "accessKeyID": "$accessKeyIDBase64", "accessKeySecret": "$accessKeySecretBase64", "region": "cn-hangzhou" } } EOF kubectl create -f cloud-config.yaml
bash configmap-ccm.sh
執行指令碼後,系統將在Kube-system下建立一個名為cloud-config的ConfigMap。
部署CCM。
修改
${ImageVersion}
和{$ClusterCIDR}
,然後將以下內容儲存為ccm.yaml。您可以根據CCM的變更記錄擷取
ImageVersion
。具體請參見Cloud Controller Manager。您可以通過
kubectl cluster-info dump | grep -m1 cluster-cidr
命令查看ClusterCIDR
。
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:cloud-controller-manager rules: - apiGroups: - coordination.k8s.io resources: - leases verbs: - get - list - update - create - apiGroups: - "" resources: - persistentvolumes - services - secrets - endpoints - serviceaccounts verbs: - get - list - watch - create - update - patch - apiGroups: - "" resources: - nodes verbs: - get - list - watch - delete - patch - update - apiGroups: - "" resources: - services/status verbs: - update - patch - apiGroups: - "" resources: - nodes/status verbs: - patch - update - apiGroups: - "" resources: - events - endpoints verbs: - create - patch - update --- apiVersion: v1 kind: ServiceAccount metadata: name: cloud-controller-manager namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:cloud-controller-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: cloud-controller-manager namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:shared-informers roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: shared-informers namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:cloud-node-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: cloud-node-controller namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:pvl-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: pvl-controller namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:route-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: route-controller namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: cloud-controller-manager tier: control-plane name: cloud-controller-manager namespace: kube-system spec: selector: matchLabels: app: cloud-controller-manager tier: control-plane template: metadata: labels: app: cloud-controller-manager tier: control-plane annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: serviceAccountName: cloud-controller-manager tolerations: - effect: NoSchedule operator: Exists key: node-role.kubernetes.io/master - effect: NoSchedule operator: Exists key: node.cloudprovider.kubernetes.io/uninitialized nodeSelector: node-role.kubernetes.io/master: "" containers: - command: - /cloud-controller-manager - --leader-elect=true - --cloud-provider=alicloud - --use-service-account-credentials=true - --cloud-config=/etc/kubernetes/config/cloud-config.conf - --configure-cloud-routes=true - --route-reconciliation-period=3m - --leader-elect-resource-lock=endpoints # replace ${cluster-cidr} with your own cluster cidr # example: 172.16.0.0/16 - --cluster-cidr=${ClusterCIDR} # replace ${ImageVersion} with the latest release version # example: v2.1.0 image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:${ImageVersion} livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10258 scheme: HTTP initialDelaySeconds: 15 timeoutSeconds: 15 name: cloud-controller-manager resources: requests: cpu: 200m volumeMounts: - mountPath: /etc/kubernetes/ name: k8s - mountPath: /etc/ssl/certs name: certs - mountPath: /etc/pki name: pki - mountPath: /etc/kubernetes/config name: cloud-config hostNetwork: true volumes: - hostPath: path: /etc/kubernetes name: k8s - hostPath: path: /etc/ssl/certs name: certs - hostPath: path: /etc/pki name: pki - configMap: defaultMode: 420 items: - key: cloud-config.conf path: cloud-config.conf name: cloud-config name: cloud-config
執行以下命令部署CCM。
kubectl create -f ccm.yaml
結果驗證
建立LoadBalancer類型的Service和一組後端Endpoint。
將以下內容儲存為ccm-test.yaml。
image地址請替換為VNode所在地區的地址,以免鏡像下載失敗。
apiVersion: v1 kind: Service metadata: name: nginx namespace: default annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet" spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer --- apiVersion: apps/v1 kind: Deployment metadata: name: test-nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry-vpc.cn-beijing.aliyuncs.com/eci_open/nginx:1.14.2
執行命令部署Service和Deployment。
kubectl create -f ccm-test.yaml
部署後,CCM組件會為Service建立並配置阿里雲負載平衡CLB,包括CLB執行個體、監聽、後端伺服器組等資源。
驗證Service是否能夠正常工作。
執行curl命令訪問Service地址,可以看到已經能通過Service地址來訪問後端的Nginx服務。
更多關於Service的使用方式,請參見CCM使用方式說明。