全部產品
Search
文件中心

Elastic Container Instance:使用eci-profile調度Pod到VNode

更新時間:Aug 22, 2024

eci-profile可以提供叢集維度ECI資源檢視,支援根據Selector實現Pod編排。本文介紹如何部署eci-profile組件,以及如何使用eci-profile。

eci-profile功能說明

eci-profile可以通過Pod、Namespace的Label篩選合格Pod,實現以下功能:

  • 添加Annotation和Label

  • 執行調度策略

    支援的調度策略如下:

    策略

    說明

    fair

    公平調度,由kube-scheduler決定調度到標準Node或VNode。

    normalNodePrefer

    優先調度到標準Node,標準Node資源不足時允許調度到VNode。

    virtualNodeOnly

    Pod只會調度到VNode。

說明

本文部署的eci-profile組件通過Selector CRD實現Pod自動調度。如果您之前已經部署了舊版eci-profile組件(通過ConfigMap實現),仍可以繼續使用。建議您升級到Selector CRD模式,後續ConfigMap模式不再支援新增功能。更多資訊,請參見升級說明

部署eci-profile組件

通過VNodectl工具

如果您已安裝配置VNodectl工具,可以執行以下命令一鍵部署eci-profile。

  1. 部署eci-profile。

    vnode addon enable eci-profile --kubeconfig /path/to/kubeconfig
  2. 查看eci-profile部署狀態。

    vnode addon list

    預期返回如下,其中eci-profile的STATUS為enabled。

    |----------------|------------|------------|-------------------------------------------------|
    |   ADDON NAME   |   STATUS   | MAINTAINER |                   REPOSITORY                    |
    |----------------|------------|------------|-------------------------------------------------|
    | eci-profile    | enabled ✅ | ECI Group  | https://github.com/aliyuneci/eci-profile.git    |
    | vnode-approver | enabled ✅ | ECI Group  | https://github.com/aliyuneci/vnode-approver.git |
    |----------------|------------|------------|-------------------------------------------------|

手動部署

  1. 將以下內容儲存為eci-profile.yaml。

    請根據您的叢集版本複製對應YAML。

    叢集版本為1.16及以上

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: eci-profile
    rules:
      - apiGroups:
          - ""
        resources:
          - nodes
          - namespaces
          - resourcequotas
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - pods
        verbs:
          - get
          - list
          - watch
          - create
          - patch
      - apiGroups:
          - "admissionregistration.k8s.io"
        resources:
          - mutatingwebhookconfigurations
        verbs:
          - get
          - patch
          - create
          - delete
      - apiGroups:
          - "eci.aliyun.com"
        resources:
          - selectors
        verbs:
          - get
          - watch
          - list
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: eci-profile
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: eci-profile
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: eci-profile
    subjects:
    - kind: ServiceAccount
      name: eci-profile
      namespace: kube-system
    ---
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: (devel)
      creationTimestamp: null
      name: selectors.eci.aliyun.com
    spec:
      group: eci.aliyun.com
      names:
        kind: Selector
        listKind: SelectorList
        plural: selectors
        singular: selector
      scope: Namespaced
      versions:
      - name: v1
        schema:
          openAPIV3Schema:
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                properties:
                  effect:
                    properties:
                      annotations:
                        additionalProperties:
                          type: string
                        type: object
                      labels:
                        additionalProperties:
                          type: string
                        type: object
                    type: object
                  namespaceLabels:
                    description: A label selector is a label query over a set of resources.
                      The result of matchLabels and matchExpressions are ANDed. An empty
                      label selector matches all objects. A null label selector matches
                      no objects.
                    properties:
                      matchExpressions:
                        description: matchExpressions is a list of label selector requirements.
                          The requirements are ANDed.
                        items:
                          description: A label selector requirement is a selector that
                            contains values, a key, and an operator that relates the key
                            and values.
                          properties:
                            key:
                              description: key is the label key that the selector applies
                                to.
                              type: string
                            operator:
                              description: operator represents a key's relationship to
                                a set of values. Valid operators are In, NotIn, Exists
                                and DoesNotExist.
                              type: string
                            values:
                              description: values is an array of string values. If the
                                operator is In or NotIn, the values array must be non-empty.
                                If the operator is Exists or DoesNotExist, the values
                                array must be empty. This array is replaced during a strategic
                                merge patch.
                              items:
                                type: string
                              type: array
                          required:
                          - key
                          - operator
                          type: object
                        type: array
                      matchLabels:
                        additionalProperties:
                          type: string
                        description: matchLabels is a map of {key,value} pairs. A single
                          {key,value} in the matchLabels map is equivalent to an element
                          of matchExpressions, whose key field is "key", the operator
                          is "In", and the values array contains only "value". The requirements
                          are ANDed.
                        type: object
                    type: object
                    x-kubernetes-map-type: atomic
                  objectLabels:
                    description: A label selector is a label query over a set of resources.
                      The result of matchLabels and matchExpressions are ANDed. An empty
                      label selector matches all objects. A null label selector matches
                      no objects.
                    properties:
                      matchExpressions:
                        description: matchExpressions is a list of label selector requirements.
                          The requirements are ANDed.
                        items:
                          description: A label selector requirement is a selector that
                            contains values, a key, and an operator that relates the key
                            and values.
                          properties:
                            key:
                              description: key is the label key that the selector applies
                                to.
                              type: string
                            operator:
                              description: operator represents a key's relationship to
                                a set of values. Valid operators are In, NotIn, Exists
                                and DoesNotExist.
                              type: string
                            values:
                              description: values is an array of string values. If the
                                operator is In or NotIn, the values array must be non-empty.
                                If the operator is Exists or DoesNotExist, the values
                                array must be empty. This array is replaced during a strategic
                                merge patch.
                              items:
                                type: string
                              type: array
                          required:
                          - key
                          - operator
                          type: object
                        type: array
                      matchLabels:
                        additionalProperties:
                          type: string
                        description: matchLabels is a map of {key,value} pairs. A single
                          {key,value} in the matchLabels map is equivalent to an element
                          of matchExpressions, whose key field is "key", the operator
                          is "In", and the values array contains only "value". The requirements
                          are ANDed.
                        type: object
                    type: object
                    x-kubernetes-map-type: atomic
                  policy:
                    properties:
                      fair:
                        type: object
                      namespaceResourceLimit:
                        properties:
                          limits:
                            additionalProperties:
                              anyOf:
                              - type: integer
                              - type: string
                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                              x-kubernetes-int-or-string: true
                            description: ResourceList is a set of (resource name, quantity)
                              pairs.
                            type: object
                          namespace:
                            type: string
                        required:
                        - limits
                        - namespace
                        type: object
                      normalNodeOnly:
                        type: object
                      normalNodePrefer:
                        properties:
                          cpuRatio:
                            type: integer
                          memoryRatio:
                            type: integer
                        type: object
                      virtualNodeOnly:
                        type: object
                    type: object
                  priority:
                    format: int32
                    type: integer
                type: object
            required:
            - spec
            type: object
        served: true
        storage: true
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        role: eci-profile
      name: eci-profile
      namespace: kube-system
    spec:
      ports:
        - port: 443
          targetPort: 443
      selector:
        app: eci-profile
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: eci-profile
      namespace: kube-system
      labels:
        app: eci-profile
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: eci-profile
      template:
        metadata:
          labels:
            app: eci-profile
        spec:
          serviceAccount: eci-profile
          containers:
          - name: eci-profile
            image: registry.cn-beijing.aliyuncs.com/eci-release/eci-profile:0.0.3
            imagePullPolicy: Always
            resources:
              requests: 
                cpu: 2
                memory: 4Gi   
              limits:
                cpu: 4    
                memory: 8Gi   
            env:
            - name: KUBERNETES_MASTER
              value: https://kubernetes:443

    叢集版本為1.16以下

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: eci-profile
    rules:
    - apiGroups:
        - ""
      resources:
        - nodes
        - namespaces
        - resourcequotas
      verbs:
        - get
        - list
        - watch
    - apiGroups:
        - ""
      resources:
        - pods
      verbs:
        - get
        - list
        - watch
        - create
        - patch
    - apiGroups:
        - "admissionregistration.k8s.io"
      resources:
        - mutatingwebhookconfigurations
      verbs:
        - get
        - patch
        - create
        - delete
    - apiGroups:
      - "eci.aliyun.com"
      resources:
        - selectors
      verbs:
        - get
        - watch
        - list
    ---
    apiVersion: apiextensions.k8s.io/v1beta1
    #apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      name: selectors.eci.aliyun.com
    spec:
      group: eci.aliyun.com
      version: v1beta1
      names:
        kind: Selector
        plural: selectors
        shortNames:
          - selectors
        categories:
          - all
      scope: Cluster
      validation:
        openAPIV3Schema:
          type: object
          required:
          - metadata
          - spec
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              type: object
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: eci-profile
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: eci-profile
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: eci-profile
    subjects:
    - kind: ServiceAccount
      name: eci-profile
      namespace: kube-system
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        role: eci-profile
      name: eci-profile
      namespace: kube-system
    spec:
      ports:
      - port: 443
        targetPort: 443
      selector:
        app: eci-profile
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: eci-profile
      namespace: kube-system
      labels:
        app: eci-profile
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: eci-profile
      template:
        metadata:
          labels:
            app: eci-profile
        spec:
          serviceAccount: eci-profile
          containers:
          - name: eci-profile
            image: registry.cn-beijing.aliyuncs.com/eci-release/eci-profile:2.0.0-477875b-aliyun
            imagePullPolicy: Always
            resources:
              requests: 
                cpu: 2
                memory: 4Gi   
              limits:
                cpu: 4    
                memory: 8Gi   
            env:
            - name: KUBERNETES_MASTER
              value: https://kubernetes:443

  2. 部署eci-profile組件。

    kubectl create -f eci-profile.yaml
  3. 查看部署結果。

    kubectl -n kube-system get pods

    預期返回如下,eci-profile對應Pod的狀態為Running。

    NAME                                                     READY   STATUS    RESTARTS   AGE
    eci-profile-6454756cb8-8xlz8                   1/1      Running    0                76s

配置說明及樣本

部署eci-profile組件後,您可以建立Selector來配置Pod的調度策略,以及需要添加的Annotation和Label。Selector的YAML配置如下:

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-fair
spec:
  objectLabels:
    matchLabels:
      app: nginx
  namespaceLabels:
    matchLabels:
      app: test
  effect:
    annotations:
      k8s.aliyun.com/eci-auto-imc: "true"
    labels:
      eci-schedulable: "true"
  policy:
    fair: {}
  priority: 3

spec中的相關參數說明如下:

名稱

描述

objectLabels.matchLabels

要匹配的Pod Label。

namespaceLabels.matchLabels

要匹配的Namespace Label。

effect.annotations

要添加的Annotation。

effect.labels

要添加的Label。

policy

調度策略。支援以下幾種:

  • fair

  • normalNodePrefer

  • virtualNodeOnly

priority

優先順序。存在多個Selector且發生衝突時,優先順序高的Selector會生效。數值越大,表示該Selector優先順序越高。

說明

objectLabels和namespaceLabels中,至少要選擇一個配置。如果同時配置了兩者,則Pod需要同時匹配兩者。

配置樣本一:fair調度

建立以下Selector,eci-profile會為帶有app: nginx Label的Pod添加VNode Toleration,由kube-scheduler決定調度,同時添加effect中定義的Annotation和Label到Pod。

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-fair
spec:
  objectLabels:
    matchLabels:
      app: nginx
  effect:
    annotations:
      k8s.aliyun.com/eci-auto-imc: "true"
    labels:
      eci-schedulable: "true"
  policy:
    fair: {}

配置樣本二:normalNodePrefer調度

建立以下Selector,eci-profile會在標準節點資源不足時將帶有app: nginx Label的Pod調度到VNode,同時添加effect中定義的Annotation和Label到Pod。

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-normal-node-prefer
spec:
  objectLabels:
    matchLabels:
      app: nginx
  effect:
    annotations:
      k8s.aliyun.com/eci-image-cache: "true"
    labels:
      eci-schedulable: "true"
  policy:
    normalNodePrefer: {}

配置樣本三:virtualNodeOnly調度

建立以下Selector,eci-profile會為帶有app: nginx Label的Pod添加VNode Toleration和VNode nodeSelector,同時添加effect中定義的Annotation和Label到Pod。

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-virtual-node-only
spec:
  objectLabels:
    matchLabels:
      app: nginx
  effect:
    annotations:
      k8s.aliyun.com/eci-auto-imc: "true"
    labels:
      eci-schedulable: "true"
  policy:
    virtualNodeOnly: {}

升級說明

如果您之前已經部署了舊版eci-profile組件(ConfigMap模式),建議您升級到新版eci-profile組件(Selector CRD模式)。升級方式如下:

  1. 記錄kube-system命名空間下,ConfigMap eci-profile中的selectors內容。

  2. 刪除舊的eci-profie組件。

  3. 部署新的eci-profile組件。

  4. 根據原有的selectors內容,建立新的Selector資源。

說明

如果您在使用eci-profile的過程中遇到任何需求或問題時,歡迎使用DingTalk搜尋群號44666389,加入VNode客戶支援群。