如何部署eci-profile组件和使用eci-profile - 弹性容器实例

eci-profile可以提供集群维度的ECI资源视图，支持根据Selector实现Pod编排。本文介绍如何部署eci-profile组件，以及如何使用eci-profile。

eci-profile功能说明

eci-profile可以通过Pod、Namespace的Label筛选符合条件的Pod，实现以下功能：

添加Annotation和Label

执行调度策略

支持的调度策略如下：

策略	说明
fair	公平调度，由kube-scheduler决定调度到标准Node或VNode。
normalNodePrefer	优先调度到标准Node，标准Node资源不足时允许调度到VNode。
virtualNodeOnly	Pod只会调度到VNode。

说明

本文部署的eci-profile组件通过Selector CRD实现Pod自动调度。如果您之前已经部署了旧版eci-profile组件（通过ConfigMap实现），仍可以继续使用。建议您升级到Selector CRD模式，后续ConfigMap模式不再支持新增功能。更多信息，请参见升级说明。

部署eci-profile组件

通过VNodectl工具

如果您已安装配置VNodectl工具，可以执行以下命令一键部署eci-profile。

部署eci-profile。

vnode addon enable eci-profile --kubeconfig /path/to/kubeconfig

查看eci-profile部署状态。

vnode addon list

预期返回如下，其中eci-profile的STATUS为enabled。

|----------------|------------|------------|-------------------------------------------------|
|   ADDON NAME   |   STATUS   | MAINTAINER |                   REPOSITORY                    |
|----------------|------------|------------|-------------------------------------------------|
| eci-profile    | enabled ✅ | ECI Group  | https://github.com/aliyuneci/eci-profile.git    |
| vnode-approver | enabled ✅ | ECI Group  | https://github.com/aliyuneci/vnode-approver.git |
|----------------|------------|------------|-------------------------------------------------|

手动部署

将以下内容保存为eci-profile.yaml。

请根据您的集群版本复制对应YAML。

集群版本为1.16及以上

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eci-profile
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
      - namespaces
      - resourcequotas
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
      - create
      - patch
  - apiGroups:
      - "admissionregistration.k8s.io"
    resources:
      - mutatingwebhookconfigurations
    verbs:
      - get
      - patch
      - create
      - delete
  - apiGroups:
      - "eci.aliyun.com"
    resources:
      - selectors
    verbs:
      - get
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eci-profile
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eci-profile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eci-profile
subjects:
- kind: ServiceAccount
  name: eci-profile
  namespace: kube-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
  creationTimestamp: null
  name: selectors.eci.aliyun.com
spec:
  group: eci.aliyun.com
  names:
    kind: Selector
    listKind: SelectorList
    plural: selectors
    singular: selector
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              effect:
                properties:
                  annotations:
                    additionalProperties:
                      type: string
                    type: object
                  labels:
                    additionalProperties:
                      type: string
                    type: object
                type: object
              namespaceLabels:
                description: A label selector is a label query over a set of resources.
                  The result of matchLabels and matchExpressions are ANDed. An empty
                  label selector matches all objects. A null label selector matches
                  no objects.
                properties:
                  matchExpressions:
                    description: matchExpressions is a list of label selector requirements.
                      The requirements are ANDed.
                    items:
                      description: A label selector requirement is a selector that
                        contains values, a key, and an operator that relates the key
                        and values.
                      properties:
                        key:
                          description: key is the label key that the selector applies
                            to.
                          type: string
                        operator:
                          description: operator represents a key's relationship to
                            a set of values. Valid operators are In, NotIn, Exists
                            and DoesNotExist.
                          type: string
                        values:
                          description: values is an array of string values. If the
                            operator is In or NotIn, the values array must be non-empty.
                            If the operator is Exists or DoesNotExist, the values
                            array must be empty. This array is replaced during a strategic
                            merge patch.
                          items:
                            type: string
                          type: array
                      required:
                      - key
                      - operator
                      type: object
                    type: array
                  matchLabels:
                    additionalProperties:
                      type: string
                    description: matchLabels is a map of {key,value} pairs. A single
                      {key,value} in the matchLabels map is equivalent to an element
                      of matchExpressions, whose key field is "key", the operator
                      is "In", and the values array contains only "value". The requirements
                      are ANDed.
                    type: object
                type: object
                x-kubernetes-map-type: atomic
              objectLabels:
                description: A label selector is a label query over a set of resources.
                  The result of matchLabels and matchExpressions are ANDed. An empty
                  label selector matches all objects. A null label selector matches
                  no objects.
                properties:
                  matchExpressions:
                    description: matchExpressions is a list of label selector requirements.
                      The requirements are ANDed.
                    items:
                      description: A label selector requirement is a selector that
                        contains values, a key, and an operator that relates the key
                        and values.
                      properties:
                        key:
                          description: key is the label key that the selector applies
                            to.
                          type: string
                        operator:
                          description: operator represents a key's relationship to
                            a set of values. Valid operators are In, NotIn, Exists
                            and DoesNotExist.
                          type: string
                        values:
                          description: values is an array of string values. If the
                            operator is In or NotIn, the values array must be non-empty.
                            If the operator is Exists or DoesNotExist, the values
                            array must be empty. This array is replaced during a strategic
                            merge patch.
                          items:
                            type: string
                          type: array
                      required:
                      - key
                      - operator
                      type: object
                    type: array
                  matchLabels:
                    additionalProperties:
                      type: string
                    description: matchLabels is a map of {key,value} pairs. A single
                      {key,value} in the matchLabels map is equivalent to an element
                      of matchExpressions, whose key field is "key", the operator
                      is "In", and the values array contains only "value". The requirements
                      are ANDed.
                    type: object
                type: object
                x-kubernetes-map-type: atomic
              policy:
                properties:
                  fair:
                    type: object
                  namespaceResourceLimit:
                    properties:
                      limits:
                        additionalProperties:
                          anyOf:
                          - type: integer
                          - type: string
                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                          x-kubernetes-int-or-string: true
                        description: ResourceList is a set of (resource name, quantity)
                          pairs.
                        type: object
                      namespace:
                        type: string
                    required:
                    - limits
                    - namespace
                    type: object
                  normalNodeOnly:
                    type: object
                  normalNodePrefer:
                    properties:
                      cpuRatio:
                        type: integer
                      memoryRatio:
                        type: integer
                    type: object
                  virtualNodeOnly:
                    type: object
                type: object
              priority:
                format: int32
                type: integer
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
---
apiVersion: v1
kind: Service
metadata:
  labels:
    role: eci-profile
  name: eci-profile
  namespace: kube-system
spec:
  ports:
    - port: 443
      targetPort: 443
  selector:
    app: eci-profile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-profile
  namespace: kube-system
  labels:
    app: eci-profile
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eci-profile
  template:
    metadata:
      labels:
        app: eci-profile
    spec:
      serviceAccount: eci-profile
      containers:
      - name: eci-profile
        image: registry.cn-beijing.aliyuncs.com/eci-release/eci-profile:0.0.3
        imagePullPolicy: Always
        resources:
          requests: 
            cpu: 2
            memory: 4Gi   
          limits:
            cpu: 4    
            memory: 8Gi   
        env:
        - name: KUBERNETES_MASTER
          value: https://kubernetes:443

集群版本为1.16以下

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eci-profile
rules:
- apiGroups:
    - ""
  resources:
    - nodes
    - namespaces
    - resourcequotas
  verbs:
    - get
    - list
    - watch
- apiGroups:
    - ""
  resources:
    - pods
  verbs:
    - get
    - list
    - watch
    - create
    - patch
- apiGroups:
    - "admissionregistration.k8s.io"
  resources:
    - mutatingwebhookconfigurations
  verbs:
    - get
    - patch
    - create
    - delete
- apiGroups:
  - "eci.aliyun.com"
  resources:
    - selectors
  verbs:
    - get
    - watch
    - list
---
apiVersion: apiextensions.k8s.io/v1beta1
#apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: selectors.eci.aliyun.com
spec:
  group: eci.aliyun.com
  version: v1beta1
  names:
    kind: Selector
    plural: selectors
    shortNames:
      - selectors
    categories:
      - all
  scope: Cluster
  validation:
    openAPIV3Schema:
      type: object
      required:
      - metadata
      - spec
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          type: object
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eci-profile
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eci-profile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eci-profile
subjects:
- kind: ServiceAccount
  name: eci-profile
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    role: eci-profile
  name: eci-profile
  namespace: kube-system
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: eci-profile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-profile
  namespace: kube-system
  labels:
    app: eci-profile
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eci-profile
  template:
    metadata:
      labels:
        app: eci-profile
    spec:
      serviceAccount: eci-profile
      containers:
      - name: eci-profile
        image: registry.cn-beijing.aliyuncs.com/eci-release/eci-profile:2.0.0-477875b-aliyun
        imagePullPolicy: Always
        resources:
          requests: 
            cpu: 2
            memory: 4Gi   
          limits:
            cpu: 4    
            memory: 8Gi   
        env:
        - name: KUBERNETES_MASTER
          value: https://kubernetes:443

部署eci-profile组件。
```
kubectl create -f eci-profile.yaml
```

查看部署结果。

kubectl -n kube-system get pods

预期返回如下，eci-profile对应Pod的状态为Running。

NAME                                                     READY   STATUS    RESTARTS   AGE
eci-profile-6454756cb8-8xlz8                   1/1      Running    0                76s

配置说明及示例

部署eci-profile组件后，您可以创建Selector来配置Pod的调度策略，以及需要添加的Annotation和Label。Selector的YAML配置如下：

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-fair
spec:
  objectLabels:
    matchLabels:
      app: nginx
  namespaceLabels:
    matchLabels:
      app: test
  effect:
    annotations:
      k8s.aliyun.com/eci-auto-imc: "true"
    labels:
      eci-schedulable: "true"
  policy:
    fair: {}
  priority: 3

spec中的相关参数说明如下：

名称	描述
objectLabels.matchLabels	要匹配的Pod Label。
namespaceLabels.matchLabels	要匹配的Namespace Label。
effect.annotations	要添加的Annotation。
effect.labels	要添加的Label。
policy	调度策略。支持以下几种： fair normalNodePrefer virtualNodeOnly
priority	优先级。存在多个Selector且发生冲突时，优先级高的Selector会生效。数值越大，表示该Selector优先级越高。

说明

objectLabels和namespaceLabels中，至少要选择一个配置。如果同时配置了两者，则Pod需要同时匹配两者。

配置示例一：fair调度

创建以下Selector，eci-profile会为带有app: nginx Label的Pod添加VNode Toleration，由kube-scheduler决定调度，同时添加effect中定义的Annotation和Label到Pod。

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-fair
spec:
  objectLabels:
    matchLabels:
      app: nginx
  effect:
    annotations:
      k8s.aliyun.com/eci-auto-imc: "true"
    labels:
      eci-schedulable: "true"
  policy:
    fair: {}

配置示例二：normalNodePrefer调度

创建以下Selector，eci-profile会在标准节点资源不足时将带有app: nginx Label的Pod调度到VNode，同时添加effect中定义的Annotation和Label到Pod。

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-normal-node-prefer
spec:
  objectLabels:
    matchLabels:
      app: nginx
  effect:
    annotations:
      k8s.aliyun.com/eci-image-cache: "true"
    labels:
      eci-schedulable: "true"
  policy:
    normalNodePrefer: {}

配置示例三：virtualNodeOnly调度

创建以下Selector，eci-profile会为带有app: nginx Label的Pod添加VNode Toleration和VNode nodeSelector，同时添加effect中定义的Annotation和Label到Pod。

apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
  name: test-virtual-node-only
spec:
  objectLabels:
    matchLabels:
      app: nginx
  effect:
    annotations:
      k8s.aliyun.com/eci-auto-imc: "true"
    labels:
      eci-schedulable: "true"
  policy:
    virtualNodeOnly: {}

升级说明

如果您之前已经部署了旧版eci-profile组件（ConfigMap模式），建议您升级到新版eci-profile组件（Selector CRD模式）。升级方式如下：

记录kube-system命名空间下，ConfigMap eci-profile中的selectors内容。
删除旧的eci-profie组件。
部署新的eci-profile组件。
根据原有的selectors内容，创建新的Selector资源。

说明

如果您在使用eci-profile的过程中遇到任何需求或问题时，欢迎使用钉钉搜索群号44666389，加入VNode客户支持群。