使用ECI彈性資源運行Spark作業

本文介紹如何在ACK叢集中使用Elastic Container Instance運行Spark作業。通過使用ECI彈性資源並配置合適的調度策略，您可以按需建立ECI Pod，並按資源使用量按需付費，從而有效減少資源閑置帶來的成本浪費，進而更加經濟高效地運行Spark作業。

前提條件

已部署ack-spark-operator組件，請參見部署ack-spark-operator組件。
已部署使用ECI資源所需的ack-virtual-node組件，請參見部署ack-virtual-node組件。
已通過kubectl工具串連叢集。具體操作，請參見擷取叢集KubeConfig並通過kubectl工具串連叢集。

使用ECI運行Spark作業的優勢

Elastic Container Instance為ACK叢集提供基礎的容器Pod運行環境，通過動態調度Spark作業中的Driver和Executor Pod到ECI上，可以實現無伺服器Spark作業執行。每個容器執行個體通過輕量級虛擬化安全沙箱技術實現完全隔離，確保互不干擾。

在ECI上運行Spark作業具備以下優勢：

超大容量：叢集無需額外配置即可輕鬆獲得最多50,000個Pod容量，無需提前規劃容量。
秒級彈性：始終確保在極短時間內建立出數千Pod，無需擔心突發業務流量因Pod建立時延受到影響。
節約成本：ECI Pod按需建立並隨用隨付，避免資源閑置造成的浪費。支援Spot執行個體及多種執行個體組合，進一步降低成本。

配置ECS和ECI資源使用原則

您可以通過配置汙點（Taints）、容忍（Tolerations）和節點親和性（nodeAffinity）來聲明只使用ECS資源或者ECI彈性資源，您還可以優先使用ECS資源，當ECS資源不足時自動申請ECI資源。關於如何指定ECS和ECI的資源分派，請參見指定ECS和ECI的資源分派。以下是配置樣本。

只使用ECS資源

ECI節點預設會被標記汙點virtual-kubelet.io/provider=alibabacloud:NoSchedule，預設情況下不會使用ECI資源。以下的SparkApplication將只使用到ECS資源。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-ecs-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
  executor:
    instances: 2
    cores: 2
    memory: 4g

只使用ECI彈性資源

通過容忍ECI節點上的汙點使得作業可以調度到ECI上執行，並通過添加節點親和性使得作業只使用ECI彈性資源。virtual-kubelet.io/provider=alibabacloud:NoSchedule的汙點，從而只使用ECI彈性資源。以下的SparkApplication將只使用ECI彈性資源。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-eci-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        # 調度到ECI節點上
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        # 調度到ECI節點上。
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點。
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

優先使用ECS資源，當ECS資源不足時自動申請ECI資源

一種常見的彈性策略是優先使用ECS資源，當ECS資源不足時，自動使用ECI彈性資源。這種策略可以減少常駐資源閑置造成的浪費，還能在業務流量高峰期提供額外的彈性資源。下面的SparkApplication樣本展示了如何配置這種彈性策略。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-ecs-first
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        # 優先調度到非ECI節點上。
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: type
              operator: NotIn
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點。
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        # 優先調度到非 ECI 節點上
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: type
              operator: NotIn
              values:
              - virtual-kubelet
    tolerations:
    # 容忍 ECI 汙點
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

自訂彈性資源優先順序調度

ACK調度器支援自訂彈性資源優先順序調度策略。通過自訂ResourcePolicy，您可以設定Pod在不同類型節點上的調度優先順序。詳細資料，請參見自訂彈性資源優先順序調度。

建立如下樣本ResourcePolicy資訊清單檔並儲存為resourcepolicy.yaml，該檔案定義了每個Spark作業的Pod按照一定的策略去使用ECS/ECI資源。resourcepolicy.yaml的ResourcePolicy資訊清單檔，用於定義Spark作業啟動Pod的資源調度策略。

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: sparkapplication-resource-policy
  namespace: default                      # 指定調度策略僅作用於命名空間default中的Pod。
spec:
  ignorePreviousPod: true     
  ignoreTerminatingPod: false     
  matchLabelKeys:
  - sparkoperator.k8s.io/submission-id    # 根據Spark作業的Submission ID進行Pod計數分組。
  preemptPolicy: AfterAllUnits            # 調度策略中的搶佔策略，表示在所有調度單元消耗完之前不進行搶佔。
  selector:                               # 指定調度策略作用於同一命名空間中的哪些Pod。
    sparkoperator.k8s.io/launched-by-spark-operator: "true"  # 僅作用於由Spark Operator啟動的Pod。
  strategy: prefer              
  units:                                 # 定義了兩個調度單元。調度單元的順序決定資源擴縮容的順序：擴容按順序進行，縮容按逆序進行。
  - max: 2                               # 第一個單元：最多調度2個Pod至具有kubernetes.io/arch=amd64標籤ECS節點。
    resource: ecs               
    nodeSelector:
      kubernetes.io/arch: amd64  
  - max: 3                               # 第二個單元：最多調度3個Pod至ECI節點。
    resource: eci

執行以下命令，建立一個僅作用於Spark作業啟動Pod的調度策略。
```
kubectl apply -f resourcepolicy.yaml
```

建立如下SparkApplication資訊清單檔，並儲存為spark-pi.yaml。

在資源充足的情況下，調度策略的結果將為1個Driver Pod和1個Executor Pod被調度到AMD 64架構的ECS節點，3個Executor Pod被調度到ECI節點，另有1個Executor Pod因超出最大數量限制而處於Pending狀態。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    tolerations:
    - key: virtual-kubelet.io/provider       # 容忍ECI汙點。
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    instances: 5
    cores: 1
    coreLimit: 1200m
    memory: 512m
    tolerations: 
    - key: virtual-kubelet.io/provider         # 容忍ECI汙點。
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

執行以下命令，提交Spark作業。
```
kubectl apply -f spark-pi.yaml
```

查看Pod調度情況確實符合預期。

kubectl get pods  -o wide -l sparkoperator.k8s.io/app-name=spark-pi

NAME                                        READY   STATUS      RESTARTS   AGE       IP                  NODE                          
spark-pi-34c0998f9f832e61-exec-1            1/1     Running     0          28s       192.XXX.XX.34       cn-beijing.192.XXX.XX.250       
spark-pi-34c0998f9f832e61-exec-2            1/1     Running     0          28s       192.XXX.XX.87       virtual-kubelet-cn-beijing-i   
spark-pi-34c0998f9f832e61-exec-3            1/1     Running     0          28s       192.XXX.XX.88       virtual-kubelet-cn-beijing-i   
spark-pi-34c0998f9f832e61-exec-4            1/1     Running     0          28s       192.XXX.XX.86       virtual-kubelet-cn-beijing-i   
spark-pi-34c0998f9f832e61-exec-5            0/1     Pending     0          28s       <none>              <none>                         
spark-pi-driver                             1/1     Running     0          34s       192.XXX.XX.37       cn-beijing.192.XXX.XXX.250

使用ImageCache加速鏡像拉取

ECI支援鏡像緩衝功能，能夠加速鏡像拉取，從而提升Pod建立速度。更多資訊，請參見使用ImageCache加速建立ECI Pod。

下文以SparkApplication為例，比較正常情況下的拉取時間與使用鏡像緩衝時的拉取時間，以及在實際環境如何使用自動建立和匹配鏡像緩衝。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
  executor:
    instances: 2
    cores: 2
    memory: 4g

第一次提交作業（未使用鏡像緩衝）

在不使用鏡像緩衝的情況下提交該作業，然後查看Driver Pod的事件。

kubectl describe pod spark-pi-driver

Events:
  ...
  Warning  ImageCacheMissed       24m   EciService         [eci.imagecache]Missed image cache.
  Normal   ImageCacheAutoCreated  24m   EciService         [eci.imagecache]Image cache imc-2zeXXXXXXXXXXXXXXXXX is auto created
  Normal   Pulling                24m   kubelet            Pulling image "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2"
  Normal   Pulled                 23m   kubelet            Successfully pulled image "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2" in 1m41.289s (1m41.289s including waiting)
  ...

事件顯示ImageCache未命中，並建立了新的ImageCache，拉取鏡像總共花了100秒左右。

第二次提交作業（使用鏡像緩衝）

我們在Driver和Executor中添加如下註解以明確指定該鏡像緩衝。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-eci-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    annotations:
      # 手動指定鏡像緩衝ID
      k8s.aliyun.com/eci-image-snapshot-id: imc-2zeXXXXXXXXXXXXXXXXX
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        # 調度到ECI節點上
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    annotations:
      # 手動指定鏡像緩衝ID
      k8s.aliyun.com/eci-image-snapshot-id: imc-2zeXXXXXXXXXXXXXXXXX
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        # 調度到ECI節點上
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

運行作業並查看Driver Pod的事件記錄。

 kubectl describe pod spark-pi-driver

Events:
  ...
  Normal  SuccessfulHitImageCache  23s   EciService         [eci.imagecache]Successfully hit image cache imc-2zeXXXXXXXXXXXXXXXXX, eci will be scheduled with this image cache.
  Normal  Pulled                   4s    kubelet            Container image "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2" already present on machine
  ...

事件顯示鏡像緩衝成功命中，無需再次拉取鏡像。

自動建立和匹配鏡像緩衝

在實際使用過程中，您可以在.spec.[driver|executor].annotations中添加註解k8s.aliyun.com/eci-image-cache: "true"以實現自動建立並匹配鏡像緩衝，而無需明確指定鏡像緩衝ID。

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-eci-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.2
  driver:
    annotations:
      # 自動建立並匹配鏡像緩衝
      k8s.aliyun.com/eci-image-cache: "true"
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        # 調度到ECI節點上
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    annotations:
      # 自動建立並匹配鏡像緩衝
      k8s.aliyun.com/eci-image-cache: "true"
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        # 調度到ECI節點上
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # 容忍ECI汙點
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule