E-MapReduce：使用kubectl管理作業

操作步驟

1. 通過kubectl串連Kubernetes叢集，詳情請參見擷取叢集KubeConfig並通過kubectl工具串連叢集。

   您也可以通過API等方式串連Kubernetes叢集，詳情請參見使用Kubernetes API。

2. 執行以下命令，管理作業。

   • 您可以執行以下命令，查看作業狀態。

     kubectl describe SparkApplication <作業名> --namespace <叢集對應的namespace>

     返回資訊如下所示。

     Name:         spark-pi-simple
     Namespace:    c-48e779e0d9ad****
     Labels:       <none>
     Annotations:  <none>
     API Version:  sparkoperator.k8s.io/v1beta2
     Kind:         SparkApplication
     Metadata:
       Creation Timestamp:  2021-07-22T06:25:33Z
       Generation:          1
       Resource Version:  7503740
       UID:               930874ad-bb17-47f1-a556-55118c1d****
     Spec:
       Arguments:
         1000
       Driver:
         Core Limit:  1000m
         Cores:       1
         Memory:      4g
       Executor:
         Core Limit:           1000m
         Cores:                1
         Instances:            1
         Memory:               8g
         Memory Overhead:      1g
       Image:                  registry-vpc.cn-hangzhou.aliyuncs.com/emr/spark:emr-2.4.5-1.0.0
       Main Application File:  local:///opt/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.4.5.jar
       Main Class:             org.apache.spark.examples.SparkPi
       Spark Version:          2.4.5
       Type:                   Scala
     Status:
       Application State:
         State:  RUNNING
       Driver Info:
         Pod Name:                spark-pi-simple-driver
         Web UI Address:          172.16.230.240:4040
         Web UI Ingress Address:  spark-pi-simple.c-48e779e0d9ad4bfd.c7f6b768c34764c27ab740bdb1fc2a3ff.cn-hangzhou.alicontainer.com
         Web UI Ingress Name:     spark-pi-simple-ui-ingress
         Web UI Port:             4040
         Web UI Service Name:     spark-pi-simple-ui-svc
       Execution Attempts:        1
       Executor State:
         spark-pi-1626935142670-exec-1:  RUNNING
       Last Submission Attempt Time:     2021-07-22T06:25:33Z
       Spark Application Id:             spark-15b44f956ecc40b1ae59a27ca18d****
       Submission Attempts:              1
       Submission ID:                    d71f30e2-9bf8-4da1-8412-b585fd45****
       Termination Time:                 <nil>
     Events:
       Type    Reason                     Age   From            Message
       ----    ------                     ----  ----            -------
       Normal  SparkApplicationAdded      17s   spark-operator  SparkApplication spark-pi-simple was added, enqueuing it for submission
       Normal  SparkApplicationSubmitted  14s   spark-operator  SparkApplication spark-pi-simple was submitted successfully
       Normal  SparkDriverRunning         13s   spark-operator  Driver spark-pi-simple-driver is running
       Normal  SparkExecutorPending       7s    spark-operator  Executor spark-pi-1626935142670-exec-1 is pending
       Normal  SparkExecutorRunning       6s    spark-operator  Executor spark-pi-1626935142670-exec-1 is running

     本文範例程式碼中的<作業名>，您可以登入E-MapReduce on ACK控制台，在作業詳情頁面，查看已建立的作業名。

   • 您可以執行以下命令，停止並刪除作業。

     kubectl delete SparkApplication <作業名> -n <叢集對應的namespace>

     返回資訊如下所示。

     sparkapplication.sparkoperator.k8s.io "spark-pi-simple" deleted

   • 您可以執行以下命令，查看作業日誌。

     kubectl logs <作業名-driver> -n <叢集對應的namespace>

     說明

     例如，作業名為spark-pi-simple，叢集對應的namespace為c-d2232227b951****，則對應的命令為kubectl logs spark-pi-simple-driver -n c-d2232227b951****。

     返回如下類似資訊。

     ......
     Pi is roughly 3.141488791414888
     21/07/22 14:37:57 INFO SparkContext: Successfully stopped SparkContext
     21/07/22 14:37:57 INFO ShutdownHookManager: Shutdown hook called
     21/07/22 14:37:57 INFO ShutdownHookManager: Deleting directory /var/data/spark-b6a43b55-a354-44d7-ae5e-45b8b1493edb/spark-56aae0d1-37b9-4a7d-9c99-4e4ca12deb4b
     21/07/22 14:37:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-e2500491-6ed7-48d7-b94e-a9ebeb899320