E-MapReduce:Manage Spark Thrift Servers

Create a token

1. On the Spark Thrift Server Sessions tab, click the name of the created Spark Thrift Server.

2. On the page that appears, click the Tokens tab.

3. On the Tokens tab, click Create Token.

4. In the Create Token dialog box, configure parameters and click OK. The following table describes the parameters.

   Parameter	Description
   Name	The name of the token.
   Expired At	The validity period of the token. The validity period must be greater than or equal to 1 day. By default, this parameter is enabled and set to 365 days.

5. Copy the token.

   Important

   After the token is created, you must immediately copy the token. You can no longer view the token after you leave the page. If your token expires or is lost, reset the token or create another token.

Use Python to connect to the Spark Thrift Server

1. Run the following command to install PyHive and Thrift:

   pip install pyhive thrift

2. Write a Python script to connect to the Spark Thrift Server.

   The following Python sample code provides an example on how to connect to Hive and query databases.

   from pyhive import hive
   
   if __name__ == '__main__':
       # Replace <endpoint>, <username>, and <token> based on your business requirements. 
       cursor = hive.connect('<endpoint>', port=443, scheme='https', username='<username>', password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()
   

Use Java to connect to the Spark Thrift Server

1. Add the following Maven dependencies to the pom.xml file.

   <dependencies>
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-common</artifactId>
               <version>3.0.0</version>
           </dependency>
           <dependency>
               <groupId>org.apache.hive</groupId>
               <artifactId>hive-jdbc</artifactId>
               <version>2.1.0</version>
           </dependency>
       </dependencies>
   

   Note

   The version of Hive that is built in Serverless Spark is 2.x. Therefore, only hive-jdbc 2.x is supported.

2. Write Java code to connect to Spark Thrift Server.

   The following sample Java code provides an example on how to connect to the Spark Thrift Server and query databases.

   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.ResultSetMetaData;
   
   public class Main {
       public static void main(String[] args) throws Exception {
           String url = "jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token>";
           Class.forName("org.apache.hive.jdbc.HiveDriver");
           Connection conn = DriverManager.getConnection(url);
           HiveStatement stmt = (HiveStatement) conn.createStatement();
   
           String sql = "show databases";
           System.out.println("Running " + sql);
           ResultSet res = stmt.executeQuery(sql);
   
           ResultSetMetaData md = res.getMetaData();
           String[] columns = new String[md.getColumnCount()];
           for (int i = 0; i < columns.length; i++) {
               columns[i] = md.getColumnName(i + 1);
           }
           while (res.next()) {
               System.out.print("Row " + res.getRow() + "=[");
               for (int i = 0; i < columns.length; i++) {
                   if (i != 0) {
                       System.out.print(", ");
                   }
                   System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
               }
               System.out.println(")]");
           }
           
           conn.close();
       }
   }

Use the Spark Beeline client to connect to the Spark Thrift Server

1. Optional. If you use an EMR on ECS cluster, go to the bin directory of Spark.

   cd /opt/apps/SPARK3/spark-3.4.2-hadoop3.2-1.0.3/bin/

2. Run the following command to connect to the Spark Thrift Server:

   beeline -u "jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token>"

   If the following error occurs when you run the command, the Hive Beeline client may be incompatible with the Serverless Spark Thrift Server. Make sure that you use the Spark Beeline client to connect to the Spark Thrift Server.

   24/08/22 15:09:11 [main]: ERROR jdbc.HiveConnection: Error opening session
   org.apache.thrift.transport.TTransportException: HTTP Response code: 404

Configure Apache Superset to connect to the Spark Thrift Server

Apache Superset is a modern data exploration and visualization platform that supports various types of charts, from simple line charts to highly detailed geospatial charts. For more information about Superset, see Superset.

1. Install the Thrift dependency.

   Make sure that you installed Thrift of a later version. We recommend that you install Thrift of a version that is later than 16.0.0. If Thrift is not installed, you can run the following command to install Thrift:

   pip install thrift==20.0.0

2. Start Superset.

   For more information, see Superset.

3. In the upper-right corner of the page that appears, click DATABASE.

4. In the Connect a database dialog box, select Apache Spark SQL from the SUPPORTED DATABASES drop-down list.

   

5. Enter the connection string and configure the data source parameters.

   hive+https://<username>:<token>@<endpoint>:443/<db_name>

6. Click FINISH and confirm that the database is connected.

Configure Hue to connect to the Spark Thrift Server

Hue provides the UI that allows you to interact with the Hadoop ecosystem. For more information about Hue, see Hue.

1. Install the Thrift dependency.

   Make sure that you installed Thrift of a later version. We recommend that you install Thrift of a version that is later than 16.0.0. If Thrift is not installed, you can run the following command to install Thrift:

   pip install thrift==20.0.0

2. Add the Spark SQL connection string to the configuration file of Hue.

   Find the configuration file of Hue and add the following code to the file. In most cases, the configuration file of Hue is stored in the /etc/hue/hue.conf directory.

      [[[sparksql]]]
        name = Spark Sql
        interface=sqlalchemy
        options='{"url": "hive+https://<username>:<token>@<endpoint>:443/"}'

3. Restart Hue.

   After you modify the configurations, you must run the following command to restart the Hue service for the modification to take effect:

   sudo service hue restart

4. Verify the connection.

   After Hue is restarted, access the web UI of Hue and go to the Spark Sql page. If the configurations are correct, you can connect to the Spark Thrift Server and perform SQL queries.