Object Storage Service:Use JindoSDK with Trino to query data stored in OSS-HDFS

Procedure

1. Connect to the ECS instance. For more information, see Connect to an instance.

2. Configure JindoSDK.

   1. Download the latest version of the JindoSDK JAR package. For more information, visit GitHub.

   2. Decompress the JindoSDK JAR package.

      The following sample code provides an example on how to decompress a package named jindosdk-x.x.x-linux.tar.gz. If you use another version of JindoSDK, replace the package name with the name of the corresponding JAR package.

      tar zxvf jindosdk-x.x.x-linux.tar.gz

      Note

      x.x.x indicates the version number of the JindoSDK JAR package.

   3. Optional. If Kerberos-related and SASL-related dependencies are not included in your environment, install the following dependencies on all nodes on which JindoSDK is deployed.

      • Ubuntu or Debian

        sudo apt-get install libkrb5-dev krb5-admin-server krb5-kdc krb5-user libsasl2-dev libsasl2-modules libsasl2-modules-gssapi-mit

      • Red Hat Enterprise Linux or CentOS

        sudo yum install krb5-server krb5-workstation cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain

      • macOS

        brew install krb5

   4. Install the downloaded JindoSDK JAR package to the path specified by classpath.

      cp jindosdk-x.x.x-linux/lib/*.jar  $Trino_HOME/plugin/hive-hadoop2/

3. Configure the implementation class of OSS-HDFS and specify the AccessKey pair that you want to use to access the bucket.

   1. Configure the implementation class of OSS-HDFS in the Hadoop configuration file named core-site.xml on all Trino nodes.

      <configuration>
          <property>
              <name>fs.AbstractFileSystem.oss.impl</name>
              <value>com.aliyun.jindodata.oss.JindoOSS</value>
          </property>
      
          <property>
              <name>fs.oss.impl</name>
              <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
          </property>
      </configuration>

   2. In the core-site.xml file on all Trino nodes, specify the AccessKey ID and AccessKey secret that you want to use to access the bucket for which OSS-HDFS is enabled.

      <configuration>
          <property>
              <name>fs.oss.accessKeyId</name>
              <value>LTAI******** </value>
          </property>
      
          <property>
              <name>fs.oss.accessKeySecret</name>
              <value>KZo1********</value>
          </property>
      </configuration>

4. Configure the endpoint of OSS-HDFS.

   You must specify the endpoint of OSS-HDFS if you want to use OSS-HDFS to access buckets in Object Storage Service (OSS). We recommend that you configure the path that is used to access OSS-HDFS in the oss://<Bucket>.<Endpoint>/<Object> format. Example: oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/exampleobject.txt. After you configure the access path, JindoSDK calls the corresponding OSS-HDFS operation based on the specified endpoint in the access path.

   You can also configure the endpoint of OSS-HDFS by using other methods. The endpoints that are configured by using different methods have different priorities. For more information, see Appendix 1: Other methods used to configure the endpoint of OSS-HDFS.

   Important

   After you complete the preceding configurations, you must restart Trino for the configurations to take effect.

5. Query data stored in OSS-HDFS

   In the following example, HiveCatalog is used. You can use Trino to create a schema for OSS and execute SQL statements to query the data that is stored in OSS-HDFS. You must install and deploy JindoSDK in the Hive service because Trino uses Hive Metastore as a dependency. For more information, see Use JindoSDK with Hive to process data stored in OSS-HDFS.

   1. Log on to the Trino console.

      trino --server <Trino_server_address>:<Trino_server_port> --catalog hive

   2. Create a schema for OSS.

      create schema testDB with (location='oss://<Bucket>.<Endpoint>/<schema_dir>');

   3. Use the schema.

      use testDB;

   4. Create a table.

      create table tbl (key int, val int);

   5. Insert data into the table.

      insert into tbl values (1,666);

   6. Query data in the table.

      select * from tbl;