MaxCompute:MaxCompute client (odpscmd)

Limits

MaxCompute client V0.28.0 and later support Java Development Kit (JDK) 1.8 and JDK 1.9. The MaxCompute client of a version that is earlier than V0.28.0 supports only JDK 1.8. You can view the client version in the CLI after you start the MaxCompute client. For more information about how to start the MaxCompute client, see Run the MaxCompute client.

Billing rules

You are not charged for accessing a MaxCompute project on the MaxCompute client. However, you may be charged for the operations that you perform on the MaxCompute client. For example, if you execute an SQL statement to query data or write data on the MaxCompute client, the SQL statement consumes computing resources and the data that is written occupies the storage capacity. As a result, you are charged for data computing and storage. For more information about the billing rules of MaxCompute, see Overview.

Precautions

• The output format of the MaxCompute client may not be forward compatible. The command syntax and execution rules of the client vary based on the client version. We recommend that you do not rely on the output format of the client to parse data. For more information about client versions, see aliyun-odps-console.

• The first time you run the Tunnel Download command on the MaxCompute client, the MaxCompute client creates a folder named session in the client installation directory plugins/dship of your on-premises machine. The folder is used to store logs. If multiple users run the Tunnel Download command on the same machine, you can use one of the following methods to ensure data security:

  • Use the folder permission management feature provided by the machine to manage the permissions on the session folder.

  • Add the -sd <Name of the new session folder> or -session-dir <Name of the new session folder> parameter to the Tunnel Download command to download data to a different session folder. For more information about the Tunnel Download command, see Download.

• Two consecutive minus signs (-) are used to comment out a command line on the MaxCompute client.

• By default, the MaxCompute client uses UTF-8 encoding. In an on-premises environment that does not use UTF-8 encoding, garbled characters may exist in the following scenarios: 1. When you query data in a MaxCompute table by using the MaxCompute client, the system returns values that contain Chinese characters. 2. You run Tunnel commands on the MaxCompute client to upload local data files to MaxCompute.

Install and configure the MaxCompute client

To install and configure the MaxCompute client, perform the following steps:

1. Download the MaxCompute client installation package from GitHub.

   Note

   • You can click the preceding link to download the latest version of the MaxCompute client installation package odpscmd_public.zip on the page that appears.

   • If you cannot download the package after you click the preceding link, click odpscmd_public_0.48.0 to download the package. For more information about how to resolve the download failure, we recommend that you search for related solutions by using search engines.

2. Decompress the downloaded package to obtain the bin, conf, lib, and plugins folders.

3. Open the conf folder and configure the odps_config.ini file.

   In the odps_config.ini file, lines that start with a number sign (#) are comments. The following table describes the parameters in the odps_config.ini file.

   Parameter	Required	Description	Example
   project_name	Yes	The name of the MaxCompute project that you want to access. If you create a workspace in standard mode, take note of the differences in the project names between the production environment and development environment when you configure the Default Project parameter. The name of the project for the development environment ends with _dev. For more information, see Differences between workspaces in basic mode and workspaces in standard mode. To obtain the name of the MaxCompute project, perform the following steps: Log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, choose Workspace > Projects. On the Projects page, view the name of the MaxCompute project.	doc_test_dev
   access_id	Yes	The AccessKey ID of your Alibaba Cloud account or RAM user. You can click the profile picture in the upper-right corner of the MaxCompute console and select AccessKey Management to obtain the AccessKey ID.	None
   access_key	Yes	The AccessKey secret that corresponds to the AccessKey ID. You can click the profile picture in the upper-right corner of the MaxCompute console and select AccessKey Management to obtain the AccessKey secret.	None
   end_point	Yes	Obtain the endpoint of the MaxCompute project. You must configure this parameter based on the region and network connection method that you selected when you create the MaxCompute project. For more information about the endpoints that correspond to different regions and network connection types, see Endpoints. Important The value of this parameter is the endpoint of MaxCompute, which is used to access MaxCompute rather than MaxCompute Tunnel. If you specify an invalid endpoint, an error occurs when you access the MaxCompute project.	http://service.cn-hangzhou.maxcompute.aliyun.com/api
   log_view_host	No	The Uniform Resource Locator (URL) of LogView. You can view the detailed runtime information of a job by using this URL. This information helps you troubleshoot job errors. Set the value to http://logview.odps.aliyun.com. Note We recommend that you configure this parameter. If you do not configure this parameter, you cannot identify the cause of job errors.	http://logview.odps.aliyun.com
   https_check	No	Specifies whether to enable HTTPS access. If HTTPS access is enabled, requests to access MaxCompute projects are encrypted. Valid values: True: HTTPS access is enabled. False: HTTP access is enabled. Default value: False.	True
   data_size_confirm	No	The maximum size of input data. Unit: GB. You can set this parameter to any value. We recommend that you set this parameter to 100.	100
   update_url	No	A reserved parameter.	None
   use_instance_tunnel	No	Specifies whether to use InstanceTunnel to download the execution results of SQL statements. Valid values: True: InstanceTunnel is used to download the execution results of SQL statements. False: InstanceTunnel is not used to download the execution results of SQL statements. Default value: False.	True
   instance_tunnel_max_record	No	The maximum number of SQL execution results that can be returned by the client. If the use_instance_tunnel parameter is set to True, you must configure this parameter. Maximum value: 10000.	10000
   tunnel_endpoint	No	The public endpoint of MaxCompute Tunnel. If you do not configure this parameter, traffic is automatically routed to the Tunnel endpoint that corresponds to the network in which MaxCompute resides. If you configure this parameter, traffic is routed to the specified endpoint and automatic routing is not performed. For more information about the Tunnel endpoints that correspond to each region and network, see Endpoints.	http://dt.cn-hangzhou.maxcompute.aliyun.com
   set.<key>	No	The properties of the MaxCompute project. For more information about the properties of MaxCompute projects, see Properties.	set.odps.sql.decimal.odps2=true

   Note

   Make sure that the settings of the preceding parameters are valid. Invalid settings will result in a project connection failure.

Run the MaxCompute client

Perform operations on the MaxCompute client

...\odpscmd\bin>odpscmd -h

odps@project_name>whoami;

odps@project_name>quit;
-- The preceding command is equivalent to the following command: 
odps@project_name>q;

What to do next

After you log on to the MaxCompute client, you can run SQL commands in the MaxCompute project. For more information, see Use the MaxCompute client.

FAQ

After the odps_config.ini file is configured, the following common errors may be reported when you start the MaxCompute client:

Specify startup parameters

Usage: odpscmd [OPTION]...
where options include:
    --help                                  (-h)for help
    --config=<config_file>                  specify another config file
    --project=<prj_name>                    use project
    --endpoint=<http://host:port>           set endpoint
    -k <n>                                  will skip begining queries and start from specified position
    -r <n>                                  set retry times
    -f <"file_path;">                       execute command in file
    -e <"command;[command;]...">            execute command, include sql command

The dynamic return value of an odpscmd -e <SQL statement> command may be called by a shell script that is run in a shell window or the Command Prompt in Windows. A shell variable obtains the return value and uses the return value in subsequent jobs. In this scenario, only field values are required. Other information, such as the runtime information and headers, cannot be returned. To facilitate shell calls, you must set the use_instance_tunnel parameter in the odps_config.ini file to false to disable InstanceTunnel. You can run the set odps.sql.select.output.format={"needHeader":false,"fieldDelim":" "}; command to disable header display.

-- Run the following command in the Command Prompt in Windows: 
...\odpscmd\bin>odpscmd -e "set odps.sql.select.output.format={""needHeader"":false,""fieldDelim"":"" ""};select * from noheader;" >D:\test.txt
-- The returned results are stored in the test.txt file in drive D. 

-- Run the following command in a shell window: 
/Users/.../odpscmd/bin/odpscmd -e "set odps.sql.select.output.format={\"needHeader\":false,\"fieldDelim\":\"\"};select * from noheader;" >/Users/A/temp/test.txt 
-- The returned results are stored in the test.txt file in the /Users/A/temp directory. 
-- The following results are returned: 
1
2
3

Version updates

The following table describes the latest version updates of the MaxCompute client. For more information, click the URL of a specific version.