Python on MaxCompute (PyODPS) is MaxCompute SDK for Python, which helps you use PyODPS to interact with MaxCompute and process data. You can use PyODPS to develop MaxCompute jobs, analyze data, and manage MaxCompute resources. This topic describes how to use PyODPS.
Introduction to PyODPS
PyODPS supports the DataFrame framework and basic operations on MaxCompute objects. You can use PyODPS to develop MaxCompute jobs and process data in MaxCompute. PyODPS supports Python 2 and Python 3. Python 2 can be Python 2.6 or later.
For more information about PyODPs, see the following documentation:
For more information about PyODPS, see PyODPS: ODPS Python SDK and data analysis framework and PyODPS-related articles.
For more information about how to download PyODPS, visit GitHub.
For more information about how to install PyODPS, see PyODPS installation instructions.
For more information about how to develop PyODPS, see PyODPS developer guide.
If you want to help build the PyODPS ecosystem, you can perform the following operations:
Write PyODPS documentation.
Develop PyODPS code at GitHub.
Click the link to join the DingTalk group for technical communication.
Initialization
Before you can use PyODPS, you must initialize a connection to MaxCompute by using your Alibaba Cloud account. To initialize a connection, run the following command:
import os
from odps import ODPS
# Set the environment variable ALIBABA_CLOUD_ACCESS_KEY_ID to the AccessKey ID of your Alibaba Cloud account.
# Set the environment variable ALIBABA_CLOUD_ACCESS_KEY_SECRET to the AccessKey secret of your Alibaba Cloud account.
o = ODPS(
os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'),
os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'),
project='your-default-project',
endpoint='your-end-point',
)Parameters:
ALIBABA_CLOUD_ACCESS_KEY_ID: the AccessKey ID of your Alibaba Cloud account. Make sure that the account has the permissions to manage objects in a MaxCompute project. For more information about the permissions, see Operation permissions. In the upper-right corner of the MaxCompute console, click the profile picture and select AccessKey Management to obtain the AccessKey ID.
ALIBABA_CLOUD_ACCESS_KEY_SECRET: the AccessKey secret that corresponds to the AccessKey ID. In the upper-right corner of the MaxCompute console, click the profile picture and select AccessKey Management to obtain the AccessKey secret.
your-default-project: the name of your MaxCompute project. You can log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, choose Workspace > Projects to view the name of the MaxCompute project.
your-end-point: the endpoint of the region where your MaxCompute project resides. For more information, see Endpoints.
Description
The following table describes the methods that you can use to perform basic operations on MaxCompute objects.
Item | Operation | Description |
Projects | get_project(project_name) | Obtains the name of a MaxCompute project. |
exist_project(project_name) | Checks whether a MaxCompute project exists. | |
Tables | list_tables() | Lists all tables in a MaxCompute project. |
exist_table(table_name) | Checks whether a table exists. | |
get_table(table_name, project=project_name) | Obtains a specified table. You can obtain a table from another MaxCompute project. | |
create_table() | Creates a table. | |
read_table() | Reads data from a table. | |
write_table() | Writes data to a table. | |
delete_table() | Deletes an existing table. | |
Table partitions | exist_partition() | Checks whether a partition exists. |
get_partition() | Obtains information about a partition. | |
create_partition() | Creates a partition. | |
delete_partition() | Deletes an existing partition. | |
SQL | execute_sql()/run_sql() | Executes SQL statements. |
open_reader() | Reads execution results of SQL statements. | |
Instances | list_instances() | Lists all instances in a MaxCompute project. |
exist_instance() | Checks whether an instance exists. | |
get_instance() | Obtains information about an instance. | |
stop_instance() | Terminates an instance. | |
Resources | create_resource() | Creates a resource. |
open_resource() | Opens a resource. | |
get_resource() | Obtains information about a resource. | |
list_resources() | Lists all existing resources. | |
exist_resource() | Checks whether a resource exists. | |
delete_resource() | Deletes an existing resource. | |
Functions | create_function() | Creates a function. |
delete_function() | Deletes an existing function. | |
Uploads and downloads tunnels | create_upload_session() | Creates a session that is used to upload data. |
create_download_session() | Creates a session that is used to download data. |
You must specify parameters when you use create_table(), read_table(), write_table(), and delete_table() methods. For more information, see Examples of using the SDK for Python: tables.