MaxCompute:Overview of the data transmission service

Operation types

• Data transfer by running commands

  Currently, you can run commands for data transmission only on the MaxCompute client.

• Batch operations on data

  You can use a regular tunnel to perform batch operations on offline data, such as uploading data to a table and downloading data from a table, and downloading the query results of instance data.

• Data writing in streaming mode

  You can use a stream tunnel to write data in streaming mode to tables in micro-batches.

Limits on the data transmission service

• Limits on the regular tunnel

  • Data uploads

    Item	Limit
    Lifecycle of a UploadSession	24 hours
    Number of blocks written by a UploadSession	20,000
    Speed of writing a block	10 MB/s
    Amount of data in a block	100 GB
    Number of UploadSessions created for a table	500 UploadSessions per 5 minutes
    Number of blocks written to a table	500 blocks per 5 minutes
    Number of concurrent UploadSessions submitted for a table	32
    Number of blocks that are concurrently written	Subject to the number of concurrent slots. A slot is occupied when a block is written.
    Concurrent writes	MaxCompute ensures that concurrent writes are performed based on the principle of atomicity, consistency, isolation, and durability (ACID). For more information about the semantics of ACID, see ACID semantics.

  • Data downloads

    Item	Limit
    Lifecycle of a DownloadSession	24 hours
    Lifecycle of a InstanceDownloadSession	24 hours, subject to the lifecycle of an instance.
    Number of InstanceDownloadSessions created for a project	200 InstanceDownloadSessions per 5 minutes
    Number of DownloadSessions created for a table	200 DownloadSessions per 5 minutes
    Speed of a single download request	10 MB/s
    Number of DownloadSessions that can be concurrently created	Subject to the number of concurrent slots. A slot is occupied when a DownloadSession is created.
    Number of InstanceDownloadSession that can be concurrently created	Subject to the number of concurrent slots. A slot is occupied when an InstanceDownloadSession is created.
    Number of concurrent download requests	Subject to the number of concurrent slots. A slot is occupied by a data download request.

  Note

  The Delta Table Upsert feature is supported for batch data uploads and downloads. This feature has the following limits:

  • Lifecycle of a UpsertSession: 24 hours

  • Maximum data write speed of an UpsertSession: number of tables or the buckets of partitions × 10 MB/s

  • Maximum number of slots that can be occupied: the number of tables or the buckets of partitions

  • Commitment frequency of a UpsertSession: each partition in a Delta Table allows only one commit per minute. If the duration between two commitments is less than 1 minute, the system reports an error with the message ErrorCode=FlowExceeded, ErrorMessage=CommitUpsert QPS Quota exceeded.

• Limits on the stream tunnel

  Item	Limit
  Speed of writing data to a slot	1 MB/s
  Number of write requests in a slot	10 requests per second
  Number of partitions concurrently written to a table	64
  Maximum number of available slots in a partition	32
  Number of slots occupied by StreamUploadSessions	Subject to the number of concurrent slots. You can specify the number of slots when you create a StreamUploadSession.

• Limits on data upload

  • The size of each field cannot exceed the limit of the field. For more information, see Data type editions.

    Note

    The size of a field of the STRING type cannot exceed 8 MB.

  • During the upload, multiple data entries are packaged into a package file.

• Network limits on exclusive resource groups of the data transmission service

  • Only access over virtual private clouds (VPCs) is supported. Access over the Internet is not supported.

  • Data transmission is supported in the same region. Cross-region data transmission is not supported.

Shared resource groups of the data transmission service

Valid status codes of the data transmission service

• Retry policy for failed requests

  • If a request fails, the client needs to wait for a period of time before reinitiating the request.

  • The waiting time for consecutive failed requests must exponentially increase. The shortest waiting time is 1s. Example: 1s, 2s, 4s, 8s, 16s, 32s, and 32s.

• Repeated requests

  • Requests with the same URL.

  • Requests that are consecutively initiated by clients of the same IP address.

• Valid requests

  Requests whose status codes are valid and meet the requirements for the retry policy.

• Invalid requests

  Requests whose status codes are valid but do not meet the requirements for the retry policy.

  Note

  Invalid requests are not guaranteed by the service level agreement (SLA).

• Attack requests

  • Requests whose status codes are 429 and 503 and do not comply with the retry policy. Status codes 429 and 503 are returned for process control.

  • For attack requests, MaxCompute isolates the IP address, UID, and project of the client that initiates attacks. Isolated objects cannot access MaxCompute as expected.

  Note

  Attack requests are not guaranteed by the SLA.

FAQ