Object Storage Service:FAQ (Python SDK V1)

Multipart upload fails when using the OSS Python SDK

Solutions:

• Confirm whether you are uploading the object directly to OSS or through a proxy, such as a Content Delivery Network (CDN). If you upload to OSS through a CDN, you must configure cross-domain HTTP headers in OSS, such as Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers, and expose the ETag. For more information, see PutBucketCors.

• If an upload fails due to a network timeout, use resumable upload instead of a standard upload. Resumable upload supports parallel uploads and custom part sizes. If an exception is caught, check and analyze the exception message from the SDK.

• Clean up the fragments from the failed upload and retry the upload.

ossutil is much faster for uploads and downloads than the Python SDK on the same machine

• Cause

  The ossutil tool is developed based on the Go SDK and offers better performance for concurrent uploads. If the Python SDK is much slower than ossutil for uploads and downloads, this is usually because crcmod is not installed correctly.

• Solution

  For more information about how to install crcmod, see Install the Python SDK.

Multipart uploads work on CentOS but return a 403 error on Ubuntu

You can deploy tcpdump on the client to capture packets. Then, check the TCP messages to determine whether incorrect header information causes a mismatch between the signature calculated by the client and the signature on the server side.

POST /ttsservice%2Fpasswd?uploadId=D468E486D1D94D90A1AB8885A4E32AE0 HTTP/1.1
Host: rokid.oss-cn-hangzhou.aliyuncs.com
Accept-Encoding: identity
Accept: text/html
Content-Length: 137
date: Sat, 29 Dec 2018 07:32:34 GMT
authorization: OSS LTAI****************:r2kp*****************
Content-Type: application/x-www-form-urlencoded
User-Agent: aliyun-sdk-python/2.6.0(Linux/4.4.0-31-generic/x86_64;3.4.3)

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"3195544E19D99658706D5****"</ETag></Part></CompleteMultipartUpload>HTTP/1.1 403 Forbidden

Server: AliyunOSS
Date: Sat, 29 Dec 2018 07:33:43 GMT
Content-Type: application/xml
Content-Length: 1122
Connection: keep-alive
x-oss-request-id: 5C2723573183****
x-oss-server-time: 0

<?xml version="1.0" encoding="UTF-8"?>
<Error>
 <Code>SignatureDoesNotMatch</Code>
 <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
 <RequestId> 5C2723573183A12D </RequestId>
 <HostId>rokid.oss-cn-hangzhou.aliyuncs.com</HostId>
 <OSSAccessKeyId>LTAI****************</OSSAccessKeyId>
 <SignatureProvided>r2kp*****************</SignatureProvided>
 <StringToSign>POST
application/x-www-form-urlencoded
Sat, 29 Dec 2018 07:32:34 GMT
/rokid/ttsservice/passwd?uploadId=D468E486D1D94D90A1AB8885A4E3****</StringToSign>
 <StringToSignBytes>50 4F 53 54 0A 0A 61 70 70 6C 69 63 61 74 69 6F 6E 2F 78 2D 77 77 77 2D 66 6F 72 6D 2D 75 72 6C 65 6E 63 6F 64 65 64 0A 53 61 74 2C 20 32 39 20 44 65 63 20 32 30 31 38 20 30 37 3A 33 32 3A 33 34 20 47 4D 54 0A 2F 72 6F 6B 69 64 2D 6F 70 73 2D 6D 6F 64 65 6C 2F 74 74 73 73 65 72 76 69 63 65 2F 70 61 73 73 77 64 3F 75 70 6C 6F 61 64 49 64 3D 44 34 36 38 45 34 38 36 44 31 44 39 34 44 39 30 41 31 41 42 38 38 38 35 41 34 45 33 32 41 45 30 </StringToSignBytes>
</Error>

If the signature received by the server does not match the signature calculated by the client, the request content may have been modified. Use HTTPS to upload the object.

import base64
import hmac
import sha
mac = hmac.new("<Secretkey>","POST\n\napplication/x-www-form-urlencoded\nSat, 29 Dec 2018 07:32:34 GMT\n/rokid/ttsservice/passwd?uploadId=D468E486D1D94D90A1AB8885A4E3****", sha)
Signature = base64.b64encode(mac.digest())
print(Signature)

If the signature from the script matches the one from the captured data, the SDK calculation is correct. If they do not match, the MD5 hash might be different due to compilation compatibility issues with the SDK on the Ubuntu platform.

Error when you import TensorFlow in a multi-threaded Python application on macOS

objc[2483]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[2483]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Solution:

For more information, see Objective-C and fork() in macOS 10.13.

How to add a retry policy

The Python SDK does not have a built-in retry mechanism. Requests might fail in poor network conditions. In this case, you can add a retry policy to your code by referring to the following example.

def test_get_object():
    MAX_RETRIES = 3
    retry_count = 0
    while True:
        try:
            retry_count += 1
            # yourObjectName specifies the full path of the OSS file, excluding the bucket name. For example, abc/efg/example.jpg.
            # yourFileName specifies the full path of the local file where you want to download the object. For example, /users/local/example.jpg.
            bucket.get_object_to_file("yourObjectName", "yourFileName")
            break
        except Exception:
            if retry_count >= MAX_RETRIES:
                raise

What to do if the C extension mode of crcmod fails to install on some operating systems

If the C extension mode of crcmod cannot be successfully installed on your operating system, you can disable the 64-bit cyclic redundancy check (CRC-64). For more information, see Disable CRC-64 data validation.

How to ensure a generated signed URL starts with https

To ensure that a generated signed URL starts with https, set the endpoint to an address that starts with https when you initialize the SDK, such as https://oss-cn-hangzhou.aliyuncs.com.

How to use a bucket instance in a multi-process environment

The bucket instance must be created after the multi-process initialization is complete.