データ検証 - Object Storage Service - Alibaba Cloud ドキュメントセンター

Object Storage Service (OSS) SDK for Javaは、MD5の検証とCRC-64を使用して、オブジェクトのアップロード、ダウンロード、コピー時にデータの整合性を確保します。

使用上の注意

このトピックでは、中国 (杭州) リージョンのパブリックエンドポイントを使用します。 OSSと同じリージョンにある他のAlibaba CloudサービスからOSSにアクセスする場合は、内部エンドポイントを使用します。 OSSリージョンとエンドポイントの詳細については、「リージョン、エンドポイント、オープンポート」をご参照ください。
このトピックでは、アクセス資格情報は環境変数から取得します。アクセス資格情報の設定方法の詳細については、「アクセス資格情報の設定」をご参照ください。
このトピックでは、OSSエンドポイントを使用してOSSClientインスタンスを作成します。カスタムドメイン名またはSecurity Token Service (STS) を使用してOSSClientインスタンスを作成する場合は、「OSSClientインスタンスの作成」をご参照ください。

MD5検証

オブジェクトのアップロード要求でContent-MD5を設定すると、アップロードされたオブジェクトのMD5ハッシュが計算されます。計算されたMD5ハッシュがアップロード要求で設定されたMD5ハッシュと異なる場合、InvalidDigestが返されます。これにより、OSSはオブジェクトのアップロードのデータ整合性を確保できます。 InvalidDigestが返された場合は、オブジェクトを再度アップロードする必要があります。

MD5検証はマルチパートアップロードでもサポートされています。マルチパートアップロードリクエストでは、アップロードするオブジェクトのメタデータを指定します。 MD5検証は、マルチパートアップロード中に各パートに対して実行されます。 setMd5Digest関数は、クライアントがMD5ハッシュを計算するためのUploadPartRequestリクエストで呼び出されます。

説明

MD5検証は、PutObject、GetObject、AppendObject、PostObject、マルチパートアップロード、およびUploadPartでサポートされています。

次のサンプルコードは、PutObject操作でMD5検証を設定する方法の例を示しています。

import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.utils.BinaryUtil;
import com.aliyun.oss.model.ObjectMetadata;
import java.io.ByteArrayInputStream;

public class Demo {
    public static void main(String[] args) throws Throwable {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";
        // Specify the full path of the object. Do not include the bucket name in the full path. 
        String objectName = "exampledir/object";
        // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
        String region = "cn-hangzhou";

        // Create an OSSClient instance. 
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);        
        OSS ossClient = OSSClientBuilder.create()
        .endpoint(endpoint)
        .credentialsProvider(credentialsProvider)
        .clientConfiguration(clientBuilderConfiguration)
        .region(region)               
        .build();

        try {
            // Upload the string. 
            String content = "Hello OSS";

            ObjectMetadata meta = new ObjectMetadata();
            // Configure MD5 verification. 
            String md5 = BinaryUtil.toBase64String(BinaryUtil.calculateMd5(content.getBytes()));
            meta.setContentMD5(md5);

            ossClient.putObject(bucketName, objectName, new ByteArrayInputStream(content.getBytes()), meta);

        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}

次のサンプルコードは、マルチパートアップロード操作でMD5検証を設定する方法の例を示しています。

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import com.aliyun.oss.OSS;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.common.auth.CredentialsProviderFactory;
import com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider;
import com.aliyun.oss.common.utils.BinaryUtil;
import com.aliyun.oss.model.CompleteMultipartUploadRequest;
import com.aliyun.oss.model.CompleteMultipartUploadResult;
import com.aliyun.oss.model.InitiateMultipartUploadRequest;
import com.aliyun.oss.model.InitiateMultipartUploadResult;
import com.aliyun.oss.model.PartETag;
import com.aliyun.oss.model.UploadPartRequest;
import com.aliyun.oss.model.UploadPartResult;

public class Demo {
    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "http://oss-cn-hangzhou.aliyuncs.com";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the bucket. Example: examplebucket.   
        String bucketName = "examplebucket";
        // Specify the full path of the object. Do not include the bucket name in the full path. 
        String objectName = "exampledir/object";
        // Specify the path of the local file that you want to upload. 
        String localFile = "D:\\localpath\\examplefile.txt";
        // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
        String region = "cn-hangzhou";

        // Create an OSSClient instance. 
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);        
        OSS ossClient = OSSClientBuilder.create()
        .endpoint(endpoint)
        .credentialsProvider(credentialsProvider)
        .clientConfiguration(clientBuilderConfiguration)
        .region(region)               
        .build();
        // Create an InitiateMultipartUploadRequest object. 
        InitiateMultipartUploadRequest request = new InitiateMultipartUploadRequest(bucketName, objectName);
        // Optional. Specify the storage class.
        // ObjectMetadata metadata = new ObjectMetadata();
        // metadata.setHeader(OSSHeaders.OSS_STORAGE_CLASS, StorageClass.Standard.toString());
        // request.setObjectMetadata(metadata);
        // Initiate a multipart upload task. 
        InitiateMultipartUploadResult upresult = ossClient.initiateMultipartUpload(request);
        // Obtain an upload ID. The upload ID uniquely identifies the multipart upload task. You can use the upload ID to initiate related requests such as canceling and querying a multipart upload task. 
        String uploadId = upresult.getUploadId();
        // partETags is a set of PartETags. The PartETag of a part consists of the part number and ETag value of the part. 
        List<PartETag> partETags = new ArrayList<PartETag>();
        // Calculate the total number of parts to upload. 
        final long partSize = 1 * 1024 * 1024L;   // 1MB
        final File sampleFile = new File(localFile);
        long fileLength = sampleFile.length();
        int partCount = (int) (fileLength / partSize);
        if (fileLength % partSize != 0) {
            partCount++;
        }
        // Upload all parts. 
        for (int i = 0; i < partCount; i++) {
            long startPos = i * partSize;
            long curPartSize = (i + 1 == partCount) ? (fileLength - startPos) : partSize;
            InputStream instream = new FileInputStream(sampleFile);
            InputStream instream1 = new FileInputStream(sampleFile);
         // Skip parts that have been uploaded. 
            instream.skip(startPos);
            instream1.skip(startPos);
            String md5;
            if(i==partCount-1){
         // The size of the last part is not necessarily equal to the specified part size.
                md5 = md5(instream1,fileLength - startPos);
            }else{
                md5 = md5(instream1,partSize);
            }
          // instream1.skip(n)
            UploadPartRequest uploadPartRequest = new UploadPartRequest();
            uploadPartRequest.setBucketName(bucketName);
            uploadPartRequest.setKey(objectName);
            uploadPartRequest.setUploadId(uploadId);
            uploadPartRequest.setInputStream(instream);
            uploadPartRequest.setMd5Digest(md5);
         // Specify the part size. Each part except the last part must be equal to or greater than 100 KB. 
            uploadPartRequest.setPartSize(curPartSize);
         // Configure part numbers. Each part is configured with a part number. The value ranges from 1 to 10000. If you configure a number beyond the range, OSS returns an InvalidArgument error code. 
            uploadPartRequest.setPartNumber( i + 1);
         // Parts are not necessarily uploaded in order and can be uploaded from different OSS clients. OSS sorts the parts based on the part numbers and combines the parts into a complete object. 
            UploadPartResult uploadPartResult = ossClient.uploadPart(uploadPartRequest);
         // System.out.println("server md5" +uploadPartResult.getETag());
         // Each time a part is uploaded, OSS returns a result that contains a PartETag. The PartETag is stored in partETags. 
            partETags.add(uploadPartResult.getPartETag());
        }
        // Create a CompleteMultipartUploadRequest object. 
        // When you call the CompleteMultipartUpload operation, you must provide all valid PartETags. After OSS receives the partETags, OSS verifies all parts one by one. After part verification is successful, OSS combines these parts into a complete object. 
        CompleteMultipartUploadRequest completeMultipartUploadRequest =
                new CompleteMultipartUploadRequest(bucketName, objectName, uploadId, partETags);
        // Optional. Set the access control list (ACL) of the object. 
        // completeMultipartUploadRequest.setObjectACL(CannedAccessControlList.PublicRead);
        // Complete the multipart upload task. 
        CompleteMultipartUploadResult completeMultipartUploadResult = ossClient.completeMultipartUpload(completeMultipartUploadRequest);
        // Close your OSSClient. 
        ossClient.shutdown();
    }
    public static String md5(InputStream in , long length1) throws Exception{
        byte[] bytes = new byte[(int) length1];
        long length_tmp = length1;
        int readSize = in.read(bytes, (int) 0, (int) length_tmp);
        return BinaryUtil.toBase64String(BinaryUtil.calculateMd5(bytes));
    }
}

CRC-64検証

CRC-64検証は、オブジェクトのアップロード、ダウンロード、およびコピー操作でデフォルトで有効になり、データの整合性を確認します。

説明

CRC-64は、PutObject、GetObject、AppendObject、およびUploadPart操作で使用できます。デフォルトでは、オブジェクトをアップロードするとCRC-64が有効になります。クライアントで計算されたCRC-64チェックサムが、OSSサーバーによって返されたCRC-64チェックサムと異なる場合、InconsistentExceptionエラーが返されます。
範囲のダウンロードではCRC-64検証はサポートされていません。
CRC-64はCPUリソースを消費し、アップロードとダウンロードを遅くします。

オブジェクトのダウンロードのCRC-64

次のサンプルコードは、オブジェクトをダウンロードするときにCRC-64を実行する方法の例を示しています。

import com.aliyun.oss.*;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.common.utils.IOUtils;
import com.aliyun.oss.internal.OSSHeaders;
import com.aliyun.oss.internal.OSSUtils;
import com.aliyun.oss.model.GetObjectRequest;
import com.aliyun.oss.model.OSSObject;
import java.io.BufferedReader;
import java.io.InputStreamReader;

public class Demo {
    public static void main(String[] args) throws Throwable {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";
        // Specify the full path of the object. Do not include the bucket name in the full path. 
        String objectName = "exampledir/object";
        // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
        String region = "cn-hangzhou";

        // Create an OSSClient instance. 
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);        
        OSS ossClient = OSSClientBuilder.create()
        .endpoint(endpoint)
        .credentialsProvider(credentialsProvider)
        .clientConfiguration(clientBuilderConfiguration)
        .region(region)               
        .build();

        try {
            // Download the object in streaming mode. 
            GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, objectName);
            OSSObject ossObject = ossClient.getObject(bucketName, objectName);

            // Read the object from the stream to obtain clientCRC. 
            System.out.println("Object content:");
            BufferedReader reader = new BufferedReader(new InputStreamReader(ossObject.getObjectContent()));
            while (true) {
                String line = reader.readLine();
                if (line == null) break;

                System.out.println("\n" + line);
            }
            // You must close the obtained stream after the object is read. Otherwise, connection leaks may occur. Consequently, no connections are available and an exception occurs. 
            reader.close();

            // Check whether CRC-64 is enabled on the client. By default, CRC-64 is enabled on the client. 
            Boolean isCrcCheckEnabled = ((OSSClient)ossClient).getClientConfiguration().isCrcCheckEnabled();
            // Check whether the download quest is a range download request. CRC-64 verification is not supported for range download. 
            Boolean isRangGetRequest = getObjectRequest.getHeaders().get(OSSHeaders.RANGE) != null;

            // Check whether the CRC-64 value in the returned result is the same as that of the downloaded object. 
            if (isCrcCheckEnabled && !isRangGetRequest) {
                Long clientCRC = IOUtils.getCRCValue(ossObject.getObjectContent());
                OSSUtils.checkChecksum(clientCRC, ossObject.getServerCRC(), ossObject.getRequestId());
            }
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}

追加アップロードのCRC-64

次のサンプルコードは、追加アップロードを使用してオブジェクトをアップロードするときにCRC-64を実行する方法の例を示しています。

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.*;
import java.io.ByteArrayInputStream;

public class Demo {
    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";
        // Specify the full path of the object. Example: exampleobject.txt. Do not include the bucket name in the full path. 
        String objectName = "exampleobject.txt";
        // Enter the content of the first append upload, such as Hello. 
        String firstAppendContent = "Hello";
        // Enter the content of the second append upload, such as World. 
        String secondAppendContent = "World";
        // Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
        String region = "cn-hangzhou";

        // Create an OSSClient instance. 
        ClientBuilderConfiguration clientBuilderConfiguration = new ClientBuilderConfiguration();
        clientBuilderConfiguration.setSignatureVersion(SignVersion.V4);        
        OSS ossClient = OSSClientBuilder.create()
        .endpoint(endpoint)
        .credentialsProvider(credentialsProvider)
        .clientConfiguration(clientBuilderConfiguration)
        .region(region)               
        .build();

        try {
            // Perform the first append operation. 
            AppendObjectRequest appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(firstAppendContent.getBytes()));
            appendObjectRequest.setPosition(0L);
            // Set the initial CRC-64 value. OSS SDK for Java performs CRC-64 on the uploaded content by default. 
            appendObjectRequest.setInitCRC(0L);
            AppendObjectResult appendObjectResult = ossClient.appendObject(appendObjectRequest);

            // Perform the second append upload operation. 
            appendObjectRequest = new AppendObjectRequest(bucketName, objectName, new ByteArrayInputStream(secondAppendContent.getBytes()));
            appendObjectRequest.setPosition(appendObjectResult.getNextPosition());
            // Set the CRC-64 value to that of the uploaded object. OSS SDK for Java performs CRC-64 on the uploaded content by default. 
            appendObjectRequest.setInitCRC(appendObjectResult.getClientCRC());
            ossClient.appendObject(appendObjectRequest);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}

Object Storage Service:データ検証

使用上の注意

MD5検証

CRC-64検証

関連ドキュメント