Object Storage Service:Use OSS data to build a map dataset for random reading

Prerequisites

OSS Connector for AI/ML is installed and configured. For more information, see Install OSS Connector for AI/ML and Configure OSS Connector for AI/ML.

Build a dataset

Methods

You can use one of the following methods to build a dataset by using OssMapDataset:

• OSS_URI prefix: suitable for scenarios in which the storage paths of OSS data have uniform rules.

• OSS_URI list: suitable for scenarios in which the storage paths of OSS data are clear but scattered.

• manifest file: suitable for scenarios in which the dataset that you want to create contains a large number of files, such as tens of millions, the dataset is frequently loaded, and data indexing is enabled for the bucket. This method reduces the fees generated when you call API operations to list OSS objects.

Build a dataset by using the OSS_URI prefix

The following sample code provides an example on how to use the from_prefix method of OssMapDataset to build a dataset by specifying the OSS_URI prefix in OSS:

from osstorchconnector import OssMapDataset

ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
CONFIG_PATH = "/etc/oss-connector/config.json"
CRED_PATH = "/root/.alibabacloud/credentials"
OSS_URI = "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/"

# Use the from_prefix method of OssMapDataset to build a dataset.
map_dataset = OssMapDataset.from_prefix(oss_uri=OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)

# Access objects in the created dataset at random.
item = map_dataset[0]
print(item.key)
content = item.read()
print(item.size)
print(len(content))

# Traverse the objects in the dataset.
for item in map_dataset:
    print(item.key)
    print(item.size)
    content = item.read()
    print(len(content))

Build a dataset by using the OSS_URI list

The following sample code provides an example on how to use the from_objects method of OssMapDataset to build a dataset by specifying the OSS_URI list in OSS:

from osstorchconnector import OssMapDataset

ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
CONFIG_PATH = "/etc/oss-connector/config.json"
CRED_PATH = "/root/.alibabacloud/credentials"

# uris is a string iterator that contains multiple OSS_URIs. 
uris = [
    "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00001.png",
    "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00002.png",
    "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00003.png"
]

# Use the from_objects method of OssMapDataset to build a dataset.
map_dataset = OssMapDataset.from_objects(object_uris=uris, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)

# Access objects in the created dataset at random.
item = map_dataset[1]
print(item.key)
print(item.size)
content = item.read()
print(len(content))

# Traverse the objects in the dataset.
for item in map_dataset:
    print(item.key)
    print(item.size)
    content = item.read()
    print(len(content))

Build a dataset by using a manifest file

You need to create a manifest file and use the manifest file to build a dataset.

1. Create a manifest file:

   Run the touch manifest_file command to create a manifest file and specify the manifest file by referring to the following examples:

   Example of a manifest file that has the names of OSS objects:

   Img/BadImag/Bmp/Sample001/img001-00001.png
   Img/BadImag/Bmp/Sample001/img001-00002.png
   Img/BadImag/Bmp/Sample001/img001-00003.png

   Example of a manifest file that has the names and labels of OSS objects:

   Img/BadImag/Bmp/Sample001/img001-00001.png label1
   Img/BadImag/Bmp/Sample001/img001-00002.png label2
   Img/BadImag/Bmp/Sample001/img001-00003.png label3

2. Build a dataset by using the manifest file.

   The following sample code provides an example on how to use the from_manifest_file method of OssMapDataset to build a dataset by specifying the manifest file:

   import io
   from typing import Iterable,Tuple,Union
   from osstorchconnector import OssMapDataset
   from osstorchconnector import imagenet_manifest_parser
   
   ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
   CONFIG_PATH = "/etc/oss-connector/config.json"
   CRED_PATH = "/root/.alibabacloud/credentials"
   OSS_BASE_URI = "oss://ai-testset/EnglistImg/"
   MANIFEST_FILE_URI = "oss://manifest_fileai-testset/EnglistImg/manifest_file"
   
   # Use the from_manifest_file method of OssMapDataset to build a dataset from a local file.
   # The manifest_file_path parameter specifies the local path of the manifest file.
   # The manifest_parser parameter specifies the method used for parsing the manifest file. In the example, the built-in parsing method imagenet_manifest_parser is used.
   # The oss_base_uri parameter specifies BASE_OSS_URI, which is used to combine the URI parsed from the manifest file to form FULL_OSS_URI.
   MANIFEST_FILE_LOCAL = "/path/to/manifest_file.txt"         
   iterable_dataset = OssMapDataset.from_manifest_file(manifest_file_path=MANIFEST_FILE_LOCAL, manifest_parser=imagenet_manifest_parser, oss_base_uri=OSS_BASE_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
   for item in iterable_dataset:
       print(item.key)
       print(item.size)
       print(item.label)
       content = item.read()
       print(len(content))
   
   # Use the from_manifest_file method of OssMapDataset to build a dataset by specifying the manifest file in the OSS bucket.
   iterable_dataset = OssMapDataset.from_manifest_file(manifest_file_path=MANIFEST_FILE_URI, manifest_parser=imagenet_manifest_parser, oss_base_uri=OSS_BASE_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)

Data type in OSS Connector for AI/ML

The data type of objects in a dataset supports I/O methods. For more information, see Data type in OSS Connector for AI/ML.

Parameter description

If you use OssMapDataset or OssIterableDataset to build a dataset, you must configure the parameters. The following table describes the parameters.

Built-in methods

# The default transform function.
def identity(obj: DataObject) -> DataObject:
    if obj is not None:
        return obj.copy()
    else:
        return None

import sys
import io
import torchvision.transforms as transforms
from PIL import Image
from osstorchconnector import OssMapDataset

ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
CONFIG_PATH = "/etc/oss-connector/config.test.json"
CRED_PATH = "/root/.alibabacloud/credentials"
OSS_URI = "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/"

# Specify the transform operations on image data.
trans = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

# Use a transform method to process the input object.
def transform(object):
    try:
        img = Image.open(io.BytesIO(object.read())).convert('RGB')
        val = trans(img)
    except Exception as e:
        raise e
    return val, object.label

# Specify the transform=transform parameter when you build the dataset.
iterable_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH)
 

from osstorchconnector import imagenet_manifest_parser

def imagenet_manifest_parser(reader: io.IOBase) -> Iterable[Tuple[str, str]]:
    lines = reader.read().decode("utf-8").strip().split("\n")
    for i, line in enumerate(lines):
        try:
            items = line.strip().split('\t')
            if len(items) >= 2:
                key = items[0]
                label = items[1]
                yield (key, label)
            elif len(items) == 1:
                key = items[0]
                yield (key, '')
            else:
                raise ValueError("format error")
        except ValueError as e:
            logging.error(f"Error: {e} for line {i}: {line}")

Create a data loader through dataset by using PyTorch

The following sample code provides an example on how to create a data loader by using PyTorch based on a dataset built by using OssMapDataset as a data source:

import torch
from osstorchconnector import OssMapDataset

ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
CONFIG_PATH = "/etc/oss-connector/config.test.json"
CRED_PATH = "/root/.alibabacloud/credentials"
OSS_URI = "oss://ai-testset/EnglistImg/Img/BadImag/Bmp/Sample001/"

def transform(object):
 data = object.read()
 return object.key, object.label

# Use the from_prefix method of OssMapDataset to build a dataset.
map_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform,cred_path=CRED_PATH, config_path=CONFIG_PATH)

# Create a data loader by using PyTorch based on map_dataset.
loader = torch.utils.data.DataLoader(map_dataset, batch_size=256, num_workers=32, prefetch_factor=2, shuffle=True)
# Use data in the training loop.
# for batch in loader:
     # Perform the training operations.
   ...

References

If you perform data training jobs in a containerized environment, OSS Connector for AI/ML is also suitable for the containerized environment. For more information, see Build a Docker image that contains an OSS Connector for AI/ML environment.