FAQ about ossimport and troubleshooting - Data Online Migration

This topic provides answers to some frequently asked questions (FAQ) about ossimport.

Overview

All ossimport commands mentioned in this topic are shortened. The complete form of the commands must be used in actual scenarios.

Add console.bat to Windows commands. For example, change submit to console.bat submit.
Add bash console.sh to Linux commands. For example, change submit to sudo bash console.sh submit.

How do I check whether data is migrated by using ossimport?

After full data migration is complete, you can run one of the following command to view the job status:

In Windows, run the console.bat stat command in Command Prompt.
In Linux, run the sudo bash console.sh stat command in the terminal.

If the migration job is in the Succeed state, the migration is successful.

If you set the isIncremental parameter to true for the job, ossimport periodically scans the source directory at the specified intervals, checks for new or modified files, and synchronizes incremental data to Object Storage Service (OSS). You can view the incremental objects in the corresponding OSS bucket.

Important

ossimport does not verify data after migration and therefore does not ensure data consistency or integrity. After a migration job is complete, make sure that you verify data consistency between the migration source and destination.

If you delete the source data without verifying data consistency between the source and destination, you are responsible for any losses and consequences that arise.

Common migration failures

If a migration job fails, we recommend that you view the logs about the migration failure to identify the cause. After you resolve the migration issues, you can run the retry command to migrate the files again. The logs about migration failures are stored in the following path: master/jobs/${JobName}/failed_tasks/${TaskName}/audit.log.

What do I do if the job state is displayed as failed when I run the stat command for a migration job?

The following command is used to check the status of a migration job:

sudo bash console.sh stat

If failed is returned for the JobState parameter, the migration job fails.

Solution: After the migration job is complete, run the retry command to retry the job.

What do I do if some files fail to be migrated and the retries also fail?

Solution:

View the files that fail to be migrated in the master/jobs/${JobName}/failed_tasks/${TaskName}/error.list path to obtain the relative paths to the files.
Check whether you are authorized to access the files, whether the files are deleted, whether the files are symbolic links, and whether the file names are garbled.
After you resolve the preceding issues, run the retry command to migrate the files again.

What do I do if the The bucket you are attempting to access must be addressed using the specified endpoint. error message is reported in the migration failure log?

Exception:com.aliyun.oss.OSSException: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
<Error>
  <Code>AccessDenied</Code>
  <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
  <RequestId>56EA98DE815804**21B23EE6</RequestId>
  <HostId>my-oss-bucket.oss-cn-qingdao.aliyuncs.com</HostId>
  <Bucket>my-oss-bucket</Bucket>
  <Endpoint>oss-cn-hangzhou.aliyuncs.com</Endpoint>
</Error>

Cause: The value of the srcDomain or destDomain parameter is invalid.
Solution: Specify a valid endpoint. For more information, see Regions and endpoints.

What do I do if the The request signature we calculated does not match the signature you provided. error message is reported in the migration failure log?

Exception:com.aliyun.oss.OSSException: The request signature we calculated does not match the signature you provided. Check your key and signing method.
[ErrorCode]: SignatureDoesNotMatch
[RequestId]: xxxxxxx
[HostId]: xxx.oss-cn-shanghai.aliyuncs.com

Cause: The value of the destAccessKey or destSecretKey parameter is invalid.
Solution: Specify a valid AccessKey pair.

What do I do if the The bucket name "xxx/xx" is invalid. error message is reported in the migration failure log?

java.lang.IllegalArgumentException: The bucket name "xxx/xx" is invalid. A bucket name must: 1) be comprised of lower-case characters, numbers or dash(-); 2) start with lower case or numbers; 3) be between 3-63 characters long.

Cause: The value of the destBucket parameter is invalid.
Solution: Enter a valid bucket name. For more information about bucket naming, see Bucket naming conventions.

What do I do if the Connect to xxx.oss-cn-beijing-internal.aliyuncs.com:80 timed out. error message is reported in the migration failure log?

Unable to execute HTTP request: Connect to xxx.oss-cn-beijing-internal.aliyuncs.com:80 timed out
[ErrorCode]: ConnectionTimeout
[RequestId]: Unknown

Cause: The connection timeout error is returned because the configuration file uses the internal endpoint of OSS, but the device that is used to migrate data is not an Elastic Compute Service (ECS) instance or is not an ECS instance that resides in the same region as the OSS bucket. The internal endpoint of an OSS bucket allows access only from ECS instances that reside in the same region as the OSS bucket.
Solution:
- Set the domain name to a public endpoint in the configuration file. Delete the migration job, recreate the job, and then resubmit the job.
- Use an ECS instance that resides in the same region as the bucket to perform the migration job.

What do I do if the The specified bucket is not valid. error message is reported in the migration failure log?

com.aliyun.oss.OSSException: The specified bucket is not valid.
[ErrorCode]: InvalidBucketName
[RequestId]: 57906B4DD0EBAB0FF553D661
[HostId]: you-bucket.you-bucketoss-cn-hangzhou-internal.aliyuncs.com

Cause: The value of the destDomian parameter is invalid.
Solution: Set the destDomain parameter to an endpoint that belongs to the region in which the bucket resides instead of a second-level domain name that contains the bucket name. For example, if the bucket resides in the China (Beijing) region, enter oss-cn-beijing.aliyuncs.com. For more information, see the Configuration file examples section of the "Overview" topic.

What do I do if the Unable to execute HTTP request: The Difference between ... is too large. error message is reported in the migration failure log?

Unable to execute HTTP request: The Difference between the request time and the current time is too large.
[ErrorCode]: RequestTimeTooSkewed
[RequestId]: xxxxxxx

Cause:
- In most cases, the difference between the system time on the on-premises machine or device and the time on the OSS server exceeds 15 minutes.
- An excessively large number of requests are sent at the same time. This results in high CPU utilization and slow concurrent uploads.
Solution:
- Synchronize the system time on the on-premises machine or device with the time on the OSS server.
- If an excessively large number of requests are sent at the same time, set the workerTaskThreadNum parameter in the sys.properties file to a smaller value.

What do I do if the No route to host. error message is reported in the migration failure log?

Cause: The network connection fails due to local firewalls or iptables.
Solution: Run the ping command to check the network connection between the migration source and destination.
- If the network connection is normal, check whether restrictions are configured on the computer firewalls and local firewalls. You can also disable the firewalls to test network connectivity.
- If the network connection is abnormal, troubleshoot the issue, resolve the issue, and then perform the migration again.

What do I do if the Unknown http list file format. error message is reported in the migration failure log when I migrate files over HTTP?

Cause: The format or the content of the specified HTTP list file is invalid.
Solution:
- If the files are copied from a different operating system, convert the files to a valid format for migration. For example, you can run the mac2unix or doc2unix command in Linux and use tools such as notepad in Windows to convert the file format.
- If the format of the HTTP list file is invalid, use a valid file format. For more information about the supported formats of the HTTP list file, see the "Configuration files" section of the Overview (discontinued) topic.

What do I do if the The object key "/xxxxx.jpg" is invalid error message is reported in the migration failure log?

Exception:java.lang.IllegalArgumentException: The object key "/xxxxx.jpg" is invalid. An object name should be between 1 - 1023 bytes long when encoded as UTF-8 and cannot contain LF or CR os unsupported chars in XML1.0, and cannot begin with "/" or "\".

Cause: The value of the srcPrefix or destPrefix parameter is invalid.
Solution:
- Check whether the srcPrefix parameter specifies a directory. If yes, the value of the parameter must end with a forward slash (/).
- Check whether the value of the destPrefix parameter starts with a forward slash (/) or a backslash (\). If yes, delete the forward slash (/) or backslash (\).

Common issues during migration

If an error occurs during data migration, you can view the run log of the migration job.

If ossimport is deployed in standalone mode, the run log file is stored in the following path: logs/ossimport2.log.
If ossimport is deployed in distributed mode, the run log file is stored in the following path: logs/import.log.

What do I do if the UnsupportedClassVersionError. error message appears when I run a command?

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/aliyun/ossimport2/OSSImport2 : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at com.simontuffs.onejar.JarClassLoader.defineClass(JarClassLoader.java:693)
        at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:599)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at com.simontuffs.onejar.Boot.run(Boot.java:300)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)

Cause: The Java version is outdated.
Solution: Upgrade the Java version to 1.7 or 1.8.

What do I do if the InvocationTargetException. error message appears when I run the submit command to submit a job?

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: java.lang.NullPointerException
        at com.aliyun.ossimport2.config.JobConfig.load(JobConfig.java:44)
        at com.aliyun.ossimport2.OSSImport2.doSubmitJob(OSSImport2.java:289)
        at com.aliyun.ossimport2.OSSImport2.main(OSSImport2.java:120)
        ... 6 more

Cause: The configuration file is incomplete. For example, some configuration items are deleted or commented out from the configuration file.
Solution: Restore the configuration items that are deleted or commented out from the configuration file. If a configuration item is not required, leave the configuration item empty or delete the value on the right to the equal sign (=). For more information, see the "Configuration file examples" section of the Overview (discontinued) topic.

What do I do if the com.aliyun.oss.ClientException: Unknown. error message is reported in the run log of a migration job?

com.aliyun.oss.ClientException: Unknown
[ErrorCode]: NonRepeatableRequest
[RequestId]: Cannot retry request with a non-repeatable request entity.  The cause lists the reason the original request failed.

Cause: In most cases, the com.aliyun.oss.ClientException: Unknown. or SocketTimeoutException error message is returned when the server consumes full bandwidth.
Solution: ossimport automatically retries the migration job. If the job still fails, run the retry command to migrate the files again.

What do I do if the too many open files. error message is returned in the run log of a migration job in Linux?

Solution: Run the ulimit -n command to view the limit on handles in Linux.

If the handle limit is smaller than 10,000, run the ulimit -n 65536 command to increase the value and restart the process.
If the handle limit is greater than 10,000, run the sudo lsof -n command to identify the processes for which the handles are enabled. You can evaluate the processes and stop the unnecessary processes to release the handles. If a handle is no longer required, we recommend that you release the handle.

What do I do if a migration job exits unexpectedly after the job is started in Windows?

Cause:
- Java is not installed, or the Java version is earlier than 1.7.
- The configuration file is invalid.
Solution:
- Install Java 1.8.
- Modify the configuration file based on sample configurations. For more information, see the "Configuration file examples" section of the Overview (discontinued) topic.

After I run the submit command to submit a job, the no jobs is running or finished. error message appears when I run the stat command to view the job status. What do I do to fix the error?

sudo bash console.sh stat
[WARN]   List files dir not exist : /home/<user>/ossimport/workdir/master/jobs/
no jobs is running or finished.

Cause: The service is not started when you run the stat command. You can run the stat command to view the status of a job only after the job is submitted and the service is started.
Solution:
- Run the start command to start the service.
- If the service is started, the master node immediately scans the files to be migrated after the job is submitted. In this case, the tasks of the migration job are not created or dispatched, and you can ignore the error.
- If the error occurs after the service is started and the job is submitted for an extended period of time, check whether the process exits unexpectedly after it is started. If ossimport is deployed in standalone mode, view the log file in the logs/ossimport2.log path. If ossimport is deployed in distributed mode, view the job file in the logs/ossimport.log path. Troubleshoot and fix the error, and then start the service process.

What do I do if the scanFinished: false. error message appears when I run the stat command to check the job status?

Solution: Check whether the total number of tasks increases.

If the total number of tasks increases, new files are added to the file list of the migration job. In this case, you can ignore the error.
If the total number of tasks is fixed, the incremental data migration mode is enabled, and the value of the scanFinished parameter is not true, ossimport scans the source directory at the specified intervals to check for new or modified files.
If the incremental data migration mode is disabled and the number of tasks does not increase, check the run log of the migration job. If ossimport is deployed in standalone mode, view the log file in the logs/ossimport2.log path. If ossimport is deployed in distributed mode, view the job file in the logs/ossimport.log path. Troubleshoot and fix the error, and then start the service process.

Why is no error reported in the log file when the service process in Linux is abnormal?

Cause: If the available memory size of the system is less than 2 GB, errors may occur in the service process due to insufficient memory.
Solution: Check whether the dmesg log contains the records about process errors caused by insufficient memory.

What do I do if I want to restart the service after an error occurs in the service process?

Solution: If you do not run the clean command to delete jobs that use the same name, checkpoint files are generated for all the submitted jobs. In this case, you can run the start command to directly start the service without submitting the jobs again.

How do I upload a file whose name is garbled from Linux to OSS?

Solution:

Check the encoding format of the garbled file name.
Run the export LANG="<your file name encode>" command to parse the file name.
Run the clean command to delete the original job, and then run the submit command to submit the job again.

What do I do if the java.nio.file.AccessDeniedException. error message appears when the service is started?

Cause: You are not authorized to access the configuration file.
Solution:
- Modify the access control list (ACL) of the configuration file to allow all users to access the configuration file.
- Log on to Linux as an administrator to start the service.

Why is SUCCEED returned for the JobState parameter when 0 is returned for the Task Count parameters?

[2015-12-28 16:12:35]   [INFO]  JobName:dir_data
[2015-12-28 16:12:35]   [INFO]  Pending Task Count:0
[2015-12-28 16:12:35]   [INFO]  Dispatched Task Count:0
[2015-12-28 16:12:35]   [INFO]  Succeed Task Count:0
[2015-12-28 16:12:35]   [INFO]  Failed Task Count:0
[2015-12-28 16:12:35]   [INFO]  Is Scan Finished:true
[2015-12-28 16:12:35]   [INFO]  JobState:SUCCEED

Cause:
- Files cannot be listed because the srcPrefix parameter is invalid.
- The directory specified by the srcPrefix parameter does not contain files. The directory cannot be uploaded to OSS because it is simulated by OSS.
Solution: Specify a valid value for the srcPrefix parameter and make sure that the directory specified by the srcPrefix parameter contains available files.

What do I do if the InvocationTargetException. error message appears when I submit a job?

sudo submit job:/disk2/ossimport2/local_job.cfg
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: java.lang.NullPointerException
        at com.aliyun.ossimport2.OSSImport2.doSubmitJob(OSSImport2.java:289)
        at com.aliyun.ossimport2.OSSImport2.main(OSSImport2.java:120)
        ... 6 more

Cause: The configuration file or the path of the configuration file is invalid.
Solution:
- Specify a valid value for the workingDir parameter in the conf/sys.properties file.
- Check the path of the configuration file.

Why do the source files to be migrated not exist during synchronization?

Cause: When the master node performs a migration job, it first lists the files and then migrates the files in the list. If some files are deleted from the source after the list operation is complete, the files cannot be found. In this case, the master node skips the deleted source files and reports the deleted files as errors.

The configuration file of a migration job is valid. Why is the status of the job inconsistent with the settings in the configuration file during the migration?

Cause: After a migration job is submitted, modifications of the configuration file do not take effect. For example, after you pause a submitted job and modify the configuration file, the modifications do not take effect.
Solution: Run the clean command to delete the original job. After the configuration file is modified, submit the job again.

Why does the NullPointerException. error message appear when files are transmitted as expected?

Cause: ossimport 2.3.5 provides the file statistics feature, which loads CPT files. However, the loading format for CPT files of HTTP jobs is different from that of jobs that use other sources. In this case, a program mismatch error occurs.
Solution:
- Downgrade the ossimport version to 2.3.4.
- Ignore the error. This error is related to information display and does not affect the migration job.

Common issues about migrating data from UPYUN Storage Service (USS)

What is the number of migration jobs always 0?

Solution: View the run log of the migration job.

[2016-07-21 10:21:46] [INFO] [name=YoupaiList, totalRequest=1729925, avgLatency=38,
          recentLatency=300000]

If the value of the recentLatency parameter in the run log is smaller than or equal to 30,000, the files are normally listed. In most cases, it takes more than 30 seconds to list the files in USS. The number of returned files varies based on the number of files that are listed in 30 seconds. Wait until the list operation is complete.
If the value of the recentLatency parameter is small, the account or password is invalid. If an error is reported by USS SDK, only null is returned and no error result is returned. In this case, capture packets to obtain the error code returned by USS for troubleshooting.

How do I specify the `srcAccessKey` and `srcSecretKey` parameters when I migrate data from USS?

Solution: Enter the operator account and password of USS.

Why does the HTTP 429 status code repeatedly appear when I migrate data from USS?

Cause: Request throttling is enabled for USS SDK, and the number of access requests within a specific period of time exceeds the limit.
Solution: Contact USS to disable request throttling.

Why is the data size displayed in the OSS console smaller than the source data size after the migration job is complete?

Description: After all migration jobs are complete, the bucket size does not change in the OSS console, but the data size calculated by running the du command in Linux is significantly different from the actual data size.
Cause:
- The bucket size is updated in the OSS console with a delay of 1 to 2 hours. Check whether the bucket size changes 1 to 2 hours after the migration jobs are complete.
- The du command in Linux calculates the block size, which is larger than the actual data size. We recommend that you run the ls -lR <Absolute directory path> | grep "\-rw" | awk '{sum+=$5}END{print sum}' command to calculate the actual size of the local directory.

Why does the unknown command "java" or unknown command "nohup" error message appear when I run a command in Linux?

Cause: The package that is required to run the command is not installed.
Solution: Run the yum, apt-get, or zypper command to install the corresponding package.

Can I use the `srcPrefix` parameter in the configuration file to specify files?

No. You can use the srcPrefix parameter to specify only directories or prefixes.

Can I configure a proxy for ossimport?

No.

Why am I charged for migrating data within OSS?

If you migrate data by using an internal endpoint, you are charged based on the number of access requests. You are not charged traffic fees.

Are the files that are deleted from the local directory also deleted from OSS if the incremental data migration mode is enabled?

No. Delete operations performed on the local directory are not synchronized to OSS.

Why are new files in the local directory not synchronized to OSS when the incremental data migration mode is enabled?

In incremental data migration mode, the system determines whether a file is new based on the last modification time of the file. The following commands do not modify the last modification time of files: the mv command in Linux, and the cp, mv, and rsync commands in Windows. The parameters of the rsync command must contain -t or -a. Files modified by running the preceding commands are not scanned or migrated to OSS.

Does ossimport synchronize the permissions on the files to be migrated to OSS?

No. After the migration is complete, you can run the set-meta command in ossutil to modify the file permissions. For more information, see set-meta.