Data Online Migration:Deploy ossimport in standalone mode (discontinued)

Quick start

1. Download and decompress ossimport-2.3.7.zip.

   The following code shows the directory structure after you decompress the package:

   ossimport
   ├── bin
   │   └── ossimport2.jar  # The JAR package that contains the Master, Worker, TaskTracker, and Console modules.
   ├── conf
   │   ├── local_job.cfg   # The job configuration file.
   │   └── sys.properties  # The configuration file that contains system parameters.
   ├── console.bat         # The Windows command-line tool used to run tasks in a distributed manner.
   ├── console.sh          # The Linux command-line tool used to run tasks in a distributed manner.
   ├── import.bat          # The script that automatically imports files based on the conf/local_job.cfg configuration file in Windows. The configuration file contains parameters that specify data migration operations such as start, migration, verification, and retry.
   ├── import.sh           # The script that automatically imports files based on the conf/local_job.cfg configuration file in Linux. The configuration file contains parameters that specify data migration operations such as start, migration, verification, and retry.
   ├── logs                # The directory that contains logs.
   └── README.md           # The file that provides a description of ossimport. We recommend that you read this file before you use ossimport.

2. Modify the following configuration files based on your business requirements: conf/sys.properties and conf/local_job.cfg.

   Do not modify the following items:

   • The workingDir, workerUserName, workerPassword, and privateKeyFile parameters in the conf/sys.properties file.

   • The name and path of the conf/local_job.cfg file and the jobName parameter in the configuration file.

   For more information about configuration examples, see Configuration file examples.

   Important

   Before you submit a migration job, check the parameters in the sys.properties and local_job.cfg files. The parameters of a migration job cannot be modified after the job is submitted.

3. Start migration jobs.

   • Double-click the import.bat file in Windows.

   • Run the bash import.sh command in Linux.

Run migration jobs

In ossimport deployed in standalone mode, you can run migration jobs by using one of the following methods:

• One-click import: This method encapsulates all steps into a script. You can run a migration job by following the prompts of the script. If you are a beginner, we recommend that you use this method to run migration jobs. For more information, see Quick start.

  1. Start one-click import.

     • Double-click the import.bat file in Windows.

     • Run the bash import.sh command in Linux.

     Note

     If the last migration job that you run is not complete, ossimport asks you whether to continue the job from the recorded checkpoint or restart data synchronization. If you want to start another migration job or have modified the migration source or destination, restart data synchronization.

  2. In Windows, a new Command Prompt window is opened to run the job and display logs. In the Command Prompt window that runs ossimport, the status of the job is displayed every 10 seconds. Do not close the Command Prompt windows when the job is running. In Linux, the job is run in the background.

  3. If a task fails after the job is run, you are prompted whether to retry. To retry the task, enter y. To skip the task and exit, enter n.

  4. If the job fails, you can view the master/jobs/local_test/failed_tasks/<taskid>/audit.log file to identify the cause of the failure.

• Step-by-step import: This method runs a migration job by performing multiple steps.

  1. Clear existing jobs that have the same name.

     If you want to run a migration job that has the same name as an existing job, clear the existing job first. If you want to run a new migration job or retry the failed tasks of a migration job, do not run the clear command.

     • In Windows, run the console.bat clean command in Command Prompt.

     • In Linux, run the bash console.sh clean command.

  2. Submit a migration job.

     Make sure that the name of your migration job is unique. The configuration file of the job to submit is conf/local_job.cfg. The default name of the job to submit is local_test. Run one of the following commands to submit the job:

     • In Windows, run the console.bat submit command in Command Prompt.

     • In Linux, run the bash console.sh submit command.

  3. Start the service.

     • In Windows, run the console.bat start command in Command Prompt.

     • In Linux, run the bash console.sh start command.

  4. View the job status.

     • In Windows, run console.bat stat in the Command Prompt.

     • In Linux, run the bash console.sh stat command.

  5. Retry failed tasks.

     Tasks may fail due to reasons such as network issues. When you run the retry command, only failed tasks are retried.

     • In Windows, run the console.bat retry command in Command Prompt.

     • In Linux, run the bash console.sh retry command.

  6. Stop the service.

     • In Windows, close the %JAVA_HOME%/bin/java.exe window.

     • In Linux, run the bash console.sh stop command.

View the job status and logs

ossimport
├── bin
│   └── ossimport2.jar    # The JAR package of ossimport deployed in standalone mode.
├── conf
│   ├── local_job.cfg     # The job configuration file.
│   └── sys.properties    # The configuration file that contains system parameters.
├── console.sh            # The command-line tool.
├── import.sh             # The script for one-click import.
├── logs
│   ├── import.log        # The migration logs.
│   ├── job_stat.log      # The logs that record job status.
│   ├── ossimport2.log    # The run logs of ossimport deployed in standalone mode.
│   └── submit.log        # The logs that record job submission.
├── master
│   ├── jobqueue                # The jobs that are not split.
│   └── jobs                    # The information about jobs.
│       └── local_test          # The name of the job.
│           ├── checkpoints     # The checkpoints generated when the master splits the job into tasks.
│           │   └── 0
│           │       └── 034DC9DD2860B0CFE884242BC6FF92E7.cpt
│           ├── dispatched      # The tasks that are dispatched to the worker but not complete.
│           │   └── localhost
│           ├── failed_tasks    # The failed tasks.
│           ├── pending_tasks   # The tasks that are not dispatched.
│           └── succeed_tasks   # The tasks that are successfully run.
│               └── A41506C07BF1DF2A3EDB4CE31756B93F_1499744514501@localhost
│                   ├── audit.log   # The run logs of the task. You can view the logs to identify error causes.
│                   ├── DONE        # The mark file that indicates the completion of the task.
│                   ├── error.list  # The errors of the task. You can view the errors in the file.
│                   ├── STATUS      # The mark file that indicates the task status. The content of this file is Failed or Completed.
│                   └── TASK        # The description of the task.
└── worker      # The tasks that are being run by the worker. After tasks are run, they are managed by the master.
    └── jobs
        └── local_test
            └── tasks

Verify migration results

ossimport does not verify data after migration and therefore does not ensure data consistency or integrity. After a migration job is complete, remember to verify data consistency between the migration source and destination.

If you delete the source data without verifying data consistency between the migration source and destination, you are responsible for any losses and consequences that arise.

Common causes of migration failures

• If the source data is modified during migration, a SIZE_NOT_MATCH error is recorded in the log/audit.log file. This error indicates that the original data has been uploaded, but modifications are not uploaded to Object Storage Service (OSS).

• If the source data is deleted during migration, the migration job fails.

• If the source files do not comply with the naming rules of OSS objects, the migration to OSS fails. The name of an object in OSS cannot start with a forward slash (/) and cannot be empty.

• If network issues occur or your account does not have the required permissions, the source data fails to be downloaded. To identify the actual cause, view the logs/ossimport2.log or logs/import.log file.

• If ossimport exits unexpectedly when you run a migration job, the state of the job is Abort. In this case, contact technical support.