Database migration is a complex and risky process that directly affects business stability and continuity. The compatibility and performance of the destination database have always been a major concern for users who need to migrate a database. Any compatibility issues or performance bottlenecks may result in migration failure and even cause business interruption and data loss. To effectively assess migration risks and ensure smooth migration, the traffic playback technology came into being. It provides an instrumental solution that performs comprehensive compatibility and performance tests on destination databases before migration. This minimizes potential risks. This topic describes the key role that the traffic playback feature of Cloud Migration Hub (CMH) plays during database migration. This topic also describes how to use the traffic playback feature. Traffic playback is not a simple database stress test, but a complete playback of real business traffic in the production environment. It simulates real business scenarios to comprehensively verify the performance and compatibility of destination databases. This is fundamentally different from traditional stress tests, which are based on preset scripts. Such stress tests usually cannot fully cover the complexity and diversity of real business scenarios, which can easily lead to deviations in test results. The traffic playback technology can accurately restore the operating status of the production environment, so as to evaluate the performance and compatibility of destination databases in a more effective manner.
1. Use the traffic playback feature
The process of using the traffic playback feature includes the following steps:
Step 1: Collect data. You need to select an appropriate tool to capture and record the read and write requests sent to the source database in the production environment. Exercise caution in this step to prevent excessive impacts on the production environment. Common data collection tools include database proxies, network packet capture tools, and the audit feature of databases. The collected data must include key information such as complete SQL statements, arguments, and execution time, which will be used as the input of traffic playback. Data must be collected and formatted in a way that is compatible with the traffic playback tool used in subsequent steps. You must mask sensitive information to ensure data security.
Step 2: Preprocess data. You need to preprocess the collected data, such as removing redundant information, optimizing data format, and performing data masking if required. Operations involved in data preprocessing, such as data cleansing, data conversion, and data sampling, help improve the traffic playback efficiency and reduce the load on the destination database. You may need to modify the SQL statements based on the characteristics of the destination database to ensure that they are executed as expected in the destination database.
Step 3: Perform traffic playback. The preprocessed data is imported to the traffic playback tool for playback. The traffic playback tool simulates real business traffic and performs concurrent access to the destination database to evaluate the performance and compatibility of the database. During this process, the key metrics such as CPU utilization, memory usage, and I/O performance of the destination database are monitored, and the error messages and exceptions reported during playback are recorded. The traffic playback tool allows you to customize settings such as the playback speed and concurrency level to simulate different business scenarios.
Step 4: Analyze results. After traffic playback is complete, you can analyze the collected statistics to evaluate the performance and compatibility of the destination database. For example, you can analyze metrics such as database response time, throughput, error rate, and resource utilization. This helps identify potential performance bottlenecks and compatibility issues, which provides a basis for optimizing the database migration solution. In addition, you need to analyze the error information reported during playback, find out the root cause, and take corresponding measures to resolve the errors.
Step 5: Generate a report. Finally, a detailed traffic playback report is generated, which is used to summarize the test results and offer suggestions for improvement. The report contains information such as the test environment configurations, test data summary, performance metrics, and error information to provide an important reference for database migration decisions.
You can use the traffic playback feature by following the preceding steps to effectively evaluate the compatibility and performance of the destination databases, reduce migration risks, and ensure smooth database migration. The traffic playback feature provides reliable analysis statistics for database migration projects, helping enterprises smoothly migrate databases with minimal business interruption and at a lower cost. Proper use of traffic playback tools can effectively improve the success rate of database migration projects.
2. Apply for and view a license
In the CMH console, choose Cutover > Traffic playback in the left-side navigation pane. On the Traffic playback page, an instruction is displayed. Click Apply for a License in the first step. A license is automatically generated. Licenses are used to protect agents from abuse. You are not charged for licenses, agents, or performing traffic playback. However, you are charged for purchasing Elastic Compute Service (ECS) instances to deploy agents.
After you apply for a license, you can view the validity period of the license, which is three calendar months. Click View License to view the license.
3. Install and manage an agent
After you apply for a license, you need to deploy an agent to perform traffic playback. In this case, you need to create one or more ECS instances to host the agent. Determine the number of ECS instances based on your business requirements.
Recommended specifications of the ECS instance: 8 CPU cores and 16 GB of memory or higher.
Recommended operating system: CentOS or Alibaba Cloud Linux.
Network requirements: The ECS instance is connected with the data source.
Other requirements: Java is installed on the ECS instance before an agent is deployed.
Note:
If you want to deploy an agent on an Alibaba Cloud ECS instance with Cloud Assistant installed, we recommend that you install an agent by using automated deployment. By default, Cloud Assistant is installed on Alibaba Cloud ECS instances.
If you want to deploy an agent on a host that is not an Alibaba Cloud ECS instance but is connected to an Alibaba Cloud virtual private cloud (VPC), or deploy an agent on an Alibaba Cloud ECS instance on which Cloud Assistant fails to be installed for some reason, we recommend that you manually deploy an agent that is accessible over a VPC.
If you want to deploy an agent on a host that is neither an Alibaba Cloud ECS instance nor connected to Alibaba Cloud but has a public IP address, we recommend that you manually deploy an agent that is accessible over the Internet.
The following sections describe the three types of deployment methods.
3.1 Automatically deploy an agent
Automated deployment is supported in the China (Hangzhou) and China (Shanghai) regions. If you want to deploy an agent in other regions, contact online after-sales.
Note:
Automated deployment supports only Alibaba Cloud ECS instances.
Before you install an agent on an ECS instance, you must activate PrivateLink.
After the required ECS instance is available, click Agent Management.
In the Agent Management panel, click Agent Installation.
In the Agent Installation dialog box, set the Installation method parameter to Automated Deployment, enter an agent name, select the region in which you want to deploy the agent and the ECS instance on which you want to install the agent, and then click Installation.
During the deployment, you can view the agent deployment process.
After the agent is deployed, you can view the agent in the agent list.
3.2 Manually deploy an agent
3.2.1 Manually deploy an agent that is accessible over the Internet
To manually deploy an agent that is accessible over the Internet, you must enable Internet access for the ECS instance. This way, CMH can connect with the ECS instance.
Note:
After the required ECS instance is available, click Agent Management.
In the Agent Management panel, click Agent Installation.
In the Agent Installation dialog box, set the Installation method parameter to Manual deployment, enter an agent name, set the LoadBalancerType parameter to Public network, enter an IP address and a port number, and then click Next step.
In the Installing the Agent step, an installation command is displayed. Click Copy to copy the command and then run the command on the ECS instance to install the agent.
After the agent is installed, click Next step. Then, click Connection to connect to the agent.
You can view the connection process online.
After the agent is deployed, you can view the agent in the agent list.
3.2.2 Manually deploy an agent that is accessible over a VPC
Manual deployment of an agent accessible over a VPC is supported in the China (Hangzhou) and China (Shanghai) regions. If you want to deploy an agent in this way in other regions, contact online after-sales.
Note:
In addition to Alibaba Cloud ECS instances, you can also use hosts that are not provided by Alibaba Cloud. Make sure that the hosts are connected to a VPC.
Before you install an agent on an ECS instance, you must activate PrivateLink.
After the required ECS instance is available, click Agent Management.
In the Agent Management panel, click Agent Installation.
In the Agent Installation dialog box, set the Installation method parameter to Manual deployment, enter an agent name, set the LoadBalancerType parameter to Private network, select a region and a VPC from the drop-down lists, enter an IP address and a port number, and then click Next step.
In the Installing the Agent step, an installation command is displayed. Click Copy to copy the command and then run the command on the ECS instance to install the agent.
After the agent is installed, click Next step. Then, click Connection to connect to the agent.
You can view the connection process online.
After the agent is deployed, you can view the agent in the agent list.
3.3 Manage an agent
3.3.1 Stop an agent
You can click Stop to stop an agent that is automatically deployed and is in the Online state.
After the agent on the ECS instance stops, the status of the agent changes to Stop. In this case, you cannot use this agent to run a traffic playback task.
3.3.2 Start an agent
You can click Start to start an agent that is in the Stop state.
After the agent is started, the status of the agent changes to Online.
3.3.3 Delete an agent
You can click Delete to delete an agent that you no longer need. Before you delete an agent, you must unbind the agent from the playback schedule.
3.3.4 Reconnect to an agent
After a manually deployed agent is disconnected from CMH, the status of the agent changes to Offline. After the agent is started, you can click Reconnect to connect CMH with the agent again.
After an agent is reconnected with CMH, the status of the agent changes to Online.
4. Create a playback schedule
4.1 Create a playback schedule
Prerequisites
An agent is installed.
The production traffic of the data source is captured. Select an appropriate method based on the data source to capture SQL data. After SQL data is collected, convert the SQL data format based on the sample JSON file: replay_sql_example.json.
The SQL file used for traffic playback is uploaded to the ECS instance on which the agent is installed.
After the agent is installed, you can create a playback schedule. The overall process consists of two steps: Create a playback schedule first, and then create a playback task in the playback schedule. A playback schedule can contain multiple playback tasks.
In the CMH console, choose Cutover > Traffic playback in the left-side navigation pane. On the Traffic playback page, click New Playback Schedule to create a playback schedule.
Step 1: Specify the basic information about the playback schedule.
Source and destination database types: Specify the type of the source and destination databases. The following figure shows the supported database types. Source databases cannot be PostgreSQL databases.
SQL file path: Enter the absolute path of the SQL file that is uploaded to the ECS instance on which the agent is installed.
Agent and database connectivity test: Select an agent and pass the connectivity test. Only agents that are in a normal state can be selected. After the playback schedule is created, you can create and run a playback task as expected.
Step 2: Configure parameters of the playback schedule, as shown in the following figures.
Number of simulated clients: the number of concurrent threads for playback. A higher value specifies a higher degree of concurrency.
Playback Speed Factor: the playback speed. A higher value specifies a faster speed. For example, a value of 1 specifies no speed boost, a value of 2 specifies twice the original speed, and a value of 10 specifies ten times the original speed.
4.2 View the details of the playback schedule
After the playback schedule is created, it is displayed on the Traffic playback page. You can view all the playback schedules within your account on this page. Click the name of the playback schedule or Details in the Operation column to go to the details page of the playback schedule.
Basic Information: displays information such as the schedule ID, source and destination database types, status of the last task, time when the schedule was created, and path of the SQL file.
Task Overview: displays the overall information about all playback tasks within the playback schedule, including the number of executed playback tasks, success rate of the last playback, number of successful requests, total number of requests, and agents associated with the playback schedule.
Task List: displays all the playback tasks within the playback schedule.
4.3 Create a playback task
After you create a playback schedule, you can create a playback task on the details page of the playback schedule.
Click Create Task to create a playback task. After a playback task is created, it is not immediately executed. You need to click Execution to execute the task. Therefore, you can create a playback task without executing it during the test. The following content describes the key parameters:
Select Agent: You can select an agent only in a normal state.
Number of simulated clients: the number of concurrent threads for playback.
Playback Speed Factor: the playback speed. Valid values are 1, 2, and 10. A value of 1 specifies no speed boost, a value of 2 specifies twice the original speed, and a value of 10 specifies ten times the original speed.
Whether to loop playback: specifies whether to enable loop playback. If you enable loop playback, the Playback Speed Factor parameter does not take effect and traffic is replayed at the maximum speed. After loop playback is enabled, the playback stops when the period specified by the Playback Run Time parameter elapses. During loop playback, repeated errors may be reported for SQL statements that write data, such as the INSERT and UPDATE statements. We recommend that you replay only DQL statements.
Playback Run Time: the playback period. Unit: seconds. After the period elapses, playback is automatically stopped.
schema mapping and filtering: the mapping relationship of schemas. For example, schema1:schema2,schema2 specifies that playback is performed on schema1 and schema2, and schema1 is mapped to schema2.
SQL timeout: the timeout period for executing an SQL statement.
Whether to commit: specifies whether to commit transactions. By default, the traffic playback tool frodo enables transactions to replay traffic. A SQL statement is rolled back after it is executed. If you need to commit transactions, you can set this parameter to Yes.
Whether to skip SQL error reporting: specifies whether to skip the SQL statements that have the same fingerprint as an SQL statement for which errors have been reported. By default, No is selected.
After a playback task is created, click Execution in the Operation column to execute the playback task.
4.4 View the progress of the playback task
After a playback task is created, you can click View Progress in the Status column to view the progress of the task.
4.5 View the playback report
After the playback task is executed, you can click Details in the Operation column to view the playback report.
Request Overview: displays the total number of requests, average execution time, and average response time.
Execution Details: displays the execution information of each SQL template in a list. The information includes sample SQL statement, schema, execution status, number of requests, number of request errors, and response time of the source and destination databases. You can also view error information by clicking Error Message in the Operation column.
Export Report: exports the report to an EXCEL file.
Elastic Compute Service (ECS)
Container Compute Service (ACS)
