MSE Sync is a data synchronization tool that is customized and optimized based on the open source Nacos Sync project. MSE Sync provides two-way data synchronization capabilities and is integrated with efficient features such as automatic service discovery and one-click full service synchronization. MSE Sync uses the multi-instance deployment architecture to implement a high-level disaster recovery mechanism. MSE Sync is suitable for scenarios in which data needs to be smoothly migrated between configuration management platforms such as Nacos, Zookeeper, and Eureka instances.
Solution overview
Benefits
Smooth migration
During the migration process, MSE Sync keeps the service information of the source instance consistent with the service information of the destination MSE instance. This ensures smooth migration from self-managed instances to MSE instances.
In the figures in this topic, pub refers to the service publisher, and sub refers to the service subscriber.
Multi-instance deployment
Multi-instance deployment is supported in service synchronization mode.
MSE Sync uses scheduled tasks to synchronize data that is lost on failed nodes. If a node fails, normal nodes use scheduled tasks to synchronize data that is lost due to the node failure. During the period between the node failure and the execution of scheduled tasks, the instance list on the client may be empty for a short period of time. You can use the following parameter to specify the interval at which scheduled tasks are performed. This parameter is a Java virtual machine (JVM) parameter and can be specified in the JAVA_OPT environment variable.
-Dmsesync.daemon.refresh.interval=60 // Unit: seconds.
The following code shows an example.
JAVA_OPT="$JAVA_OPT -Dmsesync.deamon.refresh.interval=10"
ImportantThe default interval for scheduled tasks is 60 seconds. Do not set the interval to a small value. Otherwise, the server and synchronization nodes may be overloaded. You must configure the interval based on the loads of the instance and the MSE Sync tool. We recommend that you set the interval to 10 seconds or longer.
Do not perform multi-instance deployment during configuration synchronization.
Ease of use
MSE Sync is compatible with the service registration logic of Zookeeper, Nacos, and Eureka engines. You can use MSE Sync to synchronize service information from open source ZooKeeper instances to MSE ZooKeeper or Nacos instances, from open source Nacos instances to MSE Nacos instances, and from open source Eureka instances to MSE Nacos instances. MSE Sync can automatically obtain service information from open source ZooKeeper instances, Nacos instances, and Eureka instances. You can select all instances at a time.
Supported instance types
The following table describes the instance types that are supported by MSE Sync.
Source instance type | Destination instance type | Synchronization support description |
Nacos | Nacos | The source instances are open source Nacos instances. |
ZooKeeper | Nacos | The Apache Curator-based service discovery feature and Dubbo services are supported. |
ZooKeeper | ZooKeeper | Service information and persistence configurations can be synchronized. Important
|
Eureka | Nacos | The source instances are open source Eureka instances. In this migration scenario, you must configure the name of the service that is registered with the MSE Nacos instance in lowercase letters. When you register a service with an open source Eureka instance, the registered service name is automatically converted into uppercase letters. When you use MSE Sync to synchronize service information to an MSE Nacos instance, MSE Sync automatically converts the service name into lowercase letters. As a result, if a service name contains uppercase letters, the service that is registered with the open source Eureka instance cannot connect to its synchronized service on the MSE Nacos instance. For example, a service named Service-1 is used for migration. After the service is registered with an open source Eureka instance, the service name is changed from Service-1 to SERVICE-1. When MSE Sync synchronizes the service information to an MSE Nacos instance, MSE Sync changes the service name from SERVICE-1 to service-1. However, if the service is registered with the MSE Nacos instance by using Nacos SDKs, the service name remains Service-1. As a result, service-1 and Service-1 are considered two different services, and the service information cannot be synchronized to the MSE Nacos instance. This issue can be resolved if you configure the service name in lowercase letters. |
Consul | Nacos | The source instances are open source Consul instances. MSE Sync synchronizes only the information related to healthy instances. MSE Sync is not aware of updates in metadata information. Therefore, updates in metadata information are not automatically synchronized. MSE Sync does not support the synchronization of Consul configuration information. |
Step 1: Deploy MSE Sync
MSE Sync needs to connect to both the open source instance and MSE instance. Therefore, you must deploy MSE Sync in a network environment where you can access both the open source instance and MSE instance. Otherwise, the migration fails. For example, you can deploy MSE Sync in a virtual private cloud (VPC) where both the open source instance and MSE instance are deployed or establish network connections between MSE Sync and the instances over the Internet.
Deployment modes
Use a MySQL database
You must build a MySQL database, and create an MSE Sync account that has the read and write permissions on the MySQL database. Specify the URL of the MySQL database, account username, and account password for MSE Sync by using environment variables. Multiple MSE Sync tools can share one MySQL database.
Use a Derby database
MSE Sync supports embedded Derby databases and can be started on a standalone basis.
Environment variables
Database-related environment variables
MSE Sync determines whether to use a MySQL database or an embedded Derby database based on environment variables. If the values of
MYSQL_URL
,MYSQL_USER_NAME
, andMYSQL_PASSWORD
are not left empty, MSE Sync is started by connecting the given environment to a MySQL database. In other cases, MSE Sync is started by using an embedded Derby database.Environment variable
Description
Example
MYSQL_URL
The URL of the MySQL database.
jdbc:mysql://localhost:3306/sync
MYSQL_USER_NAME
The username of the MSE Sync account that has read and write permissions on the MySQL database.
None
MYSQL_PASSWORD
The password of the MSE Sync account that is specified by MYSQL_USER_NAME.
None
JVM-related environment variables
Add JVM parameters to the JAVA_OPT variable in the java.env file in the conf folder.
For example, you can use
JAVA_OPT="$JAVA_OPT -Xms2048m -Xmx2048m"
in the code to specify the JVM heap memory.
Deployment methods
Deploy MSE Sync in a Kubernetes cluster
Modify the parameters such as MYSQL_USER_NAME, MYSQL_PASSWORD, and
replicas
in the following configurations. Save the modified configurations as themse-sync-deployment.yaml
file.apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/name: mse-sync name: mse-sync-svc spec: replicas: 3 selector: matchLabels: app.kubernetes.io/name: mse-sync template: metadata: labels: app.kubernetes.io/name: mse-sync spec: containers: - name: mse-sync image: msecrinstance-registry.cn-hangzhou.cr.aliyuncs.com/mse-demo/mse-sync:latest ports: - containerPort: 8000 env: - name: MYSQL_URL value: "jdbc:mysql://192.xx.xx.xx:3306/sync" - name: MYSQL_USER_NAME value: "mse-sync" - name: MYSQL_PASSWORD value: password
NoteIf you need to access the MSE Sync UI, you must add port 8000 that points to MSE Sync.
Run the following command to deploy MSE Sync:
kubectl apply -f mse-sync-deployment.yaml
Deploy MSE Sync on an ECS instance
Download the binary package of MSE Sync.
curl https://msesync.oss-cn-hangzhou.aliyuncs.com/MseSync.zip --output msesync.zip
Decompress the binary package and go to the root directory of MSE Sync.
unzip ./msesync.zip
Optional. If a MySQL database is used to start your MSE Sync, configure the following environment variables:
export MYSQL_URL="" export MYSQL_USER_NAME="" export MYSQL_PASSWORD=""
Start MSE Sync.
./MseSync/bin/startup.sh start
Step 2: Configure instance information and migration tasks
You can configure migration tasks on the MSE Sync UI. You can also create migration tasks by using configuration files.
MSE Sync UI
Open the MSE Sync tool. In the left-side navigation pane, click Service Synchronization. Then, click Import Config. In the Import Config dialog box, enter the configuration content and click OK.
Configuration file
Create a configuration file. In this example, the configuration file contains the following YAML code. The migration task that you create can automatically synchronize all services in the public and test namespaces from the src instance to the dst instance. The synchronization type is service information synchronization, and the synchronization mode is two-way synchronization.
The code contains two types of configuration fields:
clusters
andtasks
. The clusters fields specify the instance information, and the tasks fields specify the synchronization task information. For more information about theclusters
andtasks
fields, see Fields in this topic.clusters: - clusterName: src connectKeyList: - mse-xxxxx-nacos-ans.mse.aliyuncs.com:8848 clusterType: NACOS namespace: public,test - clusterName: dst connectKeyList: - mse-xxxxxx-nacos-ans.mse.aliyuncs.com:8848 clusterType: NACOS tasks: - source: src destination: dst type: Service mode: Bidirectional serviceMatchPattern: ".*" autoScanAndAddService: true
Run the following command to import the configuration file:
./bin/msesyncCMD.sh apply -f {Storage path of your configuration file}
Mount the config.yaml file or place the file in the
MseSync/conf
directory, and start MSE Sync. When MSE Sync starts, the configuration file is automatically initialized for data migration.
NoteWhen you configure your migration task on the Migration to Cloud page, the configuration file is automatically created in the Create Configuration step.
If you implement synchronization by importing configurations, new services are automatically synchronized.
Fields
The following table describes the clusters
and tasks
fields of the List type.
clusters fields
Field | Description | Example |
clusterName | The instance name. Note The instance name in the configuration must be unique. | sourceCluster |
clusterType | The instance type. |
|
namespace | The name of the Nacos namespace. Note You can enter multiple Nacos namespace names and separate them with commas (,). |
|
connectKeyList | The endpoint of the instance. This parameter is of the List type. |
|
userName | The username of the account that is used to log on to the MSE Nacos instance. | None |
password | The password of the account that is used to log on to the MSE Nacos instance. | None |
ak | The AccessKey ID of a RAM user that has the read and write permissions on MSE instances. | None |
sk | The AccessKey secret that corresponds to the AccessKey ID. | None |
tasks fields
Field | Description | Example |
source | The name of the source instance. | None |
destination | The name of the destination instance. | The namespace of the destination instance automatically keeps the same as that of the source instance. If a namespace does not exist in the destination instance, MSE Sync automatically creates the namespace. |
type | The synchronization type. |
|
mode | The synchronization mode. |
|
serviceMatchPattern | The regular expression that is used to match the service to be synchronized. | By default, information about all services is synchronized. |
autoScanAndAddService | Specifies whether to automatically scan and synchronize newly added services in the source cluster. | Default value: True. |