All Products
Search
Document Center

Microservices Engine:Migration solution based on MSE Sync

Last Updated:Dec 06, 2024

MSE Sync is a data synchronization tool that is customized and optimized based on the open source Nacos Sync project. MSE Sync provides two-way data synchronization capabilities and is integrated with efficient features such as automatic service discovery and one-click full service synchronization. MSE Sync uses the multi-instance deployment architecture to implement a high-level disaster recovery mechanism. MSE Sync is suitable for scenarios in which data needs to be smoothly migrated between configuration management platforms such as Nacos, Zookeeper, and Eureka instances.

Solution overview

Benefits

  • Smooth migration

    During the migration process, MSE Sync keeps the service information of the source instance consistent with the service information of the destination MSE instance. This ensures smooth migration from self-managed instances to MSE instances.

    In the figures in this topic, pub refers to the service publisher, and sub refers to the service subscriber.平滑迁移

  • Multi-instance deployment

    Multi-instance deployment is supported in service synchronization mode.多实例部署

    MSE Sync uses scheduled tasks to synchronize data that is lost on failed nodes. If a node fails, normal nodes use scheduled tasks to synchronize data that is lost due to the node failure. During the period between the node failure and the execution of scheduled tasks, the instance list on the client may be empty for a short period of time. You can use the following parameter to specify the interval at which scheduled tasks are performed. This parameter is a Java virtual machine (JVM) parameter and can be specified in the JAVA_OPT environment variable.

    -Dmsesync.daemon.refresh.interval=60 // Unit: seconds.

    The following code shows an example.

    JAVA_OPT="$JAVA_OPT -Dmsesync.deamon.refresh.interval=10"
    Important
    • The default interval for scheduled tasks is 60 seconds. Do not set the interval to a small value. Otherwise, the server and synchronization nodes may be overloaded. You must configure the interval based on the loads of the instance and the MSE Sync tool. We recommend that you set the interval to 10 seconds or longer.

    • Do not perform multi-instance deployment during configuration synchronization.

  • Ease of use

    MSE Sync is compatible with the service registration logic of Zookeeper, Nacos, and Eureka engines. You can use MSE Sync to synchronize service information from open source ZooKeeper instances to MSE ZooKeeper or Nacos instances, from open source Nacos instances to MSE Nacos instances, and from open source Eureka instances to MSE Nacos instances. MSE Sync can automatically obtain service information from open source ZooKeeper instances, Nacos instances, and Eureka instances. You can select all instances at a time.适配注册逻辑

Supported instance types

The following table describes the instance types that are supported by MSE Sync.

Source instance type

Destination instance type

Synchronization support description

Nacos

Nacos

The source instances are open source Nacos instances.

ZooKeeper

Nacos

The Apache Curator-based service discovery feature and Dubbo services are supported.

ZooKeeper

ZooKeeper

Service information and persistence configurations can be synchronized.

Important
  1. When you use MSE Sync to synchronize data between an open source ZooKeeper instance and an MSE ZooKeeper instance, you are not allowed to change data in the same Znode for the open source ZooKeeper instance and MSE ZooKeeper instance at the same time. Otherwise, the data change may be lost during the migration. Do not use the configuration synchronization method to synchronize services and ephemeral nodes.

  2. If you use MSE Sync to synchronize persistent nodes, MSE Sync registers watches with the source instance. Each Znode registers a watch. If the source instance has a large number of Znodes, a large number of watches are created at the same time and occupy excessive memory resources in the source instance. Make sure that the number of Znodes in the source instance and instance configurations allow MSE Sync to listen for changes.

Eureka

Nacos

The source instances are open source Eureka instances.

In this migration scenario, you must configure the name of the service that is registered with the MSE Nacos instance in lowercase letters. When you register a service with an open source Eureka instance, the registered service name is automatically converted into uppercase letters. When you use MSE Sync to synchronize service information to an MSE Nacos instance, MSE Sync automatically converts the service name into lowercase letters. As a result, if a service name contains uppercase letters, the service that is registered with the open source Eureka instance cannot connect to its synchronized service on the MSE Nacos instance.

For example, a service named Service-1 is used for migration. After the service is registered with an open source Eureka instance, the service name is changed from Service-1 to SERVICE-1. When MSE Sync synchronizes the service information to an MSE Nacos instance, MSE Sync changes the service name from SERVICE-1 to service-1. However, if the service is registered with the MSE Nacos instance by using Nacos SDKs, the service name remains Service-1. As a result, service-1 and Service-1 are considered two different services, and the service information cannot be synchronized to the MSE Nacos instance. This issue can be resolved if you configure the service name in lowercase letters.

Consul

Nacos

The source instances are open source Consul instances.

MSE Sync synchronizes only the information related to healthy instances. MSE Sync is not aware of updates in metadata information. Therefore, updates in metadata information are not automatically synchronized. MSE Sync does not support the synchronization of Consul configuration information.

Step 1: Deploy MSE Sync

Important

MSE Sync needs to connect to both the open source instance and MSE instance. Therefore, you must deploy MSE Sync in a network environment where you can access both the open source instance and MSE instance. Otherwise, the migration fails. For example, you can deploy MSE Sync in a virtual private cloud (VPC) where both the open source instance and MSE instance are deployed or establish network connections between MSE Sync and the instances over the Internet.

Deployment modes

  • Use a MySQL database

    You must build a MySQL database, and create an MSE Sync account that has the read and write permissions on the MySQL database. Specify the URL of the MySQL database, account username, and account password for MSE Sync by using environment variables. Multiple MSE Sync tools can share one MySQL database.

  • Use a Derby database

    MSE Sync supports embedded Derby databases and can be started on a standalone basis.

Environment variables

  • Database-related environment variables

    MSE Sync determines whether to use a MySQL database or an embedded Derby database based on environment variables. If the values of MYSQL_URL, MYSQL_USER_NAME, and MYSQL_PASSWORD are not left empty, MSE Sync is started by connecting the given environment to a MySQL database. In other cases, MSE Sync is started by using an embedded Derby database.

    Environment variable

    Description

    Example

    MYSQL_URL

    The URL of the MySQL database.

    jdbc:mysql://localhost:3306/sync

    MYSQL_USER_NAME

    The username of the MSE Sync account that has read and write permissions on the MySQL database.

    None

    MYSQL_PASSWORD

    The password of the MSE Sync account that is specified by MYSQL_USER_NAME.

    None

  • JVM-related environment variables

    Add JVM parameters to the JAVA_OPT variable in the java.env file in the conf folder.

    For example, you can use JAVA_OPT="$JAVA_OPT -Xms2048m -Xmx2048m" in the code to specify the JVM heap memory.

Deployment methods

  • Deploy MSE Sync in a Kubernetes cluster

    1. Modify the parameters such as MYSQL_USER_NAME, MYSQL_PASSWORD, and replicas in the following configurations. Save the modified configurations as the mse-sync-deployment.yaml file.

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          app.kubernetes.io/name: mse-sync
        name: mse-sync-svc
      spec:
        replicas: 3
        selector:
          matchLabels:
            app.kubernetes.io/name: mse-sync
        template:
          metadata:
            labels:
              app.kubernetes.io/name: mse-sync
          spec:
            containers:
            - name: mse-sync
              image: msecrinstance-registry.cn-hangzhou.cr.aliyuncs.com/mse-demo/mse-sync:latest
              ports:
              - containerPort: 8000
              env:
              - name: MYSQL_URL
                value: "jdbc:mysql://192.xx.xx.xx:3306/sync"
              - name: MYSQL_USER_NAME
                value: "mse-sync"
              - name: MYSQL_PASSWORD
                value: password
      Note

      If you need to access the MSE Sync UI, you must add port 8000 that points to MSE Sync.

    2. Run the following command to deploy MSE Sync:

      kubectl apply -f mse-sync-deployment.yaml
  • Deploy MSE Sync on an ECS instance

    1. Download the binary package of MSE Sync.

      curl https://msesync.oss-cn-hangzhou.aliyuncs.com/MseSync.zip --output msesync.zip
    2. Decompress the binary package and go to the root directory of MSE Sync.

      unzip ./msesync.zip
    3. Optional. If a MySQL database is used to start your MSE Sync, configure the following environment variables:

      export MYSQL_URL=""
      export MYSQL_USER_NAME=""
      export MYSQL_PASSWORD=""
    4. Start MSE Sync.

      ./MseSync/bin/startup.sh start

Step 2: Configure instance information and migration tasks

You can configure migration tasks on the MSE Sync UI. You can also create migration tasks by using configuration files.

  • MSE Sync UI

    Open the MSE Sync tool. In the left-side navigation pane, click Service Synchronization. Then, click Import Config. In the Import Config dialog box, enter the configuration content and click OK.

  • Configuration file

    1. Create a configuration file. In this example, the configuration file contains the following YAML code. The migration task that you create can automatically synchronize all services in the public and test namespaces from the src instance to the dst instance. The synchronization type is service information synchronization, and the synchronization mode is two-way synchronization.

      The code contains two types of configuration fields: clusters and tasks. The clusters fields specify the instance information, and the tasks fields specify the synchronization task information. For more information about the clusters and tasks fields, see Fields in this topic.

      clusters:
        - clusterName: src
          connectKeyList:
            - mse-xxxxx-nacos-ans.mse.aliyuncs.com:8848
          clusterType: NACOS
          namespace: public,test
        - clusterName: dst
          connectKeyList:
            - mse-xxxxxx-nacos-ans.mse.aliyuncs.com:8848
          clusterType: NACOS
      
      tasks:
        - source: src
          destination: dst
          type: Service
          mode: Bidirectional
          serviceMatchPattern: ".*"
          autoScanAndAddService: true
    2. Run the following command to import the configuration file:

      ./bin/msesyncCMD.sh apply -f {Storage path of your configuration file}
    3. Mount the config.yaml file or place the file in the MseSync/conf directory, and start MSE Sync. When MSE Sync starts, the configuration file is automatically initialized for data migration.

    Note
    • When you configure your migration task on the Migration to Cloud page, the configuration file is automatically created in the Create Configuration step.

    • If you implement synchronization by importing configurations, new services are automatically synchronized.

Fields

The following table describes the clusters and tasks fields of the List type.

clusters fields

Field

Description

Example

clusterName

The instance name.

Note

The instance name in the configuration must be unique.

sourceCluster

clusterType

The instance type.

  • NACOS

  • ZOOKEEPER

  • EUREKA

  • CONSUL

namespace

The name of the Nacos namespace.

Note

You can enter multiple Nacos namespace names and separate them with commas (,).

  • public

  • test

  • test1

connectKeyList

The endpoint of the instance. This parameter is of the List type.

connectKeyList:
- localhost:8848
- localhost1:8848

userName

The username of the account that is used to log on to the MSE Nacos instance.

None

password

The password of the account that is used to log on to the MSE Nacos instance.

None

ak

The AccessKey ID of a RAM user that has the read and write permissions on MSE instances.

None

sk

The AccessKey secret that corresponds to the AccessKey ID.

None

tasks fields

Field

Description

Example

source

The name of the source instance.

None

destination

The name of the destination instance.

The namespace of the destination instance automatically keeps the same as that of the source instance. If a namespace does not exist in the destination instance, MSE Sync automatically creates the namespace.

type

The synchronization type.

  • Service: service synchronization.

  • Config: configuration synchronization. Nacos and ZooKeeper instances support configuration synchronization.

  • All: The system automatically creates service and configuration synchronization tasks based on the original instance type.

mode

The synchronization mode.

  • Unidirectional: The native registration information is synchronized from the source instance to the destination instance.

  • Bidirectional: The native registration information is synchronized between the source instance and the destination instance.

serviceMatchPattern

The regular expression that is used to match the service to be synchronized.

By default, information about all services is synchronized.

autoScanAndAddService

Specifies whether to automatically scan and synchronize newly added services in the source cluster.

Default value: True.