This topic describes how to use a UI-based migration tool that is provided by Microservices Engine (MSE) to migrate traffic from a self-managed NGINX Ingress gateway to an MSE Ingress gateway.
Prerequisites
A Container Service for Kubernetes (ACK) cluster, ACK Serverless cluster, or ACS cluster is created, and NGINX Ingress Controller is deployed in the cluster.
A cloud-native gateway is created. If you do not have a cloud-native gateway, create a cloud-native gateway.
Usage notes
In this topic, an MSE Ingress gateway reuses Ingress configurations during the migration. Ingress configurations are not copied. The gateway listens to the changes of existing Ingress configurations in real time and parses the listened configuration.
During the migration, both NGINX Ingress Controller and the MSE Ingress gateway can perceive changes of Ingress configurations.
After the migration is complete, do not delete online NGINX Ingress configurations that are in use. The migration aims to allow the MSE Ingress gateway to listen to and parse existing and new Ingress configurations.
After the migration is complete, the existing and new Ingress configurations still must be associated with the Ingress class of the NGINX Ingress gateway. For example, if the value of ingressClassName in spec for the NGINX Ingress gateway is nginx, this setting remains unchanged for both existing and new Ingress configurations after the migration.
Migration methods and principles
What is an MSE Ingress gateway?
An MSE Ingress gateway provides a powerful method to manage Ingress traffic based on MSE cloud-native gateways. Traffic gateways, microservices gateways, and security gateways are integrated into MSE Ingress gateways. MSE Ingress gateways are compatible with Kubernetes Ingress API standards. MSE Ingress gateways resolve the issues that are caused by independent design and O&M of the three-layer gateway architecture. The issues include high resource usage, large performance loss, low stability, and complex security protection.
Benefits of MSE Ingress gateways
Compared with traditional gateways, MSE Ingress gateways provide the following benefits in terms of resource costs, performance, security protection, and ease of use:
Standards compliance: MSE Ingress gateways strictly follow Kubernetes Ingress API standards. The
networking.k8s.io/v1beta1andnetworking.k8s.io/v1versions are supported.High compatibility: MSE Ingress gateways are suitable for more than 90% of scenarios in which NGINX Ingress annotations are used. NGINX Ingress configurations can directly take effect on MSE Ingress gateways without modifications. For more information about the annotations supported by MSE Ingress gateways, see Annotations supported by MSE Ingress gateways.
Annotation extension: In addition to annotations that are defined in NGINX Ingress gateways, MSE Ingress gateways provide exclusive annotations that are related to features such as authentication, header control, throttling, and security protection. For more information about the exclusive annotations supported by MSE Ingress gateways, see Advanced usage of MSE Ingress.
Secure architecture: The data plane is separated from the control plane to isolate resources and improve the security of the architecture design.
High performance: HTTPS hardware acceleration is supported and queries per second (QPS) performance is improved.
Background information
The following issues may occur when you use an NGINX Ingress gateway:
Major security vulnerabilities: CVE-2021-25745, CVE-2021-25746, and CVE-2021-25748.
MSE Ingress gateways are closely coupled with the Ingress API versions and Kubernetes cluster versions. This adversely affects the process of upgrading Kubernetes clusters. In ACK managed clusters that run Kubernetes V1.24, ACK Serverless clusters that run Kubernetes V1.24, and ACS clusters that run Kubernetes V1.26, Ingress resources whose API version is networking.k8s.io/v1beta1 are no longer used. Ingress resources whose API version is networking.k8s.io/v1 are supported in NGINX Ingress Controller V1.0.0 and later. To upgrade the above-mentioned clusters, you must upgrade NGINX Ingress Controller.
The process of upgrading NGINX Ingress Controller is complex and traffic loss may occur during the upgrade process.
An additional set of NGINX Ingress Controller of a new version must be deployed.
The template parameters must be manually modified to upgrade the application version.
Only DNS resolution is supported when you migrate traffic from an old gateway to a new gateway. The timeliness is poor and rollback is not allowed when an issue occurs.
After you migrate traffic from an NGINX Ingress gateway to an MSE Ingress gateway, you can decouple the Ingress gateway from the Ingress API version and Kubernetes cluster version. This way, no version requirements are imposed on NGINX Ingress Controller during the upgrade of ACK clusters. MSE Ingress gateways provide a tool for migration scenarios. You can reuse Server Load Balancer (SLB) instances to perform fine-grained traffic switching. This improves the timeliness and stability of migration.
Migration methods
MSE Ingress gateways provide the following migration methods:
Reuse SLB instances
Principle: An MSE Ingress gateway reuses the SLB instance that is associated with the service of NGINX Ingress Controller in the ACK managed cluster, ACK Serverless cluster, or ACS cluster. The nodes of the MSE Ingress gateway are automatically added to the vServer group that has listeners and the original SLB instance. The traffic is migrated based on the configured traffic weight.
The MSE Ingress gateway reuses the existing SLB instance based on the original traffic handling process and automatically synchronizes the original rules of the NGINX Ingress gateway. If the verification is successful, the traffic is gradually switched to the MSE Ingress gateway. The original traffic ingress SLB instance remains unchanged in the entire migration process. You do not need to change DNS servers for traffic switching.
Use DNS resolution
Principle: The resolution result of the SLB instance that is associated with the MSE Ingress gateway is added for the service domain names that are associated with all NGINX Ingress gateways in the DNS server. Some DNS service providers provide weight values to control the traffic percentages of the SLB instance that is associated with the NGINX Ingress gateway and the SLB instance that is associated with the MSE Ingress gateway.
Method comparison
Migration method | Timeliness | Traffic switching level | Weight-based switching | Operation complexity |
Reuse SLB instances | Fast | SLB level | Supported | Easy |
Use DNS resolution | Slow | Domain name level | Depends on DNS service providers | Proportional to the number of domain names |
Overview of the migration solution
MSE provides a migration tool for you to migrate route configurations and traffic as prompted.
Step 1: Migrate routing rules
Log on to the MSE console. In the top navigation bar, select a region.
In the left-side navigation pane, choose Cloud-native Gateway > Migration to Cloud.
On the Migration to Cloud page, click Add Task.
In the Create Migration Configuration panel, configure the parameters.
The MSE cloud-native gateway automatically listens to the changes of all Ingress resources that are associated with the source Ingress class in the cluster, and makes the configurations of the domain names and routes of the Ingress resources take effect.
If the destination cloud-native gateway is associated with the cluster and the existing Ingress class of the cluster is not the same as the Ingress class that is configured in this step, migration is not allowed. You must make sure that the Ingress class configured in this step is the same as the Ingress class configured for the associated cluster.
Parameter
Description
Parameter
Description
Cloud-native Gateway
The MSE cloud-native gateway to which you want to migrate routing rules. The version of the MSE cloud-native gateway must be V1.2.22 or later.
ACK/ASK/ACS Cluster
The cluster to which the NGINX Ingress gateway belongs. You must make sure that the MSE cloud-native gateway and the cluster are in the same virtual private cloud (VPC).
Source Ingress Class
The Ingress class with which the Ingress resources to be migrated are associated.
Only one Ingress class is supported.
If you do not specify this parameter, the MSE cloud-native gateway listens to all the Ingress resources in the cluster.
Click Next.
The MSE cloud-native gateway automatically listens to the changes of all Ingress resources that are associated with the source Ingress class in the cluster, and makes the configurations of the domain names and routes of the Ingress resources take effect.
Deploy an Ingress named httpbin in the cluster.
In the MSE console, check that the Ingress of the cluster is automatically synchronized to the MSE cloud-native gateway, and the domain names and routes of the Ingress are generated.
Step 2: Check routes
Check the compatibility of the Ingress resources to which the cloud-native gateway listens.
If all Ingress annotations are compatible with the cloud-native gateway, proceed to the next step.
If any Ingress annotation is incompatible with the cloud-native gateway, click the link to submit a ticket.
The annotation
nginx.ingress.kubernetes.io/service-weightwith the value of double quotation marks ("") can be ignored. This annotation is added by default in the old ACK console and does not make sense.During the migration, do not delete incompatible annotations that are in use. These incompatible annotations are still parsed by NGINX Ingress Controller and take effect on your business traffic. You can add MSE extended annotations to Ingress resource configurations to implement the same feature on the MSE Ingress gateway. After all traffic is migrated to the MSE Ingress gateway, you can delete the incompatible annotations based on your business requirements.
Step 3: Select a traffic switching method
Test traffic distribution before traffic switching
We recommend that you perform a local test by performing the following operations before you switch traffic: Modify the local hosts file, add the mappings of IP addresses of Server Load Balancer (SLB) instances associated with cloud-native gateways for DNS services, and then use tools such as curl or Postman to check whether traffic distribution meets your expectations.
Select a traffic switching method
In this method, you need to add the MSE cloud-native gateway to the backend vServer group of the SLB instance that is associated with the NGINX Ingress gateway. During the migration, the SLB instance distributes business traffic to the MSE cloud-native gateway based on the configured weight. After the migration is complete, all business traffic on the SLB instance is switched to the MSE cloud-native gateway.
The following table describes the parameters when this method is selected.
Parameter | Description |
ACK Cluster Namespace | The namespace of the Kubernetes service that corresponds to the SLB instance associated with the NGINX Ingress gateway. |
ACK Cluster SLB Service | The name of the Kubernetes service that corresponds to the SLB instance associated with the NGINX Ingress gateway. |
SLB ID | The ID of the SLB instance on which the traffic you want to switch. |
Ports and Backend Servers | The listener port of the SLB instance in the cluster and the gateway protocol (HTTP or HTTPS). After you select the port and protocol, the vServer group is automatically displayed. Make sure that the port and protocol that you select are valid. Otherwise, traffic loss may occur. |
For the DNS service provided by the DNS vendor, add the mappings between all domain names involved in route migration and the IP address of the SLB instance that is associated with the cloud-native gateway. We recommend that you use DNS records to gradually switch traffic based on configured weight values.
Step 4: Switch traffic
Step 1: Click Change SLB
After you click Change SLB, the SLB instance is no longer managed by the cluster and the listener scheduling algorithm is changed to weighted round-robin.
After the change, the SLB instance cannot detect the changes in the pod IP address of NGINX Ingress Controller. Complete the next step at the earliest opportunity to re-associate the Kubernetes service with the SLB instance.
Step 2: Overwrite service annotations
After Step 1 is complete, log on to the ACK console to manually delete all the annotations of the Kubernetes service that you specify when you select Reuse Original Cluster SLB, and copy the annotations that are automatically generated on the Switch Traffic tab to the annotations of the Kubernetes service. This step enables the Kubernetes service to reuse the SLB instance. After the operation is complete, click Pre-check. After the check is passed, go to the next step.
If application pods in the same cluster access the NGINX Ingress gateway, add the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: mse-ingress-migration to the Kubernetes service. Before you add the annotation, make sure that IP address-based access control is not enabled for the SLB instance that corresponds to the Kubernetes service. This annotation forcefully enables application pods to access the NGINX Ingress gateway through the SLB instance. After the annotation is added, SLB instance bypass based on kube-proxy is not used.
Step 3: Switch traffic based on the configured weight
Specify the weight based on which traffic is switched to the MSE cloud-native gateway. You can set the weight to a value ranging from 1 to 100 based on your business requirements. We recommend that you set the weight to a value ranging from 1 to 10 for the first time.
The value is the sum of weights of all nodes in the MSE cloud-native gateway. The SLB instance distributes traffic based on the weight of each node in the MSE cloud-native gateway and the weight of each node in the NGINX Ingress gateway in the vServer group. The default total weight of all nodes in the NGINX Ingress gateway is 100. If you set the weight of the cloud-native gateway to 100, half of the traffic is routed to the cloud-native gateway. In the same way, if you set the weight of the cloud-native gateway to 50, 1/3 of the traffic is routed to the cloud-native gateway.
During the migration, you can view the metrics of the cloud-native gateway on the dashboard in the MSE console. You can monitor the health status of the cloud-native gateway and check whether business metrics are as expected.
If business metrics are as expected, you can gradually increase the weight of the cloud-native gateway. The weight configuration takes effect asynchronously in the background. Therefore, change the weight at an interval of more than 3 minutes.
If business metrics are not as expected, set the weight to 0 to terminate the migration.
To increase the weight of the cloud-native gateway, you can also decrease the value of the annotation
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weightfor the Kubernetes service that you specify when you select Reuse Original Cluster SLB. If you set the annotation to 0, all traffic on the SLB instance is routed to the cloud-native gateway.SLB serves as a Layer-4 load balancer that uses connection-level throttling. The weight cannot accurately control the request distribution proportion.
If the success rate decreases after traffic switching, you can set the weight to 0 to perform quick traffic rollback.
If you want to maintain the migrating state and be able to adjust the weights of the NGINX Ingress gateway and MSE Ingress gateway at any time, we recommend that you stay in this step for a long period of time. After you verify that the traffic is as expected and traffic rollback is not required, click Complete Traffic Verification to go to the next step.
After you click Complete Traffic Verification, you cannot modify the weights.
Step 4: Switch all traffic to the cloud-native gateway
In the ACK console, set the value of the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight for the Kubernetes service that you specify in Step 3: Select a traffic switch method to 0. You can also directly delete the Kubernetes service resources. NGINX Ingress Controller is automatically removed from the SLB instance. All traffic on the SLB instance is switched to the cloud-native gateway. Click Complete Migration to complete the migration task.
For the DNS service provided by the DNS vendor, add the mappings between all domain names involved in route migration and the IP address of the SLB instance that is associated with the cloud-native gateway. We recommend that you use DNS records to gradually switch traffic based on configured weight values.
Perform a rollback
If the traffic distribution is not as expected, you can use one of the following methods to immediately roll back the traffic to NGINX Ingress Controller.
Reuse Original Cluster SLB: Set the weight to 0 to terminate the migration.
DNS Resolution to SLB: For the DNS service provided by the DNS vendor, delete the mappings between the business domain names and the IP address of the SLB instance that is associated with the MSE cloud-native gateway.
Step 5: Complete the migration
If you select this method, make sure that all traffic on the SLB instance is distributed to the MSE cloud-native gateway. Then, you can delete the Kubernetes service and NGINX Ingress Controller based on your business requirements.
If you select this method, make sure that all domain names are parsed as the IP addresses of the SLB instance that is associated with the MSE cloud-native gateway. Then, you can delete the Kubernetes service and NGINX Ingress Controller based on your business requirements.
FAQ
What do I do if the error message mse.backend.gw.MIGRATE_INGRESS_CLASS_CONFLICT is reported?
The MSE cloud-native gateway is associated with the cluster, and Ingress listening is enabled. However, the Ingress class of the MSE cloud-native gateway differs from the source Ingress class. As a result, this error message is returned.
To modify the Ingress class, perform the following operations:
If you use an MseIngressConfig resource to manage MSE Ingresses, follow the instructions in Use MSE Ingress gateways to access services in ACK clusters and ACS clusters.
If you do not use the MseIngressConfig resource to manage MSE Ingresses, modify the Ingress listening settings of the cluster in the MSE console.
What do I do if the error message mse.backend.gw.MIGRATE_SERVICE_ANNOTATION_NOT_MATCH [annotation is not expected] is reported?
This issue occurs because the annotation of the Kubernetes service is not as expected.
What do I do if the error message mse.backend.gw.MIGRATE_SERVICE_NOT_MATCH [weight hasn't be 0 or service hasn't been deleted] is reported?
This issue occurs because the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight for the Kubernetes service is not set to 0, or the Kubernetes service is not deleted. In this case, change the value of the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight for the Kubernetes service to 0 in the ACK console, or delete the Kubernetes service.
