This article is written based on a talk presented by Huang Xiaobin, the technical director at the Shenzhen Center of China Life Insurance (Overseas) Company Limited, during Dubbo Community Developers Day in Shenzhen.
China Life Insurance (Overseas) Company Limited is responsible for the business development of China Life in overseas markets outside of mainland China, including Hong Kong, Macao, Singapore, and Indonesia. Unlike its domestic business operations in China, its overseas businesses are held to different standards, different laws and regulations, needs to use different languages and currencies, and of course faces different challenges. Speaking more specifically, all of this leads to some technical challenges on the business support side of things. In this article, we will take a close look into China Life Insurance's experience in handling these challenges.
In 2013, at China Life Insurance we were looking for an RPC framework to transform our entire business database. At that time, few mature products were available on the market, unlike today where there are a great number of products available, such as Spring Cloud and Dubbo to name two. What we were looking for was a framework that had been applied in at a mass scale in a real-world production environment. Dubbo has been implemented for a long time in Taobao, and Alibaba's business model was comparable to our business model. For example, our business model needs to support requests from several different overseas regions, each with their own requirements for business.
Considering all of this, we started using Dubbo, also in 2013. Our business system was launched in Hong Kong and Macao in 2016 and in Indonesia in May 2019. In the future, our business system will also be rapidly deployed in Singapore and across Southeast Asia, thanks in part to using Dubbo.
Dubbo has helped us a lot in terms of deployment efficiency and lower costs. In this article, I would like to discuss the architecture we used before Dubbo—one which is common to many traditional insurance companies—and then discuss how we used Dubbo and how our architecture changed.
In terms of the server hardware, what we used, as what many other traditional insurance business systems would use is minicomputers, which are usually from IBM and HP. Only these two systems were available at the time, and both ran UNIX-based operating systems.
In terms of the business architecture, previous software development mainly used the Client/Server architecture. Under this kind of architecture, a powerful middleware called Tuxedo, which is used for distributed transaction management, and it offered excellent consistency and high-concurrency performance. So, why would we want to replace Tuxedo anyway? The first reason is the high price, the second reason is that related O&M personnel are in short supply, and the third reason is that it causes a high integration of business operations, which results in poor performance and scheduling in distributed and cross-platform scenarios. The fourth reason is that Tuxedo is designed for monolithic applications. As a result, we wouldn't dare ever to split it.
This figure shows the advantages of microservices when compared with a monolithic application. Dubbo serves as a bayonet for interaction between different components, which are similar to convex and concave parts in the LEGO toys. It implements communication management and service governance. Backed by this design, we break down a large-sized traditional core architecture.
OneLife is our business support platform. Its name suggests that one platform can handle all insurance business operations, form an internal ecosystem, and provide support for several different areas including images, new services, preservation, and claim settlement. It also provides engines, such as workflows, product engines, underwriting engines, and message engines. These business capabilities are essential for any insurance company.
Backed by Dubbo, we have built a new insurance business processing platform to achieve "Six-multi" insurance business processing. "Six-multi" represents the ideas of having multiple business systems, products lineups, sets of regulation to follow, engines, as well as currency languages. For example, to generate an insurance policy, you need to determine whether to generate the policy in English, Chinese, or Indonesian. This of course also requires us to cooperate with the respective business units, and then also design and develop an appropriate processing platform.
As shown in the figure, the following four parts can form a closed loop: Dubbo-based microservice calls, Jenkins-based continuous integration, Rancher-based cloud deployment applications, and Pinpoint-based chain monitoring. In terms of cloud deployment of the Indonesian version, we have been in discussion with Alibaba Cloud. In the future, the system in Indonesia may be the first to migrate to Alibaba Cloud. In terms of Pinpoint chain monitoring, the displayed topology is very clear. Based on the topology, however, we can see several problems with it such as the formation interruption or call failures. This year, with the help of Pinpoint chain monitoring, we have identified more than 100 bugs in the system. The combination of Pinpoint chain monitoring and Dubbo is excellent.
Dubbo has more than 150 servers, more than 210 applications, more than 2,100 consumers, and more than 1,300 providers distributed in Hong Kong and Macao. An insurance company, especially its own business system, does not have very many high-frequency transactions. Most high-frequency transactions occur at the front-end. Although the business system is not always in the high-frequency state, it is still important that we are able to ensure the stability and the accurate output of all of our business units. This is one of the reasons why we chose Dubbo.
About 70% of our business uses an early Dubbo version (specifically 2.4.9). We had previously made some code modifications on the basis of Dubbo, such as the compensation for distributed transactions. Later, we found that the version could not be upgraded due to the significant code modifications. So then, you may think, what is the framework that is used for the remaining 30% of our business operations? Well, we only dare to try the latest version on our periphery units. Also, this version is applied to the core business architecture only after it has been proven feasible by practical application. At present, Dubbo of China Life Insurance (Overseas) Company Limited is called for more than 21 million times a day, and no system crash has ever occurred since deployment of Dubbo.
We will share our configuration in this section. Two issues are emphasized. The first issue is the retry mechanism. The problem with this one is that, when a service is interrupted, the control platform is used for manual intervention, or the service is alternatively compensated by means of repeated transactions if the service is a key one. The second issue is that use of the ZooKeeper registration center has some disadvantages because the network consumption is huge during peak periods. The metadata concept in Dubbo version 2.7 is promising. In the future, we will try to see whether we can use the later version of Dubbo to overcome the disadvantages of ZooKeeper or optimize ZooKeeper.
The preceding figure shows the application of the Dubbo microservice. The application starts from the online service page, and then goes to the new service component. Then, a workflow is triggered, and the underwriting result is queried. Based on the underwriting result, the premium calculation result is automatically queried and returned to the front-end. When the core business system and sales system are deployed in Indonesia, low cost is a mandatory requirement. In addition, businesses units must be separated from each other. This involves the Base development model. What is the Base development model?
When businesses are distributed across multiple regions, each region must have its own specific version. As long as the public basic versions are the same, the work efficiency can be improved. For example, the headquarters has some basic services, but Indonesia has its own regulatory requirements. If Hong Kong also has this demand in the future, after the basic version passes the review of a certain level, the basic version is rolled back to the Base version, and then different Base versions are released for different regions. That is to say, in complex situations, the Base version supports hierarchical separation of different business logic, hierarchical code management, and hierarchical and region-based service governance.
First, strengthen visual management.
Second, introduce service grids and pack them into a package to provide more functions in the microservice system, including network calling, traffic limiting, circuit breaking, and monitoring between services.
Third, support multiple languages. Dubbo currently provides the PHP, Node.js, Python, and Go clients, and is expected to support more clients in the future.
Fourth, we do not recommend that Dubbo support distributed transaction management. When we first used Dubbo, we determined that it was necessary to support distributed transactions. Therefore, we changed the code on the basis of Dubbo. The application process was smooth and the code could ensure the consistency of our transactions even across platforms. However, when a service failed, all the transactions to be submitted collapsed, which brought a fatal risk to the database.
Therefore, we prefer to let Dubbo provide more support for the message mechanism, and to compensate for distributed transactions while doing business development. These are some ideas and suggestions that we have come up with through practical application.
Q1: Is the process for replacing monolithic applications with microservice components step-by-step or all-at-once? How much manpower and how many resources are required to complete the replacement process?
A1: First, the existing structure is modularized, and the modules are independent of each other. For the most important data structure, the business logic must be separated. Then the database shards or permissions are set, and modules are replaced step by step. Before development, the entire replacement process should be well planned. When the first module was started, we had only five people, but all of them were technically competent and proficient in their business fields. The top priority was to gain the support from the management of the company.
Q2: If distributed transaction control is unavailable in the system, how can we detect data inconsistency? How can this problem be solved?
A2: Pinpoint-based chain monitoring enables O&M personnel to quickly detect problems, rectify faults, and then perform manual intervention. We use the MQ mechanism for key services. However, this mechanism consumes a lot of time and manpower. We recommend that distributed transaction control not be used on a large scale.
Q3: When the volume of the company's internal data is large, how can we replace the Oracle database when migrating data from the old system to the new system? Can you answer this question?
A3: At the database level, you must ensure that the table structure and data of the Oracle database are consistent with those of other databases. You can use tools such as ETL for comparison. At the application level, you can take two steps. The first step is to automate the processing. You can use a self-compiled tool to execute the SQL statements of the application on both databases and correct errors in a timely manner. On the third round of execution, almost all SQL statements can be executed normally. The second step is to spot check the key algorithms or interfaces, and compare the business data with the data output by the new interfaces.
Let's Talk about Some of the Chaos You'll Find on Java Servers in Startups
203 posts | 12 followers
FollowAliware - April 10, 2020
Alibaba Cloud Native Community - November 22, 2023
Alibaba Clouder - March 16, 2018
Alibaba Clouder - April 16, 2020
Alibaba Clouder - March 14, 2018
Alibaba Clouder - March 23, 2020
203 posts | 12 followers
Follow
Microservices Engine (MSE)
MSE provides a fully managed registration and configuration center, and gateway and microservices governance capabilities.
Learn More
China Gateway Solution
Power your progress in China by working with the NO.1 cloud provider of this dynamic market.
Learn More
YiDA Low-code Development Platform
A low-code development platform to make work easier
Learn More
mPaaS
Help enterprises build high-quality, stable mobile apps
Learn MoreMore Posts by Alibaba Cloud Native
