miHoYo, a First-Generation Cloud-Native Enterprise, Realizes Imagination

By Jia Ningyu

In the office area of miHoYo, there is a meeting room reserved for Alibaba Cloud engineers.

This ACGN company was established nine years ago. The friendship between miHoYo and Alibaba Cloud has lasted for eight years.

Liu Wei, President of miHoYo, still remembers the day when Dr. Wang Jian arrived at miHoYo's office with eight company executives and a team from Alibaba Cloud. Back then, miHoYo only had about 30 seats in a small rented office. miHoYo had no meeting room for the visitors.

At that time, miHoYo was a newly founded company, and Alibaba Cloud was in the initial stage of external services. The two new companies came together by coincidence and began working together.

On that day, Dr. Wang said, "If the customer flies in the sky by airplane while we just look on the ground, breakdowns are easy to happen. To avoid this, we will fly in the sky with our customers."

When Dr. Wang left miHoYo, he gave his phone number to Liu Wei and said, "If you have any questions, call me directly."

Many years passed. Fortunately, Liu did not make that call. Alibaba's core system had already been fully migrated to the cloud, realizing the goal of "flying with customers."

Time passed quickly. In September 2020, after eight years of collaborations between miHoYo and Alibaba Cloud, miHoYo launched Genshin Impact. The Alibaba Cloud cloud sytem carries all of the servers of this game.

At the end of 2020, Genshin Impact's popularity in the gaming industry rose continuously.

On November 30, 2020, Google awarded the "Google Play – Best Game of 2020" to Genshin Impact. One later, Apple awarded the "Annual Best Game of 2020 in the App Store" to Genshin Impact.

This year, Alibaba Cloud became the leader in the cloud computing industry. It serves as the digital base of many industries and provides services for many Chinese gaming enterprises.

Start With belief

Back in 2012, miHoYo developed an animation game called Guns Girl.

In 2012, the mobile gaming market was still in its infancy, and there were only a few developers of animation games. In the App Store, there were few mature mobile animation games. miHoYo's leaders believed that there must be a place for a Chinese animation game in the future game market.

Coincidentally, the development process of cloud computing was somewhat similar to that of animation games. As a new technology, China's cloud computing industry was also in its infancy at that time. However, miHoYo and Alibaba Cloud both had a dream about the future.

No one knows how to arrive in the future; the only thing they can do is believe.

In the traditional IT era, gaming companies had to purchase servers, build server rooms, and allocate O&M personnel. The mode, which launches a game by investing a large amount of money, has stopped many entrepreneurs with ideas from entering the gaming industry. At that time, only large companies could afford a game's R&D.

Cloud computing brings new opportunities. Companies can skip all of the previous IT infrastructure investments and directly use cloud computing power with one click on the web page. This is a perfect product tailored for entrepreneurs like miHoYo.

An Early Photo of the miHoYo Founding Team

When the first version of Guns Girl launched, miHoYo used two Alibaba Cloud cloud servers. This attempt started the miHoYo "cloud tour."

Looking back, miHoYo was a representative of a generation of Internet venture enterprises. miHoYo deployed all of its businesses on the cloud. It enjoys the benefits of cloud computing, such as high elasticity, high concurrency, and low-cost. It can be called a "cloud-native enterprise."

It is the rapid growth of this generation of "cloud-native enterprises" that has promoted the development and popularization of cloud computing in China.

In 2016, miHoYo launched its third game, Honkai Impact 3rd. However, miHoYo still had only two O&M personnel at that time, which was unimaginable in the era of traditional self-built server rooms.

As an adopter of early cloud computing, miHoYo encountered problems. Liu Xiao, the Technical Director of miHoYo, claimed, "Problems are difficult to avoid. The key is whether the problems can be solved."

As the years have passed, miHoYo's Team has gradually grown from three people to thousands of people. They have come a long way from the original small team without a meeting room.

miHoYo also reserved a meeting room for Alibaba Cloud engineers to use at any time.

Safely Expanded the Markets Outside of China With Support From Alibaba Cloud

At first, the business scale of miHoYo was small. miHoYo encountered some twists and turns in the process of expanding markets outside of China.

"Fortunately, the negative impact of this incident could be controlled because our business scale was small and there were not many users. If it happened on Honkai Impact 3rd or Genshin Impact, the loss would be immeasurable." When talking about that, Liu Xiao still has some fear. He said, "For most game companies, the cloud is the best choice, and there is no other option."

In 2016, miHoYo officially began expanding to markets outside of China. In these service areas, the cloud service provided by the foreign cloud vendor for Honkai Impact 3rd suffered several DDoS attacks by hackers. Once, the attacks even affected users' normal access.

A DDoS attack is a very malicious network attack. Hackers send a large number of requests to the server within a short period, resulting in network congestion. Thus, users cannot access the server as normal.

For example, a restaurant that accommodates 100 customers opens. The owner of the restaurant next door feels unhappy and hires 150 hooligans to swarm the new restaurant. They occupy the seats but don't order anything. As a result, ordinary guests that want to eat can't be seated.

miHoYo also purchased additional security protection services to defend against attacks. Due to technical support and communication efficiency issues, the network congestion did not ease off in time.

At last, the miHoYo Team thought of Alibaba Cloud and came up with a method together with Alibaba Cloud engineers. Requests from players were sent to Alibaba Cloud Anti-DDoS Premium first and then were transferred back to the original server room after traffic cleansing. Thus, hundreds of GB of DDoS attacks were blocked. A problem that plagued miHoYo for a long time was solved once and for all.

By 2017, Alibaba Cloud had built more infrastructure worldwide. At the end of that year, Alibaba Cloud had 17 regions and 53 availability zones across Asia, Europe, and the USA. It was the best time for miHoYo to fully-migrate Honkai Impact 3rd to Alibaba Cloud.

Later, when planning Genshin Impact, miHoYo did not hesitate. The global resources of Genshin Impact were all provided by Alibaba Cloud. To date, the hacker attacks outside of China that miHoYo encountered previously have not appeared on Genshin Impact.

Genshin Impact, an Unprecedented Challenge

Genshin Impact is a great challenge faced by both miHoYo and Alibaba Cloud.

How grand is the plan of Genshin Impact? This is an open-world game, which means a huge map, various character images, complex level designs, and thorough storylines. miHoYo wants to make it a multi-platform and global-synchronous public beta game. This requires going online in five global service zones and simultaneous support on gaming platforms, including PS4, iOS, Android, and PC. Data in the same service zone are interconnected seamlessly.

For example, if you play Genshin Impact on your mobile phone and then switch to the computer, the gameplay and progress are seamlessly connected.

Moreover, miHoYo adopted a "general server" for a better online game experience.

Most game companies use the "server division" where one region has many servers. When the number of players reaches the limits of servers, more servers will be added. However, "general server" puts all users in a region on the same server cluster. The number of online players increases sharply in the same server.

After Genshin Impact launched, users from all over the world would pour in, the momentary peak concurrent users (PCU) could far exceed the estimated daily PCU.

If miHoYo designed such a product, can Alibaba Cloud support this high-concurrency, high-performance, and high-elasticity scenario?

Some werebe hesitant, but miHoYo chose Alibaba Cloud as always. This trust stems from years of cooperation. Over the years, miHoYo has witnessed Alibaba Cloud's support for Taobao and Tmall during Double 11. Each year, the computing peak value supported by Alibaba Cloud is higher than the previous year.

"Although miHoYo is not an e-commerce company, technologies and games have very similar requirements for high concurrency. From the perspective of practices during Double 11, we believe that Alibaba Cloud's technical capabilities are adequate."

miHoYo also saw how Alibaba Cloud was making rapid progress. In 2017, Alibaba Cloud launched its proprietary database PolarDB. It provides services, including minute-level configuration upgrades and downgrades, second-level fault recovery, global data consistency, and data backup for disaster recovery.

For miHoYo, it took several hours to back up the data, and the game had to stop running for more than an hour when using the old database. However, with PolarDB, the backup duration can be shortened to seconds. This significantly reduces the time required for data backup and improves the efficiency of version updates.

In July 2020, Genshin Impact started the final closed beta on PS4, iOS, Android, and PC simultaneously.

Welcome to Teyvat

Alibaba Cloud engineers came to the familiar meeting room again. In the last two months of closed beta, they became residents once again. They said, "Let's take Genshin Impact as a project of Alibaba Cloud." The accumulated capabilities of Alibaba Cloud over the years have become the technical basis for realizing the grand vision of Genshin Impact.

For example, the resource assurance capability is used, which engineers call "One Architecture, Global Deployment." On different service nodes worldwide, Alibaba Cloud provides a set of architecture products from the foundation layer to the database. These products include cloud servers, network resources, storage, security, and databases. All of them adopt the same architecture and provide the same performance experience.

This provides a basic guarantee for Genshin Impact to open servers worldwide simultaneously. miHoYo only needs to build a single architecture globally. They do not need to change the architecture according to the local conditions of every service area.

The same architecture performance ensures that players in all five regions worldwide have a unified game experience. Players do not need to worry about the function inconsistency in different regions.

PolarDB reduces the investment of miHoYo in IT O&M with its perfect product packaging form while meeting the requirements of Genshin Impact's high-performance gaming database. Thus, miHoYo personnel can focus on the game. "We focus on game design, and all basic IT works are left to Alibaba Cloud," Liu Xiao concluded.

This is Alibaba Cloud's biggest comprehensive practice in the game field. The requirements of Genshin Impact for Alibaba Cloud technologies have been met before in some other fields. However, this is the first time Alibaba Cloud has challenged all of the difficult tasks at once.

Judging from the global deployment effect and reservation amount, Genshin Impact would become the world's largest game on the cloud after its launch. A few months later, the prediction became a fact.

Therefore, even the "hot migration" frequently used by Alibaba Cloud needs to be upgraded further in Genshin Impact.

Hot migration is used to completely save the running state of the entire virtual machine and restore the state to its original (or other) hardware platforms. Users will not be aware of any differences during usage.

It is similar to driving on the highway. When the engine alarms, it needs to be replaced without stopping the car's functionality.

If changing the engine of a car is a normal state, it would be comparable to changing the engine of a large airplane during a flight for Genshin Impact. Due to the high stability requirements of the game, even the slightest jitter or offset of the airplane during replacement is not allowed.

Requirements are sent from the meeting room in miHoYo to the Apsara Park at Alibaba Cloud. Engineers from different departments, such as the gaming business and virtualization, gather to figure out solutions. They developed an intelligent O&M system to adjust the underlying performance. This can achieve lossless "AI hot migration" under ultra-high QPS, high CPU usage, and memory loads.

"AI hot migration" is applied to Genshin Impact with a very obvious effect.

In the last two months, the reservation amount of Genshin Impact and the server requirements have increased. Finally, after dozens of times of server scale-out and stress testing, Alibaba Cloud engineers felt that they finally made it!

On September 28, the Alibaba Cloud Team was in the Teyvat Meeting Room.

On September 28, the globally synchronous public beta of Genshin Impact started at 10:00 am.

In that meeting room, more than 20 Alibaba Cloud engineers stared at the indexes on the dashboard with one peak after another.

In the end, the PCU peak reached a certain value, which was higher than everyone expected, but the cloud steadily withstood the pressure. The public beta was smooth and steady, just like countless previous stress tests that had been carried out before.

On the door of this meeting room, there was a name, Teyvat.

On the Teyvat continent, those selected by God will be awarded the "Eye of God." They are called Genshin.

Summary

miHoYo is a first-generation cloud-native enterprise and is a perfect example of enterprises that implement cloud-native technologies. The way miHoYo flourished on the cloud can be a reference for enterprises that want to implement cloud-native. The cloud-native era has quietly arrived. More enterprises have seen its shining points, but implementing cloud-native and generating business value have become common problems that enterprises need to solve.

Community

miHoYo, a First-Generation Cloud-Native Enterprise, Realizes Imagination

Start With belief

Safely Expanded the Markets Outside of China With Support From Alibaba Cloud

Genshin Impact, an Unprecedented Challenge

Welcome to Teyvat

Summary

Read previous post:

Read next post:

ApsaraDB

You may also like

Comments

ApsaraDB

Related Products

PolarDB for PostgreSQL

PolarDB for Xscale

Gaming Solution

PolarDB for MySQL