PolarDB:History of PolarDB-X

Introduction

In 2009, Alibaba Group found in its revenue and expenditure accounts that its demand for data storage resources exponentially increased as business grew. To balance costs and business growth, technological innovation was required.

In this context, Alibaba launched the de-IOE campaign. This campaign aimed to construct an IT architecture that was more open, flexible, and efficient to ensure the business development and innovation of enterprise users in a time of rapid development in the Internet and cloud computing fields. The de-IOE campaign accelerated the replacement of commercial software with open source software and can be seen as a fundamental step in the development of cloud computing in China.

In the replacement process, databases were the core of the de-IOE campaign.

TDDL

Keywords: large-scale application in Alibaba, pioneer of database and table sharding technology

Do we have better products or solutions to replace traditional databases? The answer to this question is yes. We can develop a database product by using open source technology and a distributed architecture. This helps reduce costs and resolve issues related to performance and capacity.

On November 11, 2009, Alibaba launched Taobao Distributed Data Layer (TDDL), a distributed database middleware that can be used in transaction systems with high concurrency.

Back then, TDDL was only a JAR file that needed to be installed on clients. However, a topology that features the splitting of the Matrix, Group, and Atom layers was already used. This topology helped meet different splitting requirements and resolve scaling and availability issues in enterprise applications.

By 2015, TDDL had become the unified access standard of Alibaba database systems and started to provide distributed database services to all business in Alibaba. At present, approximately 300,000 TDDL instances are running for business in Alibaba, covering fields such as payment, funds, instant messaging, and media.

Put to the test by the Double 11 Shopping Festival and developed for diverse business models and huge business scale, TDDL features high stability, performance, throughput, and compatibility with MySQL syntax.

While developing TDDL, Alibaba also started to put distributed databases into commercial use.

DRDS

Keywords: cloud commercialization, high-performance SQL engine

In early 2016, Distributed Relational Database Service (DRDS) welcomed its first paying user. Since then, DRDS has been optimized to improve its unit processing capability and to help users reduce costs and improve efficiency.

In 2017, the DRDS development team released a high-performance distributed SQL engine. Using the PlanCache component, the FastSQL component, and custom underlying drivers, the SQL engine improved the performance of operations such as batch writing, shard key selecting, and read/write splitting by 300%, and the performance of operations such as cross-database integration, distributed joins, and distributed transactions by 200%.

DRDS supports join operations on tables of different dimensions and secondary sorting in memory. DRDS also allows you to perform function operations on memory results. This gives DRDS lower usage costs compared with other database services.

DRDS provides various enterprise-grade features for scenarios in which distributed databases are used. These features include DRDS Sequence, read/write splitting, database accounts, and backend O&M instruction sets.

Given its excellent performance and good user experience, DRDS has gained popularity among public cloud users.

The successful commercial use of DRDS marks the transformation of the distributed database technology from internal incubation to market-oriented operations as well as the transition from distributed database middleware to distributed database systems.

PolarDB-X 1.0

Keywords: upgrade of the architecture and brand, crucial projects for social and economic development

With optimal stability, ultra-high performance, and a wide range of enterprise-level features, DRDS was used in multiple crucial projects for social and economic development in China from 2018 to 2019 and helped accelerate the digital transformation of Chinese IT infrastructure. Since then, DRDS has become a key service in Alibaba.

Later, DRDS was upgraded and renamed PolarDB-X. In this name, PolarDB is the name of a portfolio of cloud database services developed by Alibaba Cloud and X comes from the word extreme.

PolarDB-X 1.0 allows you to use PolarDB for MySQL clusters as storage nodes. This enhances I/O capability and improves flexible transaction performance. PolarDB-X also provides enhanced security features for public service sector and enterprise users, such as consistent backup and restoration, SQL flashback, and SQL audit.

PolarDB-X 2.0

Keywords: finance-grade high availability, transparent distribution, integration of centralized and distributed architectures, HTAP, open source and multi-cloud deployment

PolarDB-X 2.0 is the Alibaba distributed database release with the most updates. PolarDB-X 2.0 provides automatic sharding policies based on the concept of transparent distribution and ensures transactional consistency based on the Timestamp Oracle (TSO) policy and the multiversion concurrency control (MVCC) method. PolarDB-X 2.0 supports linear scaling of distributed databases and uses consistent hashing for partitioning. PolarDB-X 2.0 also provides the global binary logging feature and the global consistent backup capability. The data nodes (DNs) of PolarDB-X 2.0 use a MySQL branch developed by Alibaba based on X-Paxos. The MySQL branch provides three replicas and strong consistency to ensure a zero recovery point objective (RPO) during disaster recovery.

PolarDB-X 2.0 uses a brand-new architecture and can be deployed in 13 countries and regions around the world. To meet different requirements, PolarDB-X 2.0 provides diverse deployment forms, including public cloud, private cloud, software, and DBStack.

The source code of PolarDB-X 2.0 is available in open source MySQL communities. This helps improve the quality of high-performance distributed databases and simplify the use of the databases.

In the future, PolarDB-X 2.0 will focus on areas such as localization, hybrid transaction/analytical processing (HTAP) loads, quick capacity change, and multi-active disaster recovery.