This topic describes multi-master sharded tables.
Background information
As business rapidly grows, a single database may struggle to handle the increased amount of data and access requests, and the system may become slow and inefficient. To address the challenges, PolarDB for MySQL provides the multi-master sharded table solution. The multi-master sharded table solution is an innovative cloud-native database service that features automatic horizontal sharding, hybrid shared storage, and shared-nothing architecture. The solution can automatically distribute data to multiple compute nodes based on the defined sharding rules. Each node can handle read and write operations. This solution provides enhanced distributed processing capabilities. The solution not only linearly improves the overall concurrent throughput of databases by using horizontal sharding, but also allows for on-demand resource allocation and high resource utilization based on the shared storage architecture.
A multi-master sharded table uses an architecture that decouples storage and computing.
Storage layer: pools and consolidates storage resources to enable shared access to storage and flexible, on-demand storage allocation.
Compute layer: allows simultaneous read and write operations on multiple nodes without resource contention, which enables horizontal scaling of read and write performance.
Supported engine versions
Only PolarDB for MySQL 8.0 supports the multi-master sharded table solution.
Benefits
Automatic horizontal sharding
When you create a multi-master sharded table, you can specify a shard key. The system automatically distributes data to different physical shards based on the shard key. You do not need to manually shard data.
Business transparency
You can use a multi-master sharded table as if it was a standalone database. You do not need to be concerned about the physical distribution of data.
Second-level horizontal scale-out
The solution supports simultaneous write operations on up to 63 compute nodes, which greatly enhances the parallel processing capability and expands the auto-scaling range of the system. The scaling can be completed in the load balanced manner within seconds without the need to migrate data.
Unified distributed DDL framework
The solution uses a unified distributed DDL framework, which provides an integrated approach to managing database schema changes across a distributed system. The framework ensures that all related DDL operations, such as adding or removing indexes and modifying column definitions, are seamlessly and transparently executed within a unified process, which significantly simplifies maintenance tasks.
Automatic aggregation
A global read-only node aggregates all data in the shards. You can access data from the global read-only data without the need for additional storage or synchronization links.
Scenarios
The multi-master sharded table solution (in a multi-master cluster) is designed for high-concurrency read and write scenarios, such as e-commerce and gaming scenarios.
E-commerce transaction scenarios
Challenges: E-commerce transaction systems must maintain long-term stability and low latency under high workloads. At the same time, E-commerce transaction systems face various challenges such as extremely large tables with tens of billions of rows, high-concurrency write bottlenecks, risks of hot spots during flash sales, and single-node storage limits.
Solution: The multi-master sharded table solution provides transparent sharding capability and can scale out physical shards based on the shard key. This way, the workloads are distributed across multiple nodes, which helps alleviate the processing pressure on a single node. This solution ensures that the transactions are executed as if they were executed on a single node and significantly reduces the O&M complexity.
Table and database sharding scenarios
Challenges: In large-scale business scenarios, table and database sharding middleware is used to handle the challenges of substantial data quantities, intensive concurrent requests, and stringent requirements for system stability. However, traditional distributed middleware lacks efficient support for distributed DDL operations and flexible scaling capabilities.
Solution: By using a unified distributed DDL framework, the multi-master sharded table solution ensures that all related DDL operations, such as adding or removing indexes and modifying column definitions, are seamlessly and transparently executed within a unified process. The solution can intelligently adjust resource allocation based on business workloads and allow for horizontal scaling without the need for data migration. The solution provides a unified and intuitive management interface that enables centralized monitoring and management of all database nodes. This helps O&M personnel quickly identify issues and perform daily O&M tasks and ensures system stability and reliability.
Performance comparison
A performance comparison test was performed by using sysbench on mainstream commercial distributed databases on the market represented by distributed database 1, distributed database 2, and a PolarDB multi-master sharded table:
Test setup: Purchase distributed database 1, distributed database 2, and a PolarDB multi-master partition table on the PolarDB cluster buy page and make sure that the costs of the purchased clusters are the same.
Test method: Create an Elastic Compute Service (ECS) instance that has 32 cores and 128 GB of memory. Install sysbench on the ECS instance and run sysbench with 128 concurrent request threads to test the performance of the distributed databases. Use the following workload patterns, including oltp_point_select, oltp_read_only, oltp_write_only, and oltp_read_write to evaluate the performance of each distributed database.
Test result: The following figure compares the performance of the distributed databases. The performance metric used for comparison is queries per second (QPS).
The figure shows that the PolarDB multi-master sharded table significantly outperforms the mainstream distributed databases 1 and 2 on the market.
