By Jin Heng
In the era of big data, with the rapid growth of data, conventional computational software can no longer serve the needs of enterprises. This has boosted the growth of big data; in Silicon Valley, giants such as Google, AWS, and Microsoft have all invested heavily on the research and development of big data technology. Alibaba Cloud, led by Dr. Wang Jian, has also been actively involved in big data research, with its renowned product - MaxCompute. Alibaba Cloud MaxCompute was developed ten years ago, being one of the first independently developed big data computing platforms in China.
After ten years of accumulation, MaxCompute now supports storing exabytes of data and computing hundreds of petabytes of data daily. MaxCompute has been providing services to enterprises in various sectors in more than ten regions inside and outside of China. These sectors include e-commerce, industry, healthcare, agriculture, meteorology, and education. Now, MaxCompute is able to easily process large amounts of data to serve the society and consumers.
In fact, the big data development history of Alibaba Cloud clearly reflects the history of the entire big data industry.
Ten years ago, Alibaba embraced the challenge of large scale development earlier than other Internet enterprises. Back then, Oracle databases were used by most enterprises around the world, and Alibaba ran the largest Oracle cluster in Asia to compute petabytes of data.
With the rapid growth of Taobao users, Oracle clusters would soon be incapable of supporting the business development of Alibaba. The most fundamental problem is the insufficient computing power. Then, Alibaba migrated data to Greenplum with greater computing power, and soon encountered bottlenecks when running a cluster with less than one hundred servers. This caused huge obstacle for business development of Alibaba.
Open-source technologies such as Hadoop also had unavoidable reliability and security drawbacks.
In 2008, Dr. Wang Jian joined Alibaba Cloud to solve the bottleneck of big data computing power. Mr. Wang found that none of the existing solutions, such as Oracle, Greenplum, and Hadoop, are the optimal solution to big data computation. We must independently develop a big data processing platform of our own.
The long match of Alibaba Cloud for big data computation solution started on 2009. Dr. Wang Jian and his team aimed at independently developing a big data computing platform MaxCompute to integrate the internal data and big data computing systems of Alibaba Group.
Time proved that Alibaba made the right choice.
After four years of hard work, MaxCompute made a historical breakthrough on August 15, 2013, by allowing up to 5,000 servers to run simultaneously within the same cluster. Today, after ten years of development, a single MaxCompute cluster can scale up to 10,000 servers. Only a few technology companies can do this.
An Apsara 5K monument stands still in the Alibaba Cloud Yunqi town, bearing the names of technical engineers who have contributed in solving this technological problem.
A big data computing platform allows orders to accurately gather in real time, meteorologists to precisely predict changing weather conditions, and enterprises of all industries to provide better services. The key to implement this capability is the analysis result of large amounts of data.
However, to properly deal with such amount of data is never easy. In addition to the soaring data volume increase, the diversified data types of different industries, such as structured data and unstructured data, have posed new challenges on the big data platform.
One of the innovations of MaxCompute is the adoption of the Data Lake technology. This allows MaxCompute to store different data sources in similar means, compute data in a unified manner, and provide a standard language to quickly implement computation of different data types.
On the basis of this innovative technology, Alibaba Cloud set six world records in 2015 and 2016 Sort Benchmark contests, the "Olympics" of the computing world. In 2017, MaxCompute completed the public-cloud-based 100 TB BigBench test for the first time in the world.
High speed query is also one of the core problems of big data computing. MaxCompute uses interactive queries to speed up querying large amounts of data. Simply put, the system can predict the queries that you are going to perform and get prepared in advance. This significantly reduces the time required for large scale data queries.
In addition, MaxCompute offers a multi-tenant security isolation technology, breaking down the security limits of conventional big data platforms. MaxCompute refines the security boundaries to the user-level, process-level, and kernel level, fully satisfying financial-grade security requirements.
In the last ten years, the daily computing power of MaxCompute has been increasing from 100 PB in 2015, to 180 PB in 2016, 320 PB in 2017, and 600 PB in 2018.
MaxCompute continues to set new records and win recognition from authoritative institutions. According to Forrester WaveTM: CloudData Warehouse, Q4 2018, three Alibaba Cloud products - MaxCompute, DataWorks, and ADB have been included in the report. Alibaba Cloud outperforms Microsoft in terms of the current offering feature.
The world-class big data computing power of MaxCompute gradually displays its value. MaxCompute enables tens of thousands of enterprises to compute large amounts of data at lower cost and higher efficiency, and to provide better services to the society and consumers.
In terms of smart life, MaxCompute helps Moji Weather provide weather forecast information for 400 million users, and there are more than 500 million daily user queries. In addition, MaxCompute also reduces the Moji Weather data storage and computing costs by an incredible 70%.
In terms of smart traffic, City Brain manages 1,300 traffic light intersections and more than 200 Traffic Police officers in Hangzhou. From 2016 to 2018, the rank of Hangzhou dropped by 52 from the list of the most congested cities in China.
In terms of smart manufacturing, the big data processing technology of Alibaba Cloud helps manufacturing companies find out the best combinations of thousands of parameters to improve the conformity rate. Many industry leaders such as Golden Concord Holdings Limited and Trina Solar Limited are trying out this new production mode.
In terms of smart government affairs, the "One Stop, One Trip, One Paper" program streamlines government affairs data by using the MaxCompute big data processing platform. MaxCompute helps government departments of Zhejiang Province centrally manage and share over seven billion data records covering more than 100 categories that are closely related to the daily lives of citizens. This means that citizens should be able to handle their affairs with the government by paying at most one visit to the relevant government department.
MaxCompute has also started empowering more users in markets outside of China, such as Singapore and Europe.
MaxCompute has significantly reduced the computing costs of various industries and sectors inside and outside of China by using its outstanding computing power. Today, more and more companies, online and offline, manufacturers and e-commerce companies, have been benefiting from the MaxCompute big data processing service.
Visit the MaxCompute product page to learn more: https://www.alibabacloud.com/product/maxcompute
137 posts | 19 followers
FollowAlibaba Clouder - November 27, 2018
Alibaba Clouder - November 8, 2018
ApsaraDB - December 22, 2022
Alibaba Clouder - March 15, 2019
Alibaba Clouder - June 7, 2018
Alibaba Clouder - August 28, 2019
137 posts | 19 followers
Follow
Database Security Solutions
Protect, backup, and restore your data assets on the cloud with Alibaba Cloud database services.
Learn More
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More
MaxCompute
Conduct large-scale data warehousing with MaxCompute
Learn More
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn MoreMore Posts by Alibaba Cloud MaxCompute
