The Open-Source Folks Talk - Episode 4: Remain True to Original Aspirations in the Cloud-Native Age

Recently, Alibaba Cloud donated Celeborn Project (formerly EMR Remote Shuffle Service Project) to Apache incubator. It is the first Apache incubation project born on Alibaba Cloud. Alibaba Cloud's open-source big data platform, E-MapReduce EMR, supports running mainstream open-source big data components (such as Hadoop, Spark, Flink, and Kafka) on Alibaba Cloud. It provides a low-threshold and easy-to-use open-source big data service on the cloud.

Cloud-native architectures and concepts are being continuously strengthened and implemented. For example, the storage and computing separation architecture is a unique architecture attribute of the cloud. Under this technical background, we find that data shuffle is required in Hive, Spark, Flink, etc. Therefore, Alibaba Cloud provides Remote Shuffle Service to support all big data computing engines with one set of data Shuffle.

This is the change brought about by the cloud in combination with open-source. After the release of the Remote Shuffle Service project, it attracted a number of companies to participate in the joint construction. It was finally open-sourced in December 2021. We donated it to the Apache Foundation to drive more companies to participate in the joint construction and make the project more influential.

The cloud provides elastic resources that allow users to run their businesses and exert their computing power with great freedom. They can run digital businesses, data analysis, databases, AI, SaaS, PaaS, and other businesses on the cloud. However, running open-source software requires many environments, hardware, and parameter configuration, which is a certain threshold.

The emergence of the cloud has driven the rise of open-source since resources are easily obtained on the cloud. Cloud elasticity meets the demand for on-demand usage and purchases, so it is easy to deploy open-source software. The cloud has become the operating base of open-source software. For example, you can use EMR to create 100 node clusters in 3 minutes. You can use mainstream open-source components (such as Hadoop, Spark, Flink, and Kafka) to easily implement a complete process (such as O&M, deployment, control, monitoring, use, and development).

The cloud has made the operation of open-source software universal. The relationship between the cloud and open-source has promoted the development of the digital economy and digital transformation. In addition, the cloud can provide a PAI DSW-consistent Notebook development experience. The cloud provides a good business model for open-source, benefiting more people through open-source. At the same time, users can build business models through open-source and get more resources to give feedback to the open-source community, which is forming a positive cycle. In a word, running open-source software on the cloud and supporting the commercialization of open-source software can make the cloud and open-source better combined..

Alibaba has over 3,000 open-source projects and more than 30,000 external contributors. Alibaba cooperate with over 100 open-source communities to jointly promote the development of open-source projects and culture in the industry.

(Open-source projects born and developed on the cloud)

The full stack of the Ali AI open-source family adopts the open-source style and has achieved many landings in different industries (such as audio and video, autonomous driving, search and recommendation, OCR text recognition, financial quantification, and intelligent transportation scheduling for smart transportation).

The core engineering platform, Machine Learning Platform for AI (PAI) also embraces open-source technologies and is fully compatible with mainstream international industry standards (such as TensorFlow and PyTorch), as well as mainstream domestic AI frameworks (such as PaddlePaddle, MindSpore, and OneFlow). We have also implemented many optimizations, including Deep series and Blade series compilation, AI algorithm, algorithm framework, AI engine, and AI compilation. Many optimizations have been fully open-sourced. Alibaba AI processes run on the open-source technology stack with open-source culture from the underlying platform to the upper-layer applications.

Flink has become the standard for real-time computing. Various domestic Internet companies, financial companies, and traditional industry companies have used Flink for real-time stream computing and analysis, and many overseas companies have used Apache Flink for real-time analysis.

In 2015, Taobao faced huge data challenges. Users constantly put forward new requirements for shopping experiences. For example, commodities/prices and personalized ranking models/recommendation models need to be updated in real-time. Specifically, on the day of Double 11, the operation strategy changed fast, and the operation strategy needed to be adjusted in real-time according to user feedback and market changes. Therefore, Alibaba needs its real-time big data engine. After comprehensive consideration, we decided to embrace the open-source community and use the open-source concept to build the next-generation real-time big data computing platform. In the end, we chose Apache Flink as the core technology cornerstone for the next few years.

In 2017, we continued to firmly embrace open-source, unifying the original Alibaba internal JStorm, Blink, and Galaxy into open-source projects and making the future real-time computing core engine based on Flink. Since then, all of Alibaba's real-time big data analysis and streaming big data analysis have been built around Flink. In 2017, real-time computing across the group was gradually unified. Based on the Flink community, we increased investment and transferred our fully accumulated capabilities to the cloud to provide Flink-based big data services on the cloud in 2018.

We started to invest in community construction in 2016. In 2017 and 2018, we participated in Flink community conferences in Europe and the United States to promote the continuous growth of the Flink community in China. In 2019, Alibaba acquired the business companies behind Flink to fully support community development. After two years of continuous investment from 2020 to 2021, Alibaba has held nearly 100 online and offline Flink MeetUp and Flink Forward conferences in China, involving a lot of workforce, material resources, and energy. Alibaba has always firmly and completely promoted Flink regardless of commercialization or return, making it the top project in the Apache community. More importantly, the achievements of the past few years are inseparable from the promotion of open-source enthusiasts and culture practitioners.

Currently, there are more than 1,500 developers in the Flink community worldwide, which has doubled in three years. The number of GitHub Stars is nearly 20,000, which has tripled in three years and is at the top of the community.

Thanks to Alibaba's continued investment, Flink's key metrics have reached the top 1 or 2 positions in the community. The firm investment in open-source has verified the value of the Chinese team in international projects.

After continuous evolution and technological innovation, the Flink community started from only Java API to SQL API, which makes development more concise with a lower threshold for more users. After that, CDC was launched for better Data Integration, Flink ML was launched to cooperate better with PAI, the table store was launched to store batch data, and the next-generation streaming data warehouse architecture is being built.

Alibaba has unreservedly opened its technological innovations to the community, benefiting more developers, and hopes to attract more developers to promote the development of the community.

The Ali AI system is the open-source representative of AI, Flink is the open-source representative of big data, and the SREWorks project is the precipitation of big data and AI operation and maintenance capabilities accumulated over 10+ years. It is also open-source. The SREWorks can provide comprehensive-operation O&M service suite management, and the capabilities of delivery, O&M, management, monitoring, operation, and data assets can be implemented through SREWorks open-source projects.

Behind many open-source projects are a group of employees and developers with a lot of passion and enthusiasm for open-source. We have regular open-source technology sharing, such as communication among Machine Learning teams, big data teams, and European and American teams, community meetups, and communication with different companies. This is the guarantee that open-source projects can be implemented perfectly.

Our team has participated in contributing over ten Apache top-level open-source projects, cultivated more than 50 top-level committers and PMCs of open-source projects, and contributed 1.5 million + lines of open-source code. Team partners contribute their experience in work and technology to the open-source community and provide them to more companies and enterprises for use. I hope more people will join us to build the open-source community.

Community

The Open-Source Folks Talk - Episode 4: Remain True to Original Aspirations in the Cloud-Native Age

Read previous post:

Read next post:

Alibaba Cloud Community

You may also like

Comments

Alibaba Cloud Community

Related Products

Big Data Consulting for Data Technology Solution

Big Data Consulting Services for Retail Solution

Realtime Compute for Apache Flink

Managed Service for Prometheus