What Can We Learn from Twitter's Move to Kubernetes

This blog outlines how Twitter's switch from Mesos to Kubernetes once again proves that Kubernetes is the industry-wide standard when it comes to all things containers.

By Zhang Lei, Staff Engineer on Alibaba Cloud Container Platform and Co-maintainer of Kubernetes Project

It's not a secret that Twitter has decided to move Twitter's infrastructure from Mesos to Kubernetes. But behind this major decision, what kind of reason and motivation would be supporting this major change to Twitter's infrastructure?

New Challenges Infrastructure in the Cloud Age

In fact, in Internet technology scenarios, infrastructures built by engineers on the basis of mature technologies will always be the first choice of leading non-cloud Internet vendors. Therefore, the relatively mature and efficient Mesos project has been widely deployed in production at scale in many cases.

However, with the popularity of cloud computing and the rise of cloud-based containerized infrastructure projects like Kubernetes, this traditional Internet infrastructure starts to show its age—being a much less efficient solution compared with that of Kubernetes:

Diversified and specialized storage systems have drastically increased the complexity of traditional infrastructures. Let's get to the nitty gritty related to this problem. Compared with the one-single-storage-system assumption in the traditional technical architecture, the software architecture in the cloud age allows users to take advantage multiple storage options. Take Alibaba Cloud for example, it provides users with more than 10 storage service systems on the public cloud, not to mention the large number of more specific solutions. With the continuous development and expansion of the infrastructure and software scale in Internet companies, Internet-based software has more refined and specialized storage requirements. For another example, at Twitter, Local Persistent Volume, an atypical storage choice, has gradually become a mainstream solution due to its increased level of the balance between performance and cost. As the author of CSI (Container Storage Interface), the Kubernetes community has the full-fledged Local PV mechanism. In addition, Kubernetes can help users solve the problems with connecting to dozens of other different storage services by using its standard interfaces as well as its PVs and PVCs. This is obviously crucial as the Internet software architecture tends to be more complex and target multiple clouds.
In house orchestration is not compliant with the Cloud-Native trend. One of the important technology development trends in the age of cloud is the trend that software lifecycles will gradually move closer to the born-and-growing-in-the-cloud model. This means that core infrastructure projects that support software inevitably need to evolve towards the principle "making the best of the cloud." Unfortunately, in many internet companies' orchestration system, cloud has never been made a first-class citizen of the entire infrastructure, perhaps because they were released a long time ago, it's also the same case for Mesos Aurora. In contrast, Kubernetes has been advocating concepts such as declarative APIs, container design mode, and controller pattern since its first release. These concepts are significant efforts in the Kubernetes community to help users implement software development, delivery, and maintenance on the cloud in a scalable, reproducible, and automated manner. Today, these top-level architecture design concepts and best practices are collectively referred to as Cloud-Native in the developer community. This is the biggest difference between Kubernetes and other competing projects.
The cost of traditional multi-cloud and multi-cluster management remains high and is expected to grow rapidly in the foreseeable future. In traditional Internet architecture, self-built data centers and infrastructures are the first assumption regarding the entire software system. Cloud acts more like a contingency resource in case of traffic peaks during traffic bursts. With cloud serving an auxiliary role, multiple clouds and multiple clusters are unlikely to become the top priorities of the entire architecture. This makes the capabilities of multiple clouds and clusters a responsibility of the underlying resource sharing layer and causes these capabilities not to be directly associated with the more important system of application development, delivery, and maintenance. This design solution may seem to work for a short period of time. However, from a long-term perspective, the maintenance and iteration cost is often out of control due to the different forms of upper-layer applications themselves and frequent iterations. In addition, another disadvantage of this design is that it pushes the infrastructure to the multi-active technology abyss, which is actually far beyond the technical capabilities of over 90% of the Internet companies. With the popularity of cloud-native architectures, countless clusters on each cloud gradually becomes a new norm in application infrastructures. This brought about a new breakthrough idea for multi-cloud and multi-cluster management: Multiple clouds and multiple clusters will naturally become capabilities included in the application infrastructure if the software is constructed, designed and implemented based on Kubernetes. As more and more services a organization need to be delivered in a multi-cloud and multi-cluster environment, the organization tends to changes its architecture to Kubernetes as it fundamentally provides a "shortcut" to quickly deliver applications on multiple clouds. This fact, also applies to Twitter's situation.

Cloud Native Technology Embraced by World-Class Internet Companies

It is not hard to see that the main technical factor behind the Twitter's switch to Kubernetes Native is to implement a balance between large-scale and customized needs and the community standard by utilizing the core concepts of Kubernetes and its scalability.

This approach is also advocated by Alibaba in the community. In 2018, Alibaba joined a large number of leading Internet companies like Google, Facebook, LinkedIn, Uber, Netflix, and Pinterest to launch the monthly Web-Scale Meetup in Silicon Valley, where Alibaba also shares several events and meetups, too. Twitter also invited engineers from the Alibaba Cloud Container Platform team as featured guests at the small annoucement conference (or meetup), including Li Xiang, Zhang Lei, and He Jian. Jago Macleod, technical engineering manager of the Kubernetes team at Google, was also invited to attend the conference.

At the conference, the Alibaba Cloud Container Platform team revealed that in the next month, Alibaba Cloud would publish the source code of Kruise, a Kubernetes Workloads Advanced project that Alibaba Cloud has worked on for a long time. Kruise makes full use of declarative APIs and controller pattern in Kubernetes to provide the important in-place update feature for containerized applications in Internet scenarios. It also allows users to have more fine-grained business release policies. Many companies like Twitter, Pinterest, and Netflix will collaboratively participate in this innovative cloud-native job management project.

The continuously increasing scale and performance of Kubernetes are another technical reason for the change of Twitter from an on-looker to an adopter. In his speech, Jago Macleod mentioned the ongoing research and cooperation between Google and Alibaba in this field. Recently, engineers from both companies were attempting to add bookmarks for the large number of WATCH operations in k8s. This allows creators of these WATCH operations to only trace a small number of historical changes outside the bookmarks after the restart. In some cases, the Kubernetes API server can even show 40 times better performance.

Kubernetes: An Application-Centered Expressway

As the Internet business grows rapidly, the infrastructure software team at the company soon began to surpass its expected scale boundary, but the return on investment ratio did not increase accordingly.

The past in-house infrastructure software development and architecture design are undergoing subtle changes as the cloud computing and Cloud Native become increasingly popular. With the Kubernetes platform project standard, many Internet companies have already had the ability to migrate their infrastructures to the cloud at a relatively low cost. More importantly, with the Kubernetes standard layer, the integration itself does not have to be implemented on a solid binding relationship between Internet companies and cloud service providers, as it is for Netflix and AWS. Instead, Kubernetes retains most advantages of in-house development and also enables the multi-cluster management and the full utilization of the cloud power. This advantage will become more obvious after all AWS engineers in an Internet company become k8s engineers.

It is easy to see that Kubernetes is application-centered and connects cloud, application developers, and infrastructure software teams together. The communication, connection, and delivery capabilities on this "expressway" project serve as meaningful references for fast-changing Internet companies like Twitter deciding the future evolution of their infrastructure and architecture. The change allows a commercial organization like Twitter with rapid business growth to maintain an infrastructure team made up of just a few dozen members.

Conclusion

Twitter's switch from its initial adoption of Mesos to the use of Kubernetes Native today proves again the assertion that Kubernetes has become an industry standard for container orchestration. More importantly, Twitter's embrace of Cloud Native is expected to provide a classic learning model for the large-scale implementation of the cloud-native technology in production.

Alibaba has invested lots of technical resources in the Cloud Native ecosystem and is being recognized as an important leader in the large-scale implementation of the cloud-native technology in production by other global leading Internet companies including Facebook, LinkedIn, Uber, Netflix, and Pinterest besides just Twitter. As cloud computing becomes more popular, traditional Internet technology architectures begin to show many issues with their age. Specifically now, with the rise of cloud-based container infrastructure projects like Kubernetes, more and more international enterprises start to think about how to embrace the open-source ecosystem and open technology standards by using the power of cloud and cloud-native technologies and how to be prepared for a cloud-based digital future featuring strong iteration capability.

Community

What Can We Learn from Twitter's Move to Kubernetes

New Challenges Infrastructure in the Cloud Age

Cloud Native Technology Embraced by World-Class Internet Companies

Kubernetes: An Application-Centered Expressway

Conclusion

Read previous post:

Read next post:

Alibaba Container Service

You may also like

Comments

Alibaba Container Service

Related Products

ACK One

Container Service for Kubernetes

Cloud-Native Applications Management Solution

CloudBox