An In-Depth Interview with ACK Distro – Part 2: How to Build a Hybrid Cloud Unified Network Plane with Hybridnet

By Ruohe, Yusheng, and Yujia

Reporter: Hello, Alibaba Cloud-Native readers. Welcome to Part 2 of this series of exclusive interviews. Today, we have our old friend, the Distribution of Alibaba Cloud Container Service for Kubernetes (ACK Distro). In Part 1, we introduced Sealer, the open-source cluster image technology of Alibaba, and how they cooperate with each other to achieve fast and stable delivery of ACK. If you missed that interview, do not forget to go back and review it.

ACK Distro: Hello again, everyone! In this interview, I will explain one of my good partners in detail, Hybridnet, an open-source Kubernetes container network solution of Alibaba, and I will explain how it can build a hybrid cloud unified network plane.

Reporter: OK, please talk about what Hybridnet is and which design concept inspired the project team members to create Hybridnet.

Definition of Hybridnet and Its Design Concept

ACK Distro: First, Hybridnet is an open-source Kubernetes container network solution of Alibaba for hybrid cloud scenarios. It can help users build a unified underlay and overlay network plane on the heterogeneous environments of physical and virtual machines and provide enormous capabilities for management and O&M. At the same time, Hybridnet provides a new solution to deal with problems of container network deployment and O&M that occur during cluster deployment and application delivery in hybrid cloud scenarios.

Its basic design guidelines are:

Determine a unified network model, reduce costs of cognition and maintenance, and ensure stable long-term evolution
Shield the underlying heterogeneous infrastructure, improve the robustness of delivery, and reduce the costs of production delivery and PoC
Under the constraints of the unified model, it can provide underlay high-performance pass-through network solutions to meet the dual requirements of network access and performance and offer overlay virtual network solutions in certain performance-insensitive scenarios.
Minimize dependence on the external environment and ensure the simple data surface
It is deeply integrated with Kubernetes and provides high-level IPAM capabilities (such as dual-stack, IP retention, and IP fixation) to ensure users retain their usage habits after moving to the cloud.

It is different from the container network solutions (such as Terway and AWS-CNI) bound with cradles of public cloud or private cloud of a single IaaS vendor. The Project Team members hope Hybridnet can deal with problems of consistency and adaptability caused by heterogeneous cradles in multi-cloud and hybrid-cloud scenarios. In addition, it can provide agile, universal, and stable delivery capabilities in different basic network environments and solve problems of network planning, management, and O&M in complex scenarios through model constraints and O&M control from a unified perspective.

Reporter: Hybridnet strives to achieve underlay/overlay hybrid deployment and underlay/overlay unified management and O&M. Do I understand it correctly?

ACK Distro: Yes, and I can expand the description further. In a Kubernetes cluster that uses Hybridnet, there can be underlay and overlay pods on the same node. The cluster internal access behavior among all pods is completely consistent without any additional awareness. Therefore, users can freely realize selection and transformation among the pure overlay cluster, pure underlay cluster, and underlay hybrid cluster. At the same time, users can benefit from the high performance and network pass-through capability brought by underlay networks and the unlimited resources and high adaptability of overlay networks. Moreover, under the constraints of the unified model, underlay networks and overlay networks are consistent in the concepts of management and O&M.

Reporter: In addition to its design concept, can you use a more intuitive method to help readers understand what Hybridnet offers?

ACK Distro: Of course! Let me start by introducing its core model to explain its features and attributes. In order to enable users of Hybridnet to flexibly describe the basic network environment by initializing different core models and allow the container network to run in different forms, the Project Team members abstract the concepts in the classic network and introduce the following three core CRD models to abstract and manage network resources.

The Core Models of Hybridnet

Network

In Hybridnet, each network represents a scheduling domain, and a scheduling domain represents a group of nodes with the same network nature. The network is the main entry for the incoming environment topology information. A specific IP can be freely migrated among nodes in the scheduling domain to which it belongs.

Networks are associated with nodes through nodeSelector. nodeSelector may be empty for some special networks, such as overlay networks. Therefore, the scheduling domain of this kind of network consists of all nodes in the cluster.

Subnet

In Hybridnet, a subnet represents the IP resources that can be allocated within a scheduling domain. A subnet is the main entry for incoming network IP resource planning in the environment. Each subnet must belong to a network. A subnet has more flexible attributes:

It supports the selection of CIDR allocable address range and can accurately carve out a small network segment from a CIDR range through spec.range.start and spec.range.end.
It keeps discrete IP addresses not be allocated. When IP addresses are already used in the network segment, Hybridnet will no longer allocate these IP addresses to pods by filling the used IP addresses into the array field of spec.range.excludeIPs.
It keeps specified IP addresses not to be used. When the IP addresses of certain pods need to be retained in the network segment and are only used by specific pods, Hybridnet will no longer use these IP addresses for allocation of non-specified IP addresses by filling these IP addresses into the array field of spec.range.reservedPs.

IPInstance

IPInstance is only used for monitoring. Each IPInstance represents an IP address that has been allocated out of the container network. The information, such as its corresponding pod, subnet, and the corresponding node of the pod, can be viewed by the kubectl get IPInstance.

Reporter: How does Hybridnet exert its advantages? How do you manage Hybridnet to achieve best practices?

How to Manage Hybridnet in ACK Distro

ACK Distro: Let me display how to operate network management by operating the preceding CRD model.

Hybridnet will be deployed as the only built-in network plug-in. (It is also feasible to customize third-party network plug-ins with the help of Sealer's abilities. You can refer to Part 1 in this series.)

Default Behavior

As my fixed behavior, there must be an overlay network in initialization, and the default network type is overlay at this time. You can run the following command to view the information about the network and subnet at this time:

[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get network
NAME        NETID   SWITCHID
network-0   4       virtual-switch
[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get subnet
NAME                 VERSION   CIDR            START   END   GATEWAY      TOTAL   USED   AVAILABLE   NETID   NETWORK
subnet-0-network-0   4         100.64.0.0/16                 100.64.0.1   65533   2      65531               network-0

As you can see, if I use the default configuration in initialization, I will create a network named network-0 and a subnet named subnet-0-network-0. The CIDR of the container block is 100.64.0.0/16. At this time, the newly created pods will be pulled up in the way of the overlay.

Since my basic components have no special network demands, the overlay network helps me shield the underlying basic network facilities, which is the biggest advantage. In other words, I can use Hybridnet to directly start in any network environment with the same configuration without affecting the possibility of subsequent network expansion.

From the delivery experience of the hybrid cloud environment, this method can delay network planning (mainly underlay networks) to the O&M stage. Therefore, the method minimizes the implementation cost of the delivery stage and improves the deployment efficiency.

Adding Underlay Networks

If some demands for underlay networks exist (for example, due to the overlay performance bottleneck or the capability of Pod IP address for direct disclosure) and underlay pods account for a relatively small proportion, you can add underlay networks and the corresponding subnets in addition to overlay networks created by default. You can add underlay networks and the corresponding subnets, especially if you do not want to occupy IP resources in the basic network environment. (The original and new overlay or underlay networks do not have any dependency and sequenced relationship on the model.)

In the experimental environment of this example, the node network segment is 192.168.56.0/24. (All nodes are in a classic two-layer network.) Since the node IP addresses, 192.168.56.1， 192.168.56.2， 192.168.56.3， and 192.168.56.4, are used, we consider leaving the unused address range from 192.168.56.100 to 192.168.56.150 to the container to build the simplest underlay network. Therefore, we only need to apply the following YAML:

---
apiVersion: networking.alibaba.com/v1
kind: Network
metadata:
  name: underlay-network1
spec:
  netID: 0
  nodeSelector:
    network: network1
  type: Underlay

---
apiVersion: networking.alibaba.com/v1
kind: Subnet
metadata:
  name: underlay-subnet1
spec:
  network: underlay-network1
  netID: 0
  range:
    version: "4"
    cidr: "192.168.56.0/24"
    gateway: "192.168.56.254"
    start: "192.168.56.100"
    end: "192.168.56.150"

Since a network is associated with a node through a nodeSelector, we need to mark the node where you want to deploy an underlay pod with the corresponding Network nodeSelector label. Here, we only want to have a pod of the underlay type on the node izf8zdygpbo4hx57g2wah8z:

kubectl label node izf8zdygpbo4hx57g2wah8z network=network1

The default network type is still overlay. If you want to create an underlay pod, you need to specify it by adding the annotation of networking.alibaba.com/network-type: Underlay to the pod. The effect is shown in the figure below:

[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get po -owide -n test
NAME                                 READY   STATUS    RESTARTS   AGE     IP               NODE                      NOMINATED NODE   READINESS GATES
curl-deployment-1-5cfb5dcb8c-65fr7   1/1     Running   0          11m     100.64.0.29      izf8zdygpbo4hx57g2wahbz   <none>           <none>
curl-deployment-1-5cfb5dcb8c-hp626   1/1     Running   0          11m     100.64.0.26      izf8zdygpbo4hx57g2wahbz   <none>           <none>
curl-deployment-1-5cfb5dcb8c-qbr6w   1/1     Running   0          11m     100.64.0.27      izf8zdygpbo4hx57g2wah7z   <none>           <none>
curl-deployment-1-5cfb5dcb8c-zclv2   1/1     Running   0          11m     100.64.0.31      izf8zdygpbo4hx57g2wahbz   <none>           <none>
curl-deployment-1-5cfb5dcb8c-zfqkp   1/1     Running   0          11m     100.64.0.28      izf8zdygpbo4hx57g2wah7z   <none>           <none>
curl-ss-0                            1/1     Running   0          6m24s   192.168.56.140   izf8zdygpbo4hx57g2wah8z   <none>           <none>
curl-ss-1                            1/1     Running   0          6m5s    192.168.56.141   izf8zdygpbo4hx57g2wah8z   <none>           <none>
curl-ss-2                            1/1     Running   0          6m1s    192.168.56.142   izf8zdygpbo4hx57g2wah8z   <none>           <none>

Changing the Default Network Type as Underlay

If you want to create overlay pods by default when underlay network demands account for the major proportion, you can also change the default network type to underlay. After the change is completed, pods will be created on underlay networks by default, and you can still specify that pods are created on overlay networks by adding annotations to pods. The existing overlay pods are not affected.

If you want to modify the default network type, you need to perform kubectl edit deploy hybridnet-webhook -n kube-system, and the kubectl edit deploy hybridnet-manager -n kube-system and are required to modify the environment variable for container startup from DEFAULT_NETWORK_TYPE to Underlay:

spec:
  containers:
    - name: hybridnet-[manager|webhook]           
      command:
        - /hybridnet/hybridnet-[manager|webhook]
      env:
        - name: DEFAULT_NETWORK_TYPE
          # "Overlay" or "Underlay", 
          # default "Underlay" if environment variable not configured. 
          value: Underlay

After this modification, pods will be created by default in the underlay mode. The network connectivity between new underlay pods and original overlay pods is not affected. In simple terms, underlay pods have an overlay identity that communicates with other overlay pods.

Adding or Removing Network Resources

As shown in the example above, adding network resources of networks or subnets only needs to apply the YAML corresponding to CR. Once networks or subnets are applied, Hybridnet will consider that the basic network configuration has been completed in the environment and use the corresponding CR to allocate network resources.

The operation of deleting networks or subnets has basic constraints for security purposes. The subnet itself can only be deleted when no IP in a subnet is used. Similarly, a network itself can only be deleted when all subnets are deleted in the network first.

ACK Distro: All in all, I can enable ACK to build a unified network plane of underlay and overlay networks on top of heterogeneous environments to improve the capabilities for management and O&M and bring a better container service experience to developers with the help of Hybridnet.

Reporter: Okay. Thank you very much for your careful explanation this time! The second in-depth interview is coming to an end, and I look forward to meeting all of our readers next time.

ACK Distro: We'll see you next time!

References

[1] The Open-Source Repository Link of Hybridnet
https://github.com/alibaba/hybridnet

[2] The Community Documentation of Hybridnet
https://github.com/alibaba/hybridnet/wiki

[3] The Official Website of Alibaba Cloud Container Service for Kubernetes (ACK)
https://www.alibabacloud.com/product/kubernetes

[4] The Official GitHub of ACK Distro
https://github.com/AliyunContainerService/ackdistro

Community

An In-Depth Interview with ACK Distro – Part 2: How to Build a Hybrid Cloud Unified Network Plane with Hybridnet

Definition of Hybridnet and Its Design Concept

The Core Models of Hybridnet

Network

Subnet

IPInstance

How to Manage Hybridnet in ACK Distro

Default Behavior

Adding Underlay Networks

Changing the Default Network Type as Underlay

Adding or Removing Network Resources

References

Read previous post:

Read next post:

Alibaba Cloud Native

You may also like

Comments

Alibaba Cloud Native

Related Products

Hybrid Cloud Solution

ACK One

Networking Overview

Hybrid Cloud Storage