Setting a Whitelist to Specified Pod Segments in Terway for ACK

By Daina

Generally, a whitelist for services, such as databases, is configured to provide services with safer access control capabilities. The scenario in the container network has the same demand. How can we set a whitelist for dynamic Pod IPs in container scenarios?

The container network of Alibaba Cloud Container Service for Kubernetes (ACK) mainly includes Flannel and Terway. In Flannel, the access to other services of a Pod is implemented through the node NAT. Thus, when setting the whitelist to a database in Flannel, we can schedule the client Pod to a small number of fixed nodes by binding nodes. Then, on the database side, the IP address of the node can be whitelisted. This method will not be described in detail. However, in Terway, the Pod IP is provided through EtherCAT Network Information (ENI). The client IP obtained by the external service is the IP address provided by ENI. With an affinity binding between Pods and nodes, the client IP of the Pod accessing external service is the IP provided by ENI, not the nodes. A Pod IP also randomly allocates IP addresses from the vSwitch specified by Terway. Besides, the client Pods are usually configured with automatic scaling. Thus, it is still difficult to enable auto-scaling scenarios even with a fixed Pod IP. The best way is to appoint a network segment directly to the client to allocate IP and then set whitelists to the segment in the database. However, how can we specify the IP range of a Pod in Terway?

Terway allows Pods on specified nodes to use the corresponding vSwitch. For more information, please see this link (article in Chinese):

Simply speaking, it specifies the vSwitch used by the Pod by adding labels to the specified node. Therefore, when scheduled a Pod to the node with a fixed label, the Pod can use its customized vSwitch to create a Pod IP. The related steps are as follows:

Create an individual configmap eni-config-fixed instance in kube-system and specify vSwitch, vsw-2zem796p76viir02c6980 10.2.1.0/24.

apiVersion: v1
data:
  eni_conf: |
    {
       "vswitches": {"cn-beijing-h":["vsw-2zem796p76viir02c6980"]}
    }
kind: ConfigMap
metadata:
  name: eni-config-fixed
  namespace: kube-system

Create a node pool and label the nodes with terway-config: eni-config-fixed. To avert other Pods on the nodes in the node pool, label the node pool with taints, for example, fixed=true:NoSchedule.
Nodes added by the node pool contain the taints and labels above by default.
Create a Pod and schedule it to the labeled node. It is advisable to add tolerations.

apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: nginx-fixed
  labels:
    app: nginx-fixed
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-fixed
  template:
    metadata:
      labels:
        app: nginx-fixed
    spec:
      tolerations:
      - key: "fixed"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      nodeSelector:
        terway-config: eni-config-fixed
      containers:
      - name: nginx
        image: nginx:1.9.0 # replace it with your exactly <image_name:tags>
        ports:
        - containerPort: 80

The Pod IP has been assigned from the specified vSwitch.

kubectl get po -o wide | grep fixed
nginx-fixed-57d4c9bd97-lwfxr                   1/1     Running             0          39s    10.2.1.124    bj-tw.062149.aliyun.com   <none>           <none>
nginx-fixed-57d4c9bd97-tk4j9                   1/1     Running             0          39s    10.2.1.125    bj-tw.062148.aliyun.com   <none>           <none>

After scaling the deployment, all the generated Pod IPs are under the specified vSwitch.

kubectl scale deployment nginx-fixed --replicas=30
nginx-fixed-57d4c9bd97-2rxjz                   1/1     Running     0          60s     10.2.1.132    bj-tw.062148.aliyun.com   <none>           <none>
nginx-fixed-57d4c9bd97-4w76t                   1/1     Running     0          60s     10.2.1.144    bj-tw.062149.aliyun.com   <none>           <none>
nginx-fixed-57d4c9bd97-5brzw                   1/1     Running     0          60s     10.2.1.143    bj-tw.062148.aliyun.com   <none>           <none>
...

This way, we can whitelist the vSwitch directly on the database side to implement access control for dynamic Pod IPs.

Notes:

The best way is to create a new node. For existing nodes, you must unbind the ENI from the Alibaba Cloud Elastic Compute Service (ECS) instance before adding them to the cluster, which can be performed on the ECS console. Then, you can add the existing nodes automatically to the cluster to replace the system disk.
Add the labels and taints to the specific node pool to ensure that the business that does not need to be whitelisted is not scheduled to these nodes.
This operation is a configuration overlay that overwrites the previous configuration of eni-config with that in the specified configmap. Please refer to this link (article in Chinese) to configure.
It’s recommended that the number of IP addresses in the specified vSwitch is twice or more than the expected number of Pods. This leaves more space for later expansions and avoids the situation where no IP can be assigned when the Pod IPs cannot be recovered in time due to the occurrence of the fault.

Community

Setting a Whitelist to Specified Pod Segments in Terway for ACK

Notes:

Read previous post:

Read next post:

Alibaba Container Service

You may also like

Comments

Alibaba Container Service

Related Products

ACK One

Container Service for Kubernetes

Function Compute

Elastic High Performance Computing Solution