[Agones Series – Part 2] Address and Port of the Game Server

By Qiuyang Liu (Liming)

In Agones Series – Part 1, we deployed a game server, and gs automatically obtained an ADDRESS and PORT. Where did this address and port come from, and how was it allocated? This article will reveal the network type of the Agones game server.

First, Agones uses the network type of host IP + port forwarding. Load Balancer is not used to reduce forwarding and network latency. Agones can run sidecar containers to have an independent network space for the pod. It does not use hostnetwork but uses host port forwarding. Agones will assign a host port to the game server, and the container port of the game server will be exposed through the containerPort field. The route from the host port to the container port is usually performed by the iptables or ipvs of the host, depending on the container network model.

Now, let's look at the allocation logic of addresses and ports.

Address

As mentioned, the game server uses the host IP and will get the corresponding IP information from the Kubernetes node object of the node where the game server is located. The specific logic is listed below:

// agones/pkg/gameservers/gameservers.go
func address(node *corev1.Node) (string, error) {

    externalDNS := runtime.FeatureEnabled(runtime.NodeExternalDNS)

    if externalDNS {
        for _, a := range node.Status.Addresses {
            if a.Type == corev1.NodeExternalDNS {
                return a.Address, nil
            }
        }
    }

    for _, a := range node.Status.Addresses {
        if a.Type == corev1.NodeExternalIP && net.ParseIP(a.Address) != nil {
            return a.Address, nil
        }
    }

    // There might not be a public DNS/IP, so fall back to the private DNS/IP
    if externalDNS {
        for _, a := range node.Status.Addresses {
            if a.Type == corev1.NodeInternalDNS {
                return a.Address, nil
            }
        }
    }

    for _, a := range node.Status.Addresses {
        if a.Type == corev1.NodeInternalIP && net.ParseIP(a.Address) != nil {
            return a.Address, nil
        }
    }

    return "", errors.Errorf("Could not find an address for Node: %s", node.ObjectMeta.Name)
}

Then, Agones finds the IP addresses whose type is ExternalDNS, ExternalIP, InternalDNS, and InternalIP in the node.Status.Address. The example in the previous article used the ExternalIP of nodes, namely 120.27.21.131.

Port

Let's focus on how ports are allocated. Agones uses the port of the host where the pod is located to expose to the user for connection, so we see that the port in the status field of gs is called HostPort. There are three ways to allocate HostPort:

1) Static: The exposed port is defined by the user.

2) Dynamic: Agones will select an open port for gs.

3) Passthrough: Set containerPort to HostPort. Agones uses one PortAllocator for port allocation. The following focuses on PortAllocator.

PortAllocator is an allocation logic built around cached data, which is called portAllocations. The following is the data structure:

portAllocations    []map[int32]bool

It records the cluster node and whether the corresponding port of each node is occupied. It is easier to explain with the following table. Node 0 is the first element of the slice and includes whether each open port is occupied by gs. Note: The Node here is a logical node and does not correspond to a node in the cluster. It is set up to facilitate the allocation/recycling of ports. Therefore, the cache is a slice rather than a map that records nodeName.

True / False	Node 0	...	Node N
Min Port Num	√	...	x
...	√	...	√
Max Port Num	√	...	x

In terms of cache, in addition to the portAllocations, there is another gameServerRegistry, which is map type. Key is the id of gs, and the value is bool type, indicating whether the gs has been registered and occupied.

First, after the PortAllocator is started, it will be initialized (func (pa *PortAllocator) syncAll() error). Its purpose is to realize the initial construction of portAllocations and gameServerRegistry data structures by traversing node and gameserver. The following is the construction process:

Use lister to obtain all nodes and gameserver
Initialize a nodePortAllocation according to the existing node, which is the portAllocations of the map version, and the key is nodename. There is also a nodePortCount that records how many ports each node has occupied.
Traverse all gameservers and their Ports fields, ignore the Static type, and record the gsRegistry corresponding to gs as true. If gs has a corresponding nodename, update the nodePortAllocation, and record the port corresponding to the node as true. Add one to the nodePortCount of the node. If gs hostport already exists but does not correspond to nodename, record these port numbers with nonReadyNodesPorts.
The nodePortAllocation is sorted according to the nodePortCount. The node with more ports is at the top to get a slice, which is the embryonic form of portAllocations. This means the smaller the index in the portAllocations, the more its map value is true (as shown in the example in the preceding table, Node 0 has more numbers than Node N and true).
Add all the port numbers recorded by the nonReadyNodesPorts to the portAllocations. In the order from front to back, set it to true as long as the corresponding port in the first node is not occupied, which means the port is occupied. Finally, we got portAllocations and gameServerRegistry. As such, the hostport of the node that has not been allocated will not be missed, nor will it interfere with the logic in the order of how many node ports are occupied.

Two cache portAllocations and gameServerRegistry are built. The allocation logic is simple: according to the number of gs ports to be allocated, find the corresponding number of unallocated ports from the portAllocations in sequence and assign them to the fields corresponding to gs. At the same time, set the gs corresponding to the gameServerRegistry to true.

Finally, look at the recycling logic: traverse the hostPort in gs, find the corresponding port number according to the portAllocations order, change it to false, and delete the gameServerRegistry item corresponding to gs.

Overall, we can see the PortAllocator allocation idea from the code.

1) It does not pay attention to which node the port gs wants to allocate, as long as it knows whether it is occupied or not. There is no need to use the map to increase the complexity of retrieval, just take/release it from the logical node with the most allocated ports. Make sure that the number of open ports in the cluster is the same as the number of true ports in the table.

2) Although it has nothing to do with scheduling, port allocation will break up the allocated port number and try to keep the number of open ports with the same number in the clusters down.

Community

[Agones Series – Part 2] Address and Port of the Game Server

Address

Port

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

ACK One

Cloud-Native Applications Management Solution

Gaming Solution

Container Service for Kubernetes