Most containers within Alibaba are used in rich container mode. Some containers among the rich containers in traditional virtual machine O&M mode are still stateful. Updates and upgrades of stateful services are frequently performed daily operations within Alibaba. For the container technology that delivers images, the container operations corresponding to service update and upgrade include two steps: deletion of old image container and creation of the new image container.
The stateful service upgrade requires that the new container must inherit all resources of old container, such as network and storage. The following two cases are the service provision scenarios of rich containers:
Many companies use Moby as the container engine; however, none of the Moby APIs can perform container upgrade independently. API combination will increase the number of API requests, for example, the requests for container adding and deleting APIs and the requests for IP address reservation APIs. In addition, this method may increase the risks of upgrade failure.
To solve the previous problems, PouchContainer provides an upgrade interface at the container engine layer to implement local upgrade of containers. That is, the container upgrade function is lowered to the container engine layer. This facilitates operations on container resources and significantly reduces the number of API requests, improving the container upgrade efficiency.
The bottom layer of PouchContainer is connected to Containerd v1.0.3, which has a very different storage architecture than Moby. Before understanding how the PouchContainer implements local container upgrade, it is necessary to learn about the container storage architecture in PouchContainer:
Compared with the container storage architecture of Moby, PouchContainer has the following characteristics:
In the design of each system and function, it is necessary to investigate what pain points the system or function can solve for users. After investigating the container local upgrade scenarios within Alibaba, we posed three requirements on the upgrade
function:
Data consistency means that after the upgrade, some data must be unchanged:
Flexibility means that new configurations are allowed to be introduced based on the configurations of the old container during the upgrade
operation:
Entrypoint
needs to be specified, but also the Entrypoint
of the old container must be inherited.Robustness refers to the capability of processing abnormalities during the container local upgrade. The rollback policy must be supported to restore the configurations of old container if the upgrade fails.
The upgrade
API defines which parameters can be modified in the upgrade operation. The following is the definition of ContainerUpgradeConfig
. In the container upgrade, the ContainerConfig
and HostConfig
parameters can be modified. If you view the definitions of the two parameters under the apis/types
directory in the PouchContainer GitHub code repository, you will find that the upgrade
operation can modify all the related configurations of the old container.
// ContainerUpgradeConfig ContainerUpgradeConfig is used for API "POST /containers/upgrade".
// It wraps all kinds of config used in container upgrade.
// It can be used to encode client params in client and unmarshal request body in daemon side.
//
// swagger:model ContainerUpgradeConfig
type ContainerUpgradeConfig struct {
ContainerConfig
// host config
HostConfig *HostConfig `json:"HostConfig,omitempty"`
}
The container upgrade
operation is to delete the old container without modifying the network configurations and original volumes, and create a new container by using the new image. The following is the procedure of the upgrade
operation:
Entrypoint
parameter: If the new parameters include Entrypoint
, the new Entrypoint
parameter is used. Otherwise, check the Entrypoint
parameter of the old container. If this parameter is specified in configurations but not contained in the old image, the Entrypoint
parameter of the old container is used as Entrypoint of the new container. If Entrypoint
is neither included in new parameters nor specified in configurations, the Entrypoint
in the new image is used as the Entrypoint
of the new container. Such processing on the Entrypoint
of the new container is to ensure the continuity of container service entry parameters.The upgrade
operation may have errors. Currently, a rollback is performed to restore the old container when an error occurs. First, we need to define what upgrade failures are:
The following is the basic operation of rollback:
defer func() {
if !needRollback {
return
}
// rollback to old container.
c.meta = &backupContainerMeta
// create a new containerd container.
if err := mgr.createContainerdContainer(ctx, c); err != nil {
logrus.Errorf("failed to rollback upgrade action: %s", err.Error())
if err := mgr.markStoppedAndRelease(c, nil); err != nil {
logrus.Errorf("failed to mark container %s stop status: %s", c.ID(), err.Error())
}
}
}()
If an error occurs during upgrade, the newly created resources such as snapshot are cleared. In the rollback stage, only the old container configurations need to be restored, and the new container starts with the restored configuration file.
Create a new container by using the ubuntu
image:
$ pouch run --name test -d -t registry.hub.docker.com/library/ubuntu:14.04 top
43b75002b9a20264907441e0fe7d66030fb9acedaa9aa0fef839ccab1f9b7a8f
$ pouch ps
Name ID Status Created Image Runtime
test 43b750 Up 3 seconds 3 seconds ago registry.hub.docker.com/library/ubuntu:14.04 runc
Upgrade the image of the test
container to busybox
:
$ pouch upgrade --name test registry.hub.docker.com/library/busybox:latest top
test
$ pouch ps
Name ID Status Created Image Runtime
test 43b750 Up 3 seconds 34 seconds ago registry.hub.docker.com/library/busybox:latest runc
As shown in the previous demonstration, the container's image is replace with a new one by using the upgrade
interface, and other configurations are unchanged.
In the production environment of a company, the upgrade
operation is also frequently performed like container scale-up and scale-down. However, neither the Moby community nor the Containerd community provides an API matching the upgrade operation. PouchContainer unprecedentedly implements this function to solve the stateful service update and provision problem of the container technology for companies. We are trying to improve the closeness between PouchContainer and downstream dependent components such as Containerd, and planning to provide the upgrade function to the Containerd community so as to enrich the functions of Containerd.
Exploring Alibaba Group's PouchContainer Resource Management APIs – Part 2
506 posts | 48 followers
FollowAlibaba Clouder - November 8, 2018
Alibaba System Software - August 27, 2018
Alibaba System Software - August 30, 2018
Alibaba System Software - November 29, 2018
Alibaba System Software - August 27, 2018
Amber Wang - August 6, 2018
506 posts | 48 followers
FollowProvides a control plane to allow users to manage Kubernetes clusters that run based on different infrastructure resources
Learn MoreA secure image hosting platform providing containerized image lifecycle management
Learn MoreCustomized infrastructure to ensure high availability, scalability and high-performance
Learn MoreAlibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.
Learn MoreMore Posts by Alibaba Cloud Native Community