Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Pause, Resume and Scale Kubernetes Deployments

Pause, Resume and Scale Kubernetes Deployments

In this article, we will learn how to use Kubernetes Deployments, focusing specifically on pausing, resuming, and scaling Deployments.

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Pausing, resuming and scaling Deployments are very easy to understand.

We have several commands to show the status of these 3 actions. Unfortunately we will be using the same commands repeatedly so that you can see over time which commands work best for which specific need.

This tutorial consists of 1-2 sentences of descriptions followed by a command to run. Then follows one short snippet of comments and one more command and so on. Each of those steps must be done for the tutorial to work.

Kubectl Rollout Pause

Kubernetes enable you to pause a Deployment. You can then make adjustments to the Deployment and resume it.

Deployments do not need to be paused to make a change. Use pause to pause a Deployment so that you can calmly make several changes ( that are kept in a queue till resume is ordered ).

We start with one normal, working Deployment :

nano myDeployment.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  labels:
    app: busybox
spec:
  replicas: 10
  strategy: 
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        
        command: ['sh', '-c', 'echo Container is Running ; sleep 3600']

      terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yaml

deployment.apps/mydeployment created

Just 2 seconds later the Deployment is complete. ( busybox image already on server, starting such a simple container is very fast - 10 replicas started up simultaneously ... adding to the fast speed. )

kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   10        10        10           0           2s

Check the rollout status:

kubectl rollout status deployment.v1.apps/busybox-deployment

deployment "busybox-deployment" successfully rolled out

Describe the Deployment :

kubectl describe deployment.extensions/busybox-deployment

Name:                   busybox-deployment
CreationTimestamp:      Sun, 20 Jan 2019 10:16:11 +0200
Labels:                 app=busybox
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=busybox
Replicas:               10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType:           RollingUpdate
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=busybox
  Containers:
   busybox:
    Image:      busybox
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   busybox-deployment-5d6889b44 (10/10 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  26s   deployment-controller  Scaled up replica set busybox-deployment-5d6889b44 to 10

The previous 3 commands showed a perfectly running 10 Pods Deployment.

Now we pause this Deployment and observe its status.

kubectl rollout pause deployment busybox-deployment

deployment.extensions/busybox-deployment paused

get deploy does not actually show this Deployment as paused.

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   10        10        10           10          2m12s

kubectl rollout status does not actually show this Deployment as paused.

kubectl rollout status deployment.v1.apps/busybox-deployment

deployment "busybox-deployment" successfully rolled out

This is a disappointment ... rollout status deployment does NOT show Deployment as being paused.

It shows ROLLOUT status, not DEPLOYMENT status.

Describe deployment :

kubectl describe deployment.extensions/busybox-deployment
Name:                   busybox-deployment
CreationTimestamp:      Sun, 20 Jan 2019 10:16:11 +0200
Labels:                 app=busybox
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=busybox
Replicas:               10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType:           RollingUpdate
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=busybox
  Containers:
   busybox:
    Image:      busybox
Conditions:
  Type           Status   Reason
  ----           ------   ------
  Available      True     MinimumReplicasAvailable
  Progressing    Unknown  DeploymentPaused
OldReplicaSets:  <none>
NewReplicaSet:   busybox-deployment-5d6889b44 (10/10 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  2m28s  deployment-controller  Scaled up replica set busybox-deployment-5d6889b44 to 10

ONLY one line shows that this Deployment is now paused:

  Progressing    Unknown  DeploymentPaused

IMPORTANT: ONLY use kubectl describe to reliably find paused Deployments.

Make Changes to Paused Deployment

While a Deployment is paused we can make changes to its Pods.

Change all our replicated Pods to use busybox:1.30-glibc image.

--record the fact that we are using a new image in the Deployment annotations.

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record
deployment.apps/busybox-deployment image updated

This specific change is not applied immediately ( Deployment is currently paused ).

While a Deployment is paused we can make changes to its number of replicas.

kubectl scale deployment.v1.apps/busybox-deployment --replicas=5
deployment.apps/busybox-deployment scaled

Changes to number of replicas are implemented immediately.

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   5         5         0            5           3m44s

Important: note the up-to-date column ... none of the available / running Pods are up to date. They are all using the busybox image they used at the exact pause command time.

Show list of Pods - they still run old busybox image.

kubectl get pods --show-labels

NAME                                 READY   STATUS    RESTARTS   AGE     LABELS
busybox-deployment-5d6889b44-78tq8   1/1     Running   0          4m59s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7xwds   1/1     Running   0          4m59s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-gjgc5   1/1     Running   0          4m59s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-j47xp   1/1     Running   0          4m59s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-kbh7t   1/1     Running   0          4m59s   app=busybox,pod-template-hash=5d6889b44

Kubectl Rollout Resume

We need to resume our rollout so that a new set of Pods can be created using the image we requested : busybox:1.30-glibc

kubectl rollout resume deployment busybox-deployment

deployment.extensions/busybox-deployment resumed

3 seconds later: 3 new Pods at bottom are being created.

kubectl get pods --show-labels

NAME                                 READY   STATUS              RESTARTS   AGE     LABELS
busybox-deployment-5d6889b44-78tq8   1/1     Running             0          5m52s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7xwds   1/1     Running             0          5m52s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-gjgc5   1/1     Running             0          5m52s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-kbh7t   1/1     Running             0          5m52s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-f54b89b4-8xpv2    0/1     ContainerCreating   0          3s      app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-lgbxv    0/1     ContainerCreating   0          3s      app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-x64m5    0/1     ContainerCreating   0          3s      app=busybox,pod-template-hash=f54b89b4

another 10 seconds later ...

3 Pods in middle of list are already new.

Pod at the top of list is old. Very easy to determine old versus new: old Pods have older age.

( Freshly created new Pods have young age )

kubectl get pods --show-labels

NAME                                 READY   STATUS              RESTARTS   AGE    LABELS
busybox-deployment-5d6889b44-7xwds   1/1     Running             0          6m5s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-f54b89b4-8xpv2    0/1     ContainerCreating   0          16s    app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-cbxt4    1/1     Running             0          3s     app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-lgbxv    1/1     Running             0          16s    app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-qkj2b    1/1     Running             0          4s     app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-x64m5    0/1     ContainerCreating   0          16s    app=busybox,pod-template-hash=f54b89b4

another 10 seconds later ...

All Pods are new ... short running age is the easy giveaway hint.

kubectl get pods --show-labels

NAME                                READY   STATUS    RESTARTS   AGE   LABELS
busybox-deployment-f54b89b4-8xpv2   1/1     Running   0          26s   app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-cbxt4   1/1     Running   0          13s   app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-lgbxv   1/1     Running   0          26s   app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-qkj2b   1/1     Running   0          14s   app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-x64m5   1/1     Running   0          26s   app=busybox,pod-template-hash=f54b89b4

Display Deployment status ... all 5 Pods up to date.

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   5         5         5            5           7m4s

Note the last column. Overall age of this Deployment is 7 minutes. Only last 30 seconds used the new busybox image.

Show the rollout history of the Deployment :

kubectl rollout history deployment.v1.apps/busybox-deployment

deployment.apps/busybox-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true

We see our new busybox image listed as revision 2.

Scaling Paused Deployments

Deployments can be scaled while they are running.

Scaling in this context means changing the number of running identical Pod replicas.

Below is a demo that you can freely scale a paused Deployment :

Current state for reference:

kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   5         5         5            5           27m

We have 5 running Pods.

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-8xpv2   1/1     Running   0          25m
busybox-deployment-f54b89b4-cbxt4   1/1     Running   0          25m
busybox-deployment-f54b89b4-lgbxv   1/1     Running   0          25m
busybox-deployment-f54b89b4-qkj2b   1/1     Running   0          25m
busybox-deployment-f54b89b4-x64m5   1/1     Running   0          25m

Pause Deployment :

kubectl rollout pause deployment busybox-deployment
deployment.extensions/busybox-deployment paused

Get list of Pods:

IMPORTANT: Pausing a Deployment DOES NOT pause its containers.

All Pods still running.

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-8xpv2   1/1     Running   0          25m
busybox-deployment-f54b89b4-cbxt4   1/1     Running   0          25m
busybox-deployment-f54b89b4-lgbxv   1/1     Running   0          25m
busybox-deployment-f54b89b4-qkj2b   1/1     Running   0          25m
busybox-deployment-f54b89b4-x64m5   1/1     Running   0          25m

Change replicas to 2:

kubectl scale deployment.v1.apps/busybox-deployment --replicas=2

deployment.apps/busybox-deployment scaled

Investigate change ... only 2 Pods now running:

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-lgbxv   1/1     Running   0          26m
busybox-deployment-f54b89b4-qkj2b   1/1     Running   0          25m

Change replicas to 6:

kubectl scale deployment.v1.apps/busybox-deployment --replicas=6

deployment.apps/busybox-deployment scaled

Investigate change ... 6 Pods now running:

Four new Pods have an age of only 4 seconds.

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-4fw4b   1/1     Running   0          4s
busybox-deployment-f54b89b4-lgbxv   1/1     Running   0          26m
busybox-deployment-f54b89b4-n952j   1/1     Running   0          4s
busybox-deployment-f54b89b4-qkj2b   1/1     Running   0          26m
busybox-deployment-f54b89b4-t569g   1/1     Running   0          4s
busybox-deployment-f54b89b4-xlwr8   1/1     Running   0          4s

Let's change the Pods to all use another different busybox image: busybox:1.30-uclibc

Important: right now the Deployment is paused. It only takes note of the newly desired image. It does not roll out the change immediately - it is waiting for the kubectl rollout resume command.

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record
deployment.apps/busybox-deployment image updated

Show list of Pods. They are unaware a change is coming. Still running as before.

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-4fw4b   1/1     Running   0          40s
busybox-deployment-f54b89b4-lgbxv   1/1     Running   0          26m
busybox-deployment-f54b89b4-n952j   1/1     Running   0          40s
busybox-deployment-f54b89b4-qkj2b   1/1     Running   0          26m
busybox-deployment-f54b89b4-t569g   1/1     Running   0          40s
busybox-deployment-f54b89b4-xlwr8   1/1     Running   0          40s

The Deployment status shows ZERO up to date Pods. It is aware a change is pending while Deployment is in a paused state.

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   6         6         0            6           32m

Rollout history does not show new busybox:1.30-uclibc image pending.

It only shows history of previous and running Deployments status.

kubectl rollout history deployment.v1.apps/busybox-deployment

deployment.apps/busybox-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true

Let's resume the Deployment :

kubectl rollout resume deployment busybox-deployment

deployment.extensions/busybox-deployment resumed

5 seconds later ... 3 new Pods being created.

kubectl get pods

NAME                                  READY   STATUS              RESTARTS   AGE
busybox-deployment-6fc48bf75f-92pkn   0/1     ContainerCreating   0          5s
busybox-deployment-6fc48bf75f-qv8wv   0/1     ContainerCreating   0          5s
busybox-deployment-6fc48bf75f-wgrv5   0/1     ContainerCreating   0          5s
busybox-deployment-f54b89b4-4fw4b     1/1     Running             0          3m1s
busybox-deployment-f54b89b4-lgbxv     1/1     Running             0          29m
busybox-deployment-f54b89b4-n952j     1/1     Running             0          3m1s
busybox-deployment-f54b89b4-qkj2b     1/1     Running             0          29m
busybox-deployment-f54b89b4-xlwr8     1/1     Running             0          3m1s

another 5 seconds later:

  • 3 new Pods running
  • 3 new Pods being created
  • 2 old Pods running ( at bottom ) waiting for termination
kubectl get pods

NAME                                  READY   STATUS              RESTARTS   AGE
busybox-deployment-6fc48bf75f-74wlt   1/1     Running             0          4s
busybox-deployment-6fc48bf75f-92pkn   0/1     ContainerCreating   0          10s
busybox-deployment-6fc48bf75f-qv8wv   1/1     Running             0          10s
busybox-deployment-6fc48bf75f-wgrv5   0/1     ContainerCreating   0          10s
busybox-deployment-6fc48bf75f-xcv5c   0/1     ContainerCreating   0          2s
busybox-deployment-6fc48bf75f-xqmzc   1/1     Running             0          3s
busybox-deployment-f54b89b4-lgbxv     1/1     Running             0          29m
busybox-deployment-f54b89b4-qkj2b     1/1     Running             0          29m

A few seconds later shows all Pods new ( young AGE )

kubectl get pods

NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-6fc48bf75f-74wlt   1/1     Running   0          15s
busybox-deployment-6fc48bf75f-92pkn   1/1     Running   0          21s
busybox-deployment-6fc48bf75f-qv8wv   1/1     Running   0          21s
busybox-deployment-6fc48bf75f-wgrv5   1/1     Running   0          21s
busybox-deployment-6fc48bf75f-xcv5c   1/1     Running   0          13s
busybox-deployment-6fc48bf75f-xqmzc   1/1     Running   0          14s

All Pods up to date with desired new busybox image:

kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   6         6         6            6           35m

History now shows new busybox image as revision 3.

kubectl rollout history deployment.v1.apps/busybox-deployment
deployment.apps/busybox-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true
3         kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true

Describe Deployment :

kubectl describe deployment.extensions/busybox-deployment

Name:                   busybox-deployment
CreationTimestamp:      Sun, 20 Jan 2019 10:16:11 +0200
Labels:                 app=busybox
Annotations:            deployment.kubernetes.io/revision: 3
                        kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true
Selector:               app=busybox
Replicas:               6 desired | 6 updated | 6 total | 6 available | 0 unavailable
StrategyType:           RollingUpdate
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=busybox
  Containers:
   busybox:
    Image:      busybox:1.30-uclibc
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   busybox-deployment-6fc48bf75f (6/6 replicas created)

Note the annotation that shows which image is being used.

Delete Pod.

kubectl delete -f myDeployment.yaml --force --grace-period=0

deployment.apps "busybox-deployment" force deleted

Kubectl Rollout Undo

Deployments allow you to roll back to any previous revision.

By default it keeps a list of your last 10 revisions.

Let's create our normal test set of 10 Pod replicas :

( First few steps should be boring familiar by now )

nano myDeployment.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mydeployment
  labels:
    app: busybox
spec:
  replicas: 10
  strategy: 
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        
        command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']

      terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yaml

deployment.apps/mydeployment created

Show history:

kubectl rollout history deployment.v1.apps/busybox-deployment

deployment.apps/busybox-deployment
REVISION  CHANGE-CAUSE
1         <none>

Show the Pods:

kubectl get pods --show-labels

NAME                                 READY   STATUS    RESTARTS   AGE    LABELS
busybox-deployment-5d6889b44-2sb8m   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-6dtgr   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-6t94r   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7j6bg   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-8s5lq   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-dn62f   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-gpsrv   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-rxbqx   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-tvxrn   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-vt7dd   1/1     Running   0          2m1s   app=busybox,pod-template-hash=5d6889b44

Show version of busybox running ... to compare against later.

Remember that ... BusyBox v1.30.0

kubectl exec busybox-deployment-5d6889b44-2sb8m -i -t -- /bin/sh

/ # busybox | head -1
BusyBox v1.30.0 (2018-12-31 18:16:17 UTC) multi-call binary.
/ # exit

Change all Pods to use busybox:1.29.3

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record

deployment.apps/busybox-deployment image updated

Investigate Pods during rollout process

kubectl get pods --show-labels

NAME                                  READY   STATUS              RESTARTS   AGE    LABELS
busybox-deployment-5d6889b44-6dtgr    1/1     Running             0          3m6s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-755956dfff-65vvk   0/1     ContainerCreating   0          4s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-gmlmm   1/1     Running             0          8s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-j2w84   0/1     ContainerCreating   0          4s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lcjz2   1/1     Running             0          8s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lfvzn   0/1     ContainerCreating   0          4s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-m4vzz   1/1     Running             0          5s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-mzw75   1/1     Running             0          8s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-njtm2   1/1     Running             0          8s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-nw6nh   1/1     Running             0          8s     app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-pzbmt   1/1     Running             0          5s     app=busybox,pod-template-hash=755956dfff

Seconds later ... complete set of 10 fresh new Pods with young age.

kubectl get pods --show-labels

NAME                                  READY   STATUS    RESTARTS   AGE   LABELS
busybox-deployment-755956dfff-65vvk   1/1     Running   0          14s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-gmlmm   1/1     Running   0          18s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-j2w84   1/1     Running   0          14s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lcjz2   1/1     Running   0          18s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lfvzn   1/1     Running   0          14s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-m4vzz   1/1     Running   0          15s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-mzw75   1/1     Running   0          18s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-njtm2   1/1     Running   0          18s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-nw6nh   1/1     Running   0          18s   app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-pzbmt   1/1     Running   0          15s   app=busybox,pod-template-hash=755956dfff

Check to make sure the Pods are using version 29.3 busybox:

kubectl exec busybox-deployment-755956dfff-65vvk -i -t -- /bin/sh
/ # busybox | head -1
BusyBox v1.29.3 (2018-12-24 21:25:20 UTC) multi-call binary.
/ # exit

Show history. New busybox is revision 2.

kubectl rollout history deployment.v1.apps/busybox-deployment

deployment.apps/busybox-deployment
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record=true

Describe shows annotation as expected.

kubectl describe deployment.extensions/busybox-deployment

Name:                   busybox-deployment
Namespace:              default
CreationTimestamp:      Sat, 19 Jan 2019 09:10:49 +0200
Labels:                 app=busybox
Annotations:            deployment.kubernetes.io/revision: 2
                        kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record=true

NOW we undo this latest Deployment.

We undo deployment to revision 1 via flag : --to-revision=1

kubectl rollout undo deployment busybox-deployment --to-revision=1

deployment.extensions/busybox-deployment rolled back

Follow state of undo rollout

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   10        13        8            8           10m

8 Pods up to date in a few seconds.

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   10        11        10           8           10m

Still only 8 Pods up to date.

kubectl get deploy

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   10        10        10           10          10m

Seconds later ... all 10 Pods back to the previous version busybox.

kubectl get pods --show-labels

NAME                                 READY   STATUS    RESTARTS   AGE     LABELS
busybox-deployment-5d6889b44-4frh6   1/1     Running   0          2m28s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-4kfrf   1/1     Running   0          2m28s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-59qsz   1/1     Running   0          2m26s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-6xmkx   1/1     Running   0          2m26s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-77qnl   1/1     Running   0          2m25s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7c9xb   1/1     Running   0          2m28s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-8pg76   1/1     Running   0          2m26s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-bjc5s   1/1     Running   0          2m24s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-jnqf5   1/1     Running   0          2m28s   app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-z5xql   1/1     Running   0          2m28s   app=busybox,pod-template-hash=5d6889b44

Check Pods are using busybox version 1.30 ... the number we had to remember.

kubectl exec busybox-deployment-5d6889b44-4frh6 -i -t -- /bin/sh

/ # busybox | head -1
BusyBox v1.30.0 (2018-12-31 18:16:17 UTC) multi-call binary.
/ # exit

Success. kubectl rollout undo deployment correctly rolled back to revision 1.

Delete Deployment

kubectl delete -f myDeployment.yaml --force --grace-period=0

deployment.apps "busybox-deployment" force deleted

Deployment Internals

Let's create a Deployment and explore its internals.

nano myDeployment.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  labels:
    app: busybox
spec:
  replicas: 5
  strategy: 
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        
        command: ['sh', '-c', 'echo Container is Running ; sleep 3600']
        
      terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yaml

pod/mydeployment created

List Deployment status :

kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   5         5         5            5           5s

A Deployment actually manages a ReplicaSet that runs under its command.

kubectl get rs

NAME                            DESIRED   CURRENT   READY   AGE
busybox-deployment-1111111111   5         5         5       14s

rs is ReplicaSet abbreviation for command line use.

ReplicaSet name starts with name of Deployment. Kubernetes adds a 10 digit random number.

kubectl get pods

NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-1111111111-6v8qt   1/1     Running   0          19s
busybox-deployment-1111111111-fr4q7   1/1     Running   0          19s
busybox-deployment-1111111111-p8h2l   1/1     Running   0          19s
busybox-deployment-1111111111-pr85v   1/1     Running   0          19s
busybox-deployment-1111111111-wt7hh   1/1     Running   0          19s

These Pods run under control of the ReplicaSet. Kubernetes adds 5 digit random number to individual Pod names.

Actual old and new Pods were similar to :

busybox-deployment-568495f8b6-rv8sw
and
busybox-deployment-8659fbc5bf-6v8qt

I changed it to 1111.... and 3333... so it is easy to differentiate in lists below.

  • 1111... old, Pods created first
  • 3333... new, current fresh Pods

Let's scale to 7 replicas:

  • Deployment gets command to scale to 7 replicas
  • Deployment sends this command to the ReplicaSet
  • ReplicaSet receive command, counts 5 replicas, need 7 so create 2 more Pods.
kubectl scale deployment.v1.apps/busybox-deployment --replicas=7
deployment.apps/busybox-deployment scaled

Deployment now shows 7 running replicas.

kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   7         7         7            7           113s

ReplicaSets now show 7 running replicas.

kubectl get rs
NAME                            DESIRED   CURRENT   READY   AGE
busybox-deployment-1111111111   7         7         7       116s

Detail of 7 running Pods.

kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-1111111111-6v8qt   1/1     Running   0          2m
busybox-deployment-1111111111-7bkvx   1/1     Running   0          11s
busybox-deployment-1111111111-fr4q7   1/1     Running   0          2m
busybox-deployment-1111111111-hj426   1/1     Running   0          11s
busybox-deployment-1111111111-p8h2l   1/1     Running   0          2m
busybox-deployment-1111111111-pr85v   1/1     Running   0          2m
busybox-deployment-1111111111-wt7hh   1/1     Running   0          2m

Let's use this busybox image: busybox:1.30-uclibc

  • Deployment gets command : kubectl set image ...
  • Deployment sends this command to the ReplicaSet
  • ReplicaSet receive command, determines all Pods are running wrong version of busybox
  • A new set of Pods in the series busybox-deployment-3333333333 gets created
  • Some old busybox-deployment-1111111111 Pods are kept running during rolling update

maxUnavailable determines how many old Pods may be unavailable during update

maxSurge determines how many NEW Pods may be created in addition to already running Pods

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record

deployment.apps/busybox-deployment image updated

Investigate status of rolling update:

1111... series keep on doing useful work while update is in progress
3333... series ... some new Pods already running
3333... series ... some new Pods in process of being created: ContainerCreating

kubectl get pods

NAME                                  READY   STATUS              RESTARTS   AGE
busybox-deployment-3333333333-2sqcs   0/1     ContainerCreating   0          1s
busybox-deployment-3333333333-69xq9   1/1     Running             0          3s
busybox-deployment-3333333333-6ph4z   1/1     Running             0          3s
busybox-deployment-3333333333-8trbx   0/1     ContainerCreating   0          1s
busybox-deployment-3333333333-fmqwv   0/1     ContainerCreating   0          1s
busybox-deployment-3333333333-rv8sw   1/1     Running             0          3s
busybox-deployment-1111111111-6v8qt   1/1     Running             0          3m36s
busybox-deployment-1111111111-fr4q7   1/1     Running             0          3m36s
busybox-deployment-1111111111-p8h2l   1/1     Running             0          3m36s

A few seconds later and the rolling update is complete. All old 111... Pods deleted.

kubectl get pods

NAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-3333333333-2sqcs   1/1     Running   0          5s
busybox-deployment-3333333333-69xq9   1/1     Running   0          7s
busybox-deployment-3333333333-6ph4z   1/1     Running   0          7s
busybox-deployment-3333333333-8trbx   1/1     Running   0          5s
busybox-deployment-3333333333-fmqwv   1/1     Running   0          5s
busybox-deployment-3333333333-r2cgc   1/1     Running   0          4s
busybox-deployment-3333333333-rv8sw   1/1     Running   0          7s

7 new Pods up-to-date and available.

kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment   7         7         7            7           5m8s

List of ReplicaSets.

Note the old ReplicaSet is kept in case we need to roll back a Deployment.

kubectl get rs

NAME                            DESIRED   CURRENT   READY   AGE
busybox-deployment-3333333333   7         7         7       98s
busybox-deployment-1111111111   0         0         0       5m11s

From https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#revision-history-limit

All old ReplicaSets will be kept by default, consuming resources in etcd and crowding the output of kubectl get rs, if this field is not set.

etcd is the 'database' / key-value data store Kubernetes use to keep track of all its API objects.

etcd keeps at least 3 types of information about all Kubernetes API objects:

  • meta data ( data about data ) https://en.wikipedia.org/wiki/Metadata
  • Kubernetes uses metadata to keep track of objects and to note object interrelationships ( see below )
  • our desired spec ... details of the spec we need for our objects
  • the actual status of this specific object

This can be clearly and neatly seen when we run :

kubectl get rs/busybox-deployment-568495f8b6 -o yaml

apiVersion: extensions/v1beta1
kind: ReplicaSet


metadata:
  annotations:
    deployment.kubernetes.io/desired-replicas: "7"
    deployment.kubernetes.io/max-replicas: "9"
    deployment.kubernetes.io/revision: "2"
    kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment
      busybox=busybox:1.30-uclibc --record=true
  creationTimestamp: "2019-01-21T07:09:11Z"
  generation: 6
  labels:
    app: busybox
    pod-template-hash: 568495f8b6
  name: busybox-deployment-568495f8b6
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: busybox-deployment
    uid: f510ce02-1d4a-11e9-8596-0800270102d2
  resourceVersion: "602067"
  selfLink: /apis/extensions/v1beta1/namespaces/default/replicasets/busybox-deployment-568495f8b6
  uid: 73b2480b-1d4b-11e9-8596-0800270102d2


spec:
  replicas: 7
  selector:
    matchLabels:
      app: busybox
      pod-template-hash: 568495f8b6
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: busybox
        pod-template-hash: 568495f8b6
    spec:
      containers:
      - command:
        - sh
        - -c
        - echo Container is Running ; sleep 3600
        image: busybox:1.30-uclibc
        imagePullPolicy: IfNotPresent
        name: busybox
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0


status:
  availableReplicas: 7
  fullyLabeledReplicas: 7
  observedGeneration: 6
  readyReplicas: 7
  replicas: 7

Kubernetes Deployment course complete ... delete Deployment :

kubectl delete -f myDeployment.yaml --force --grace-period=0

pod "mydeployment" force deleted

Deleting a Deployment deletes the Deployment itself, underlying ReplicaSets and all its Pods.

Your Turn

Experiment with all the Deployment settings: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#writing-a-deployment-spec

Determine the correct settings for all your production Pods and their controlling Deployments.

Container Computing Tech Share Container Service Kubernetes Tutorials Deployments
0 0 0
Share on

Read previous post:

How to Deploy PrestaShop on Alibaba Cloud ECS

Read next post:

Kubernetes Batch Jobs

Alibaba Clouder

2,599 posts | 762 followers

You may also like

Comments

Alibaba Clouder

2,599 posts | 762 followers

Related Products

More Posts by Alibaba Clouder

See All