By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.
This tutorial demonstrates the following :
CPU resource usage limits are needed to prevent any one Pod from totally consuming ALL the CPU time on one node ( server running Kubernetes and capable of hosting Pods ). You can use this functionality in development to prevent developers from accidentally hogging all CPU resources.
You can use this functionality in production - to ensure no accidental runaway process in one Pod from hogging all CPU resources - preventing everything else from doing productive work.
This tutorial will cover the following topics:
Pods can declare how much CPU they need.
See example below:
CPU resource limits are defines using two syntaxes:
The second syntax uses Millicores. 1000m equals one CPU on all computers.
( One Millicores is 1/1000 of a CPU, therefore 1000m equals 1 CPU )
A four core server has a CPU capacity of 4000m.
Millicores syntax is easy to read: 150m versus 0.150. Use the syntax you prefer. Have a standard at your company of which to use.
The Pod below defines it needs 500m minimum and 1 CPU maximum.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
resources:
limits:
cpu: "1"
requests:
cpu: 500m
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
We now need to test how those CPU limits are enforced.
We do that using sysbench - a Linux benchmark utility.
Syntax:
sysbench --threads=1 --cpu-max-prime=8005005 --verbosity=0 cpu run
( I randomly played with cpu-max-prime value till I got one that takes around 15 seconds to run. That is enough time to switch to the other console and do a screen capture of the top command output. )
We defined that our Pod needs 1 full CPU to do its work. We can test that using sysbench inside our Pod.
Use kubectl exec to ssh into the Pod. There we can run sysbench on the shown Linux Pod shell prompt.
kubectl exec mybench-pod -i -t -- /bin/bash
Throughout this tutorial enter the sysbench command as shown in these output blocks.
sysbench --threads=1 --cpu-max-prime=8005005 --verbosity=0 cpu run
Tasks: 203 total, 1 running, 202 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 1.7 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 1.7 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
31866 root 20 0 87.8m 5.1m 100.0 0.3 0:03.37 S sysbench --threads=1 --cpu-max-prime=8005005 --verbosity+
As expected, our Pod uses 1 full CPU at 100%. You can see that in the 4 individual core detail lines as well in the process detail line of our thread.
In the test below we let sysbench run 2 threads.
How to filter top command output:
sysbench --threads=2 --cpu-max-prime=8005005 --verbosity=0 cpu run
Tasks: 203 total, 1 running, 202 sleeping, 0 stopped, 0 zombie
%Cpu0 : 51.4 us, 0.0 sy, 0.0 ni, 48.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 50.0 us, 0.0 sy, 0.0 ni, 50.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 1.4 us, 0.0 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 1.4 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
32589 root 20 0 87.9m 5.1m 102.7 0.3 0:03.61 S sysbench --threads=2 --cpu-max-prime=8005005 --verbosity+
As expected, our Pod uses 1 full CPU overall at 100%.
You can see that in the 4 individual core detail lines that each thread runs on its own CPU - each using 50% CPU - to keep total usage at 100%
You will see that more clearly in tests below using 3 and 4 threads.
Test using 3 threads:
sysbench --threads=3 --cpu-max-prime=8005005 --verbosity=0 cpu run
Tasks: 203 total, 1 running, 202 sleeping, 0 stopped, 0 zombie
%Cpu0 : 32.5 us, 0.9 sy, 0.0 ni, 66.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 32.2 us, 0.0 sy, 0.0 ni, 67.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 32.2 us, 0.9 sy, 0.0 ni, 67.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.9 us, 0.9 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
335 root 20 0 88.0m 5.0m 97.4 0.3 0:08.41 S sysbench --threads=3 --cpu-max-prime=8005005 --verbosity+
As expected, our Pod still uses 1 full CPU overall at 100%.
You can see that in the 4 individual core detail lines that each of the 3 threads run on its own CPU - each using 33% CPU - to keep total usage at 100%
sysbench --threads=4 --cpu-max-prime=8005005 --verbosity=0 cpu run
Tasks: 203 total, 1 running, 202 sleeping, 0 stopped, 0 zombie
%Cpu0 : 25.5 us, 0.9 sy, 0.0 ni, 73.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 26.1 us, 0.9 sy, 0.0 ni, 72.1 id, 0.0 wa, 0.0 hi, 0.9 si, 0.0 st
%Cpu2 : 26.6 us, 0.9 sy, 0.0 ni, 72.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 26.4 us, 0.9 sy, 0.0 ni, 71.8 id, 0.0 wa, 0.0 hi, 0.9 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
678 root 20 0 88.0m 5.1m 100.9 0.3 0:05.22 S sysbench --threads=4 --cpu-max-prime=8005005 --verbosity+
As expected, our Pod still uses 1 full CPU overall at 100%.
You can see that in the 4 individual core detail lines that each of the 4 threads run on its own CPU - each using 25% CPU - to keep total usage at 100%
Basic demo completed, delete Pod.
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "mybench-pod" force deleted
I use --force --grace-period=0 to delete Pod immediately. Do not use in production. By default Pods get 30 seconds to do their shutdown routines ( after receiving delete command ).
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
resources:
limits:
cpu: "500m"
requests:
cpu: 500m
restartPolicy: Never
Note CPU limit is 500m = half a CPU.
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Exec into Pod:
kubectl exec mybench-pod -i -t -- /bin/bash
Run benchmark.
sysbench --threads=1 --cpu-max-prime=8005005 --verbosity=0 cpu run
Tasks: 208 total, 1 running, 207 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.1 us, 1.1 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 38.6 us, 1.1 sy, 0.0 ni, 60.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 11.5 us, 0.0 sy, 0.0 ni, 88.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 1.1 us, 0.0 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
4210 root 20 0 87.8m 5.1m 50.0 0.3 0:02.55 S sysbench --threads=1 --cpu-max-prime=8005005 --verbosity+
Pod got limited to half a CPU - works as expected.
Demo completed, delete Pod.
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "mybench-pod" force deleted
Same Pod as before, only using 0.5 to define half a CPU.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
resources:
limits:
cpu: .5
requests:
cpu: .5
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Enter Pod:
kubectl exec mybench-pod -i -t -- /bin/bash
Run sysbench:
sysbench --threads=1 --cpu-max-prime=8005005 --verbosity=0 cpu run
Tasks: 208 total, 1 running, 207 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.5 us, 1.5 sy, 0.0 ni, 96.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 1.6 sy, 0.0 ni, 98.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 1.6 sy, 0.0 ni, 98.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 52.3 us, 0.0 sy, 0.0 ni, 47.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
5639 root 20 0 87.8m 5.1m 50.8 0.3 0:01.49 S sysbench --threads=1 --cpu-max-prime=8005005 --verbosity+
Identical to previous test ( proving 500m = half a CPU )
Investigate how Kubernetes understands our Pod YAML spec.
( Output truncated - leaving only the good stuff. Rest of tutorial always only show relevant describe output )
kubectl describe pod/mybench-pod
Name: mybench-pod
Containers:
mybench-container:
State: Running
Started: Fri, 11 Jan 2019 09:17:09 +0200
Ready: True
Limits:
cpu: 500m
Requests:
cpu: 500m
Shows Millicores values, not 0.5
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "mybench-pod" force deleted
Pod below has no reference to limits.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Enter Pod and run sysbench test.
kubectl exec mybench-pod -i -t -- /bin/bash
sysbench --threads=4 --cpu-max-prime=8005005 --verbosity=0 cpu run
%Cpu0 : 95.3 us, 4.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
%Cpu1 : 97.8 us, 1.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
%Cpu2 : 96.8 us, 2.9 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
%Cpu3 : 97.8 us, 1.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
9142 root 20 0 88.0m 5.1m 392.4 0.3 0:33.63 S sysbench --threads=4 --cpu-max-prime=8005005 --verbosity+
Reckless Pod hogs all 4 cores at 100%. This is why you need CPU limits.
Delete this Pod:
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "mybench-pod" force deleted
We need CPU default limits that automatically limits ALL Pods on a node.
LimitRanges fits the bill.
LimitRange below adds default limits ALL Pods that do not declare their own limits.
nano mycpu-limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: mycpu-limit-range
spec:
limits:
- default:
cpu: 0.75
defaultRequest:
cpu: 0.25
type: Container
Create the LimitRange.
kubectl create -f mycpu-limit-range.yaml
limitrange/mycpu-limit-range created
We reuse the Pod spec from previous example : no limits in its spec.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Describe how Kubernetes executed our desired Pod spec.
kubectl describe pod/mybench-pod
Name: mybench-pod
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container mybench-container; cpu limit for container mybench-container
Containers:
mybench-container:
State: Running
Started: Fri, 11 Jan 2019 09:48:24 +0200
Ready: True
Limits:
cpu: 750m
Requests:
cpu: 250m
Note the annotation :
LimitRanger automatically added a cpu limit for container mybench-container
The limits listed agree with the LimitRange spec.
Best Practice Have LimitRanges at your development shop.
Delete this specific LimitRange.
kubectl delete limits/mycpu-limit-range
limitrange "mycpu-limit-range" deleted
Previously we used LimitRanges defaults.
LimitRanges also define min and max limits. See spec below:
3 examples of new Pod creations follow so you can see how this works. ( My tutorials teach via examples rather than long text descriptions with few demos. )
nano mycpu-limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: mycpu-limit-range
spec:
limits:
- default:
cpu: 0.75
defaultRequest:
cpu: 0.25
max:
cpu: "2000m"
min:
cpu: "200m"
type: Container
Create our LimitRange
kubectl create -f mycpu-limit-range.yaml
Define our Pod.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
resources:
limits:
cpu: "1000m"
requests:
cpu: 300m
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
This Pod does not need defaults - it declares its own.
This Pod spec limit fall within the allowed min-max LimitRange defined above . Pod create command went smoothly - nothing else to show here.
Describe Pod shows Pod created exactly as we specified.
kubectl describe pod/mybench-pod
Name: mybench-pod
Containers:
mybench-container:
State: Running
Ready: True
Limits:
cpu: 1
Requests:
cpu: 300m
Edit Pod spec.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
resources:
limits:
cpu: "3000m"
requests:
cpu: 100m
restartPolicy: Never
Note our CPU request is below LimitRange min
Note our CPU limit is above LimitRange max
Attempt to create the Pod:
kubectl create -f myBench-Pod.yaml
Error from server (Forbidden): error when creating "myBench-Pod.yaml": pods "mybench-pod" is forbidden: [minimum cpu usage per Container is 200m, but request is 100m., maximum cpu usage per Container is 2, but limit is 3.]
Error is as expected.
Modify Pod spec to be within limits.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The CPU Bench Pod is Running ; sleep 3600']
resources:
limits:
cpu: "1500m"
requests:
cpu: 250m
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Pod create worked. It falls within min and max limits.
Delete Pod and LimitRange .
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "mybench-pod" force deleted
and ...
kubectl delete limits/mycpu-limit-range
limitrange "mycpu-limit-range" deleted
LimitRanges live in Namespaces
If you are familiar with Namespaces all you need to know is that LimitRanges you create work in the Namespace you are currently in.
You should divide your development server into several Namespaces - per development team. Different teams may need different CPU and RAM limits. That is the purpose of Namespaces : separation of resources visibility based on your needs.
If you do not define any Namespaces and no LimitRanges your server is WIDE open to CPU and RAM abuse - deliberate and accidental.
For more information https://kubernetes.io/docs/tasks/administer-cluster/namespaces-walkthrough/
CPU defaults and limits are easy to understand, so let's put this knowledge to test. Create your own LimitRange and modify your own Pod with:
Enter Pod via kubectl exec and run sysbench. Exceed self-declared limits and LimitRange max to learn how all this interacts.
Here are several other exercises for you to try:
Hopefully by trying out these exercises, you can accurately predict any behavior for any Pod when given any LimitRange spec / definition.
2,599 posts | 762 followers
FollowAlibaba Clouder - May 21, 2019
Alibaba Container Service - September 14, 2024
Alibaba Cloud Blockchain Service Team - December 26, 2018
Alibaba Cloud Native Community - December 29, 2023
Alibaba Clouder - July 2, 2019
Alibaba Clouder - August 23, 2018
2,599 posts | 762 followers
FollowAlibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.
Learn MoreA secure image hosting platform providing containerized image lifecycle management
Learn MoreElastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn MoreMore Posts by Alibaba Clouder