Cause | Example scenario |
A cgroup does not have sufficient memory. | In a scenario in which OOM Killer is triggered as recorded in the following log, OOM Killer is triggered in the /mm_test cgroup to which the test process belongs:
[Wed Sep 8 18:01:32 2021] test invoked oom-killer: gfp_mask=0x240****(GFP_KERNEL), nodemask=0, order=0, oom_score_adj=0
[Wed Sep 8 18:01:32 2021] Task in /mm_test killed as a result of limit of /mm_test
[Wed Sep 8 18:01:32 2021] memory: usage 204800kB, limit 204800kB, failcnt 26
Cause: The memory usage of the /mm_test cgroup has reached the upper limit of 200 MB. |
A parent cgroup does not have sufficient memory. | In a scenario in which OOM Killer is triggered as recorded in the following log, the test process belongs to the /mm_test/2 cgroup but OOM Killer is triggered in the /mm_test cgroup:
[Fri Sep 10 16:15:14 2021] test invoked oom-killer: gfp_mask=0x240****(GFP_KERNEL), nodemask=0, order=0, oom_score_adj=0
[Fri Sep 10 16:15:14 2021] Task in /mm_test/2 killed as a result of limit of /mm_test
[Fri Sep 10 16:15:14 2021] memory: usage 204800kB, limit 204800kB, failcnt 1607
Cause: The memory usage of the /mm_test/2 cgroup has not reached the upper limit, but the memory usage of the /mm_test parent cgroup has reached the upper limit of 200 MB. |
An instance does not have sufficient memory. | In a scenario in which OOM Killer is triggered as recorded in the following log, limit of host indicates that the instance does not have sufficient memory. In the log data, the amount of free memory (the value of the free parameter) of Node 0 is smaller than the lower limit of free memory (the value of the low parameter).
[Sat Sep 11 12:24:42 2021] test invoked oom-killer: gfp_mask=0x62****(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0,
[Sat Sep 11 12:24:42 2021] Task in /user.slice killed as a result of limit of host
[Sat Sep 11 12:24:42 2021] Node 0 DMA32 free:155160kB min:152412kB low:190512kB high:228612kB
[Sat Sep 11 12:24:42 2021] Node 0 Normal free:46592kB min:46712kB low:58388kB high:70064kB
Cause: The amount of free memory on the instance is smaller than the lower limit of free memory, and memory reclamation cannot resolve the issue of insufficient memory. |
A memory node does not have sufficient memory. | In a scenario in which OOM Killer is triggered as recorded in the following log, the log data provides the following information: limit of host indicates that a memory node does not have sufficient memory.
The instance has two memory nodes: Node 0 and Node 1. The amount of free memory (the value of the free parameter) of Node 1 is smaller than the lower limit of free memory (the value of the low parameter). The instance has a large amount of free memory (free:4111496 ).
[Sat Sep 11 09:46:24 2021] main invoked oom-killer: gfp_mask=0x62****(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
[Sat Sep 11 09:46:24 2021] main cpuset=mm_cpuset mems_allowed=1
[Sat Sep 11 09:46:24 2021] Task in / killed as a result of limit of host
[Sat Sep 11 09:46:24 2021] Mem-Info:
[Sat Sep 11 09:46:24 2021] active_anon:172 inactive_anon:4518735 isolated_anon:
free:4111496 free_pcp:1 free_cma:0
[Sat Sep 11 09:46:24 2021] Node 1 Normal free:43636kB min:45148kB low:441424kB high:837700kB
[Sat Sep 11 09:46:24 2021] Node 1 Normal: 856*4kB (UME) 375*8kB (UME) 183*16kB (UME) 184*32kB (UME) 87*64kB (ME) 45*128kB (UME) 16*256kB (UME) 5*512kB (UE) 14*1024kB (UME) 0 *2048kB 0*4096kB = 47560kB
[Sat Sep 11 09:46:24 2021] Node 0 hugepages_total=360 hugepages_free=360 hugepages_surp=0 hugepages_size=1048576kB
[Sat Sep 11 09:46:24 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sat Sep 11 09:46:24 2021] Node 1 hugepages_total=360 hugepages_free=360 hugepages_surp=0 hugepages_size=1048576kB
[Sat Sep 11 09:46:25 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Cause: In Non-Uniform Memory Access (NUMA) storage mode, the operating system may have multiple memory nodes. You can run the cat /proc/buddyinfo command to query resource information. If you use the cpuset.mems interface to limit a specific cgroup to using the memory of specific memory nodes, OOM Killer may still be triggered even when the instance has sufficient free memory. |
A buddy system does not have sufficient memory in the event of memory fragmentation. | In a scenario in which OOM Killer is triggered as recorded in the following log, the log data provides the following information: OOM Killer is triggered in the operating system during the order-3 allocation phase. The amount of free memory (the value of the free parameter) of Node 0 is larger than the lower limit of free memory (the value of the low parameter). The memory of the buddy system of Node 0 is 0 (0*32kB (M) ).
[Sat Sep 11 15:22:46 2021] insmod invoked oom-killer: gfp_mask=0x60****(GFP_KERNEL), nodemask=(null), order=3, oom_score_adj=0
[Sat Sep 11 15:22:46 2021] insmod cpuset=/ mems_allowed=0
[Sat Sep 11 15:22:46 2021] Task in /user.slice killed as a result of limit of host
[Sat Sep 11 15:22:46 2021] Node 0 Normal free:23500kB min:15892kB low:19864kB high:23836kB active_anon:308kB inactive_anon:194492kB active_file:384kB inactive_file:420kB unevi ctable:0kB writepending:464kB present:917504kB managed:852784kB mlocked:0kB kernel_stack:2928kB pagetables:9188kB bounce:0kB
[Sat Sep 11 15:22:46 2021] Node 0 Normal: 1325*4kB (UME) 966*8kB (UME) 675*16kB (UME) 0*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
Cause: If the buddy system does not have sufficient memory when the operating system allocates memory, the system triggers OOM Killer to free up memory and allocates the freed memory to the buddy system. Note The buddy system is a kernel memory management mechanism in Linux that mitigates memory fragmentation and efficiently allocates and frees up memory blocks of different sizes. |