Kernel crash caused by ftrace filter buffer overflow on Alibaba Cloud Linux 2 - Alibaba Cloud Linux

This topic describes how to resolve a kernel crash that occurs on Elastic Compute Service (ECS) instances running Alibaba Cloud Linux 2 when the ftrace filter feature is enabled.

Applicable scope

ECS

Problem description

An ECS instance running Alibaba Cloud Linux 2 crashes with a general protection fault. The crash occurs at kmem_cache_alloc+0x90/0x190. The following conditions apply:

Image: Alibaba Cloud Linux 2.1903 LTS 64-bit
Kernel: kernel-4.19.91-23.al7 or earlier
Feature: ftrace filter is enabled

The kernel log contains the following call stack:

[4017090.993301] general protection fault: 0000 [#1] SMP NOPTI
[4017090.999211] CPU: 69 PID: 24489 Comm: kubelet Kdump: loaded Tainted: G        W  OE     4.19.91-22.2.al7.x86_64 #1
[4017091.010213] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 1.0.PL.FC.P.028.01 09/22/2020
[4017091.020613] RIP: 0010:kmem_cache_alloc+0x90/0x190
[4017091.025821] Code: 03 05 34 be db 72 4d 8b 38 49 8b 40 10 4d 85 ff 0f 84 d5 00 00 00 48 85 c0 0f 84 cc 00 00 00 41 8b 46 20 48 8d 4a 01 4d 8b 06  8b 1c 07 4c 89 f8 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 41 8b 46
[4017091.045586] RSP: 0018:ffffb2216bebbe20 EFLAGS: 00010282
[4017091.051317] RAX: 0000000000000000 RBX: 00000000006000c0 RCX: 000000002477d008
[4017091.059192] RDX: 000000002477d007 RSI: 00000000006000c0 RDI: ffff95303f004600
[4017091.067065] RBP: 00000000006000c0 R08: 0000000000027170 R09: ffff950f20dadc01
[4017091.074937] R10: ffff950f20dadc00 R11: 0000000000000000 R12: ffffffff8d2d239b
[4017091.082807] R13: ffff95303f004600 R14: ffff95303f004600 R15: e58cbcefaf98e684
[4017091.090678] FS:  00007f0205ffb700(0000) GS:ffff953040740000(0000) knlGS:0000000000000000
[4017091.099512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4017091.105761] CR2: 00007fd754130000 CR3: 0000005e9e840004 CR4: 00000000003606e0
[4017091.113637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4017091.121525] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[4017091.129399] Call Trace:
[4017091.132356]  do_epoll_ctl+0x5cb/0xf90
[4017091.136520]  ? do_sys_openat2+0x222/0x260
[4017091.141031]  __se_sys_epoll_ctl+0x4a/0x60
[4017091.145545]  do_syscall_64+0x5b/0x1b0
[4017091.149713]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[4017091.155270] RIP: 0033:0x221fec8
[4017091.158910] Code: 10 c3 8b 7c 24 08 b8 23 01 00 00 0f 05 89 44 24 10 c3 8b 7c 24 08 8b 74 24 0c 8b 54 24 10 4c 8b 54 24 18 b8 e9 00 00 00 0f 05  44 24 20 c3 cc cc cc 8b 7c 24 08 48 8b 74 24 10 8b 54 24 18 44
[4017091.178672] RSP: 002b:000000c006d57e48 EFLAGS: 00000206 ORIG_RAX: 00000000000000e9
[4017091.186978] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 000000000221fec8
[4017091.194857] RDX: 00000000000000b3 RSI: 0000000000000001 RDI: 0000000000000004
[4017091.202733] RBP: 000000c006d57e80 R08: 00007f042a5ea329 R09: 0000000000203000
[4017091.210602] R10: 000000c006d57e74 R11: 0000000000000206 R12: 00000000000000f2
[4017091.218473] R13: 0000000000000000 R14: 000000000684f29c R15: 0000000000000000
[4017091.226351] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache veth binfmt_misc tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag iptable_mangle sch_htb xt_set ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_bitmap_port ip_set dummy iptable_raw xt_CT xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat bpfilter overlay intel_rapl_msr intel_rapl_common iosf_mbi isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crct10dif_pclmul i2c_algo_bit crc32_pclmul ttm ghash_clmulni_intel pcbc drm_kms_helper syscopyarea sysfillrect aesni_intel sysimgblt fb_sys_fops crypto_simd ip_vs_rr
[4017091.300367]  cryptd glue_helper drm lpc_ich ip_vs_sh i2c_i801 ipmi_si pcspkr mei_me ipmi_devintf ip_vs_wrr mousedev ioatdma i2c_core mfd_core mei ipmi_msghandler dca wmi pcc_cpufreq acpi_pad acpi_power_meter ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter bridge stp llc auth_rpcgss sunrpc ip_tables crc32c_intel ahci libahci libata [last unloaded: AliSecNetFlt64]
[4017091.336060] ---[ end trace b5d2ca91a9c736db ]---
[4017091.459188] RIP: 0010:kmem_cache_alloc+0x90/0x190

Cause

The kernel incorrectly calculates the length of struct ring_buffer_event when the ftrace filter feature is enabled. This causes a buffer overflow, which leads to memory corruption and a kernel crash.

Before you begin

Important

Back up data or create a snapshot before performing any high-risk operation. Make sure that disaster recovery and fault tolerance mechanisms are in place.

Log on to the ECS instance. For more information, see Connection method overview.

Solution

The fix depends on your kernel version. Run the following command to check it:

uname -r

Kernel version earlier than 4.19.91-19.1.al7.x86_64

Update the kernel and restart the instance.

Update the kernel.
```
   yum update kernel
```
Restart the instance.
```
   reboot
```
After the instance restarts, verify the kernel version. The kernel version must be later than 4.19.91-23.al7.
```
   uname -r
```

If the kernel crash persists after the update, install the hotfix as described in the next section.

Kernel version from 4.19.91-19.1.al7.x86_64 to 4.19.91-23.al7.x86_64

Install the kernel hotfix.

yum install -y kernel-hotfix-5692820-`uname -r | awk -F"-" '{print $NF}'`

After the hotfix is installed, verify that the kernel crash no longer occurs.