How do I resolve errors reported in continuous profiling logs? - Application Real-Time Monitoring Service

This topic provides answers to frequently asked questions (FAQ) about the continuous profiling feature of Application Real-Time Monitoring Service (ARMS) Application Monitoring.

Why are "parse lib sigsegv handler installed" logs printed?

Cause

These useless logs are printed by the ARMS agent only when the continuous profiling feature is enabled. This does not affect the running process of your application. ARMS will disable printing for these logs in future new versions.

What do I do if the "No access to perf events" error occurs due to the limitation of perf_event_open?

Issue

The CPU profiler in the Async-Profiler configurations uses the perf_event_open system call. However, the Linux kernel may use Secure Computing Mode (seccomp) to restrict system calls (syscalls).

Error message:

[ERROR] Failed to execute 'start,jfr=0,event=cpu,interval=11ms,alloc=512k,file=/tmp/cpc-async-profiler-7729534006755968198.jfr'
[ERROR] Failed to start Continuous Profile Collector
java.lang.RuntimeException: java.lang.IllegalStateException: No access to perf events. Try --fdtransfer or --all-user option or 'sysctl kernel.perf_event_paranoid=1'

Solutions

Docker environment: Run the following command to run the container. For more information about how to configure fine-grained system call control, see Docker documentation.
```
  docker run --security-opt seccomp=unconfined  XXX
```
Kubernetes environment: Set the privileged parameter to true. Keep the container type as Unconfined.
For more information about how to configure fine-grained system call control, see Kubernetes documentation.

What do I do if the "No AllocTracer symbols found .Are JDK debug symbols installed?" error is reported?

Issue

Async-Profiler depends on the debug symbols in the Java Development Kit (JDK) to profile memory. If no symbol exists in the JDK, the following error message may appear:

[ERROR] Failed to start Continuous Profile Collector
java.lang.RuntimeException: java.lang.IllegalStateException: No AllocTracer symbols found. Are JDK debug symbols installed?

Solution

If your Java process runs in a container environment, the above error may occur or no data is available due to Alpine base images. The continuous profiling feature mainly depends on the debug symbols in the JDK. However, Alpine base images remove the debug symbols from the JDK to control memory usage. This way, the continuous profiling feature cannot be used as expected. We recommend that you install debug symbols for the JDK or do not use Alpine base images. Note that you may be unable to install debug symbols for some JDK versions due to the lack of debug symbol packages.

If the application is deployed in a CentOS physical machine, perform the following steps to install the debug symbols:

Run the following command to check whether the debuginfo source is configured:
```
yum repolist all | grep -i debug
```
Check whether the following source configuration information is returned:
```
debuginfo/7/x86_64 CentOS-7 - debuginfo - mirrors.aliyun.com Enable: 8,760
```
If the debuginfo source is not configured, perform Step 2. Otherwise, perform Step 3.

Add the debuginfo configuration to the Yum configuration file.

vi /etc/yum.repos.d/CentOS-Base.repo
[debuginfo]
name=CentOS-$releasever - debuginfo - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos-debuginfo/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-Debug-7

Save the file and run the following command:

yum clean all
yum makecache
yum-config-manager --enable debuginfo

Install the debug symbols based on the JDK version.

JDK 8

yum list installed|grep openjdk    # Query the JDK version. 
java-1.8.0-openjdk.x86_64          1:1.8.0.332.b09-1.el7_9            @updates

debuginfo-install -y java-1.8.0-openjdk      # Install the debug symbols.

JDK 11

yum list installed|grep openjdk    # Query the JDK version. 
java-11-openjdk.x86_64             1:11.0.15.0.9-2.el7_9              @updates

debuginfo-install -y java-11-openjdk      # Install the debug symbols.

Check whether the debug symbols of the JDK have been installed.

gdb $JAVA_HOME/lib/server/libjvm.so -ex 'info address UseG1GC'
gdb /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so -ex 'info address UseG1GC'
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so...Reading symbols from /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so.debug...done.
done.

If the installation is successful, the following information is returned:

Symbol "UseG1GC" is static storage at address 0x107f993.

If the installation fails, the following information is returned:

No symbol "UseG1GC" in current context.

What do I do if the "perf_event mmap failed..." error is reported?

Issue

Generally, the error appears in the standard output (stdout) of the JVM. When the continuous profiling feature performs CPU hotspot sampling, native stacks (Linux kernel, JVM, C, and C++) and Java stacks are collected at the same time. To collect the native stacks, mmap needs to be performed on the perf_event FD of each thread in Java. The Linux kernel limits the total memory size of the mmap related to the perf_event process to 516 KB. If a large number of threads exist, the alert message perf_event mmap failed... is printed in the Java stdout. The error affects the native stacks that are not displayed in the flame graph, rather than the Java stacks or services. Generally, when you locate CPU hotspots, you only need to pay attention to the Java method stacks and ignore the alert.

Solution

To resolve the error, perform the following steps:

Run the following command on the host:
```
echo 1028 > /proc/sys/kernel/perf_event_mlock_kb
```
The default threshold value is 516, which can be increased gradually until no alert is triggered. We recommend that you specify the value based on the following formula: 8*N + 4, where N is a natural number. Sample values: 516 (512 + 4) and 1028 (1024 + 4).
Restart Docker to fix the error.

Why does the "other" item appear in the flame graph?

Issue

The "other" item appears in the flame graph, as shown in the following figure.

火焰图other选项

Cause

The "other" item in the flame graph is normal. If the flame graph consists of a large number of nodes, key information is hard to be extracted. ARMS converges nodes and puts the unimportant nodes to the "other" category.

Why does the. no_Java_frame item appear in the flame graph?

This is because Alpine base images remove the debug symbols from the JDK to control memory usage. The method stacks in the C++ threads, such as VM threads and JIT compilation threads, cannot recognize function names and display no_Java_frame instead. If these stacks account for a low proportion, you can ignore the problem. Otherwise, we recommend that you install debug symbols for the JDK or do not use Alpine base images. Note that you may be unable to install debug symbols for some JDK versions due to the lack of debug symbol packages.