How to Locate Bottlenecks During Performance Tests and Address Occasional Timeouts?

This article introduces Arthas, a Java diagnostic tool that simplifies troubleshooting. It also explains the various scenarios where Arthas helps in effective diagnosis.

By Qi Zhang (Sichu)

Are you still using the tedious traditional approach to examine each line of code for unearthing errors?

Fret not! As the saying goes, "Arthas keeps you from worrying about troubleshooting." Let's talk about Arthas, the magical Java diagnostics tool.

What Is Arthas?

Arthas is an open-source online diagnostics tool that uses command-line interaction mode and supports Web-based diagnostics. It also provides the comprehensive Tab auto-completion feature to further facilitate problem locating and diagnostics. Arthas is a Java diagnostics tool, which is used by 99% of Alibaba developers, has earned 20,000 stars on GitHub since it became open-source more than a year ago. Instead of direct downloading, I recommend developers to use Arthas in Cloud Toolkit, which is an integrated development environment (IDE) plug-in, to implement one-click remote diagnostics.

Thanks to its powerful and comprehensive features, Arthas can do more than you can imagine. Arthas has a quick and relevant resolution to all your worries

You want a global view for monitoring the running status of your system.
You are wondering what occupies the CPU whenever the CPU usage rises.
You want to figure out whether there is any deadlock or blocking in multiple threads that are running simultaneously.
It takes a long time to run an app and you are not sure which part is time-consuming. You want a solution to monitor it.
You do not know the specific JAR package from which a class was loaded. You want to figure out why your system throws various class-related exceptions.
You are not sure why the execution of your modified code failed. You cannot remember whether you have committed the changes. You are not sure if you are using the right branch.
A problem occurs and you cannot debug online. You are wondering whether you have to add logs to your app and publish it again.
You want a solution to monitor the real-time running status of your JVM.

For more information about the commands and features of Arthas, see its Official Documentation.

In the following sections, I will list several scenarios that I came across recently.

Scenario 1: Locating the Performance Bottleneck During a Stress Test

Usually, requests to the server are normal. However, during a stress test, when all CPUs on the server approach 100% utilization, the dependent services and databases did not run into a bottleneck. Why?

While running the jstack command, I just saw the stack at a certain point in time. As a result, locating the real problem becomes an issue.

How to view information about the current thread and its stack.

Use thread -n 3 -i 10000 to query the top 3 busiest threads within 10 seconds and print their stacks, which facilitates problem locating. The problem that I finally found was relatively simple: The location information, including class name, method name, and line number, is printed in the log.

Dynamically acquire code information including method name and line number. This is usually done by new Throwable() -> print Throwable stack -> extract the topmost business code in the stack -> split string to acquire information including class, method, and line number.

However, printing stacks may cause significant performance loss.

Scenario 2: Detecting Occasional Timeouts

There was a time when I always came across occasional timeouts. However, logs seemed all right, and so did EagleEye traces. No database operation or HSF call was particularly slow.

The elapsed time of the requests seemed quite normal according to statistics from the monitoring system. I could not find any abnormal RT.

I thought there might be a problem with logs, but there was no evidence to support it.

I ran the ttrace command to monitor the elapsed time of each step. In addition, used conditional expressions to print detailed logs when the elapsed time exceeds a certain time.

Further, I executed the command on a machine and waited. When the timeout reoccurs, the distribution of elapsed time could be captured.

Based on the results from Arthas, I located the problem: log printing. Once I changed synchronous logs to asynchronous logs, the problem was solved.

Scenario 3: Debug? What If It Is Generated with Dynamic Bytecode?

Once I came across a problem, where the output numbers during JSON serialization were not quoted. To resolve the issue, I tried to debug and examine my code in various ways and found that it was a serialization class that was generated by using ASM dynamic bytecode. Therefore, I completely gave up debugging as it wasn't effective in locating the problem.

Alternatively, it's easy to locate and troubleshoot the problem by decompiling the classes that are generated with dynamic bytecode by using the jad command of Arthas and other commands such as watch.

The jad command decompiles the source code of a given loaded class.

Also, use the mc(memory compiler) and redefine commands to update the code online. Go on, start exploring!

Do these capabilities make us omnipotent? No. Let's take a look at the following scenario.

Scenario 4: Some Tricks

In the troubleshooting process, I found that logs were output to the console, which caused significant performance loss. Was there any quick solution to solve the problem without publishing it?

Find the Corresponding Class

sc -d ch.qos.logback.core.ConsoleAppender
 
class-info       ch.qos.logback.core.ConsoleAppender
  code-source       /home/admin/.../lib/logback-core-1.2.3.jar
  name             ch.qos.logback.core.ConsoleAppender
  isInterface       false
  isAnnotation     false
  isEnum           false
  isAnonymousClass false
  isArray           false
  isLocalClass     false
  isMemberClass     false
  isPrimitive       false
  isSynthetic       false
  simple-name       ConsoleAppender
  modifier         public
  annotation
  interfaces
  super-class       +-ch.qos.logback.core.OutputStreamAppender
                      +-ch.qos.logback.core.UnsynchronizedAppenderBase
                        +-ch.qos.logback.core.spi.ContextAwareBase
                          +-java.lang.Object
  class-loader     +-com.taobao..LaunchedURLClassLoader@58dad04a
                      +-sun.misc.Launcher$AppClassLoader@18b4aac2
                        +-sun.misc.Launcher$ExtClassLoader@58ceff1
  classLoaderHash   5f205aa

Obtain the Class Property Information and Find the Appender List

ognl -c 5f205aa '@org.slf4j.LoggerFactory@getLogger("root").aai.appenderList'

Delete Standard Output Appender

1ognl -c 5f205aa '@org.slf4j.LoggerFactory@getLogger("root").aai.appenderList.remove(0)'

A Magical Tool: Flame Graphs

While troubleshooting performance problems, use flame graphs, a magical tool that clearly display the statistics of elapsed time associated with each method within a period of time.

Get Started with Arthas

Approach 1: Use Cloud Toolkit for One-click Remote Diagnostics with Arthas

Released by Alibaba Cloud, Cloud Toolkit is a free local IDE plug-in that helps developers to efficiently develop, test, diagnose, and deploy applications. Use the plug-in to deploy local applications with one click on any server or even to the cloud, such as Elastic Compute Service (ECS), Enterprise Distributed Application Service (EDAS), Alibaba Cloud Container Service for Kubernetes (ACK), Alibaba Cloud Container Registry (ACR), and Mini Program Cloud. Also, use its built-in tools including Arthas Diagnostics, Dubbo, Terminal, File Upload, Function Compute, and MySQL Executor. In addition to IntelliJ IDEA, the mainstream version, other versions such as Eclipse, Pycharm, and Maven are also available.

We recommend using the IDEA plug-in to download Cloud Toolkit and use Arthas.

Approach 2: Directly Download Arthas

Download Arthas by using the URL: https://github.com/alibaba/arthas

Community

How to Locate Bottlenecks During Performance Tests and Address Occasional Timeouts?

What Is Arthas?

Scenario 1: Locating the Performance Bottleneck During a Stress Test

Scenario 2: Detecting Occasional Timeouts

Scenario 3: Debug? What If It Is Generated with Dynamic Bytecode?

Scenario 4: Some Tricks

Find the Corresponding Class

Obtain the Class Property Information and Find the Appender List

Delete Standard Output Appender

A Magical Tool: Flame Graphs

Get Started with Arthas

Approach 1: Use Cloud Toolkit for One-click Remote Diagnostics with Arthas

Approach 2: Directly Download Arthas

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Mobile Testing

Container Service for Kubernetes

Penetration Test

Managed Service for Prometheus