Tool for Troubleshooting Problems in Serverless Scenarios

Background Introduction

A fully hosted Serverless computing platform can bring users less operation and maintenance costs, stronger stability, and faster flexibility. During the implementation of Serverless, a major challenge is how the Serverless platform provides developers with sufficient security. Enabling developers to use and trust Serverless without a burden is our goal.

The original intention of full hosting is to reduce the complexity of developers' use and operation and maintenance, but this to some extent reduces users' control over their own services. For example, in many scenarios, users may want to know how to master the actual operation of their own applications? How can I quickly identify whether the application has a problem or a cloud platform problem? If it's a cloud platform issue, how can we quickly restore service and stop losses in a timely manner? The root cause of these problems is that users cannot fully trust the cloud platform, which further hinders them from migrating applications and expanding business scenarios. Therefore, we are also thinking about how to break this distrust situation and allow users to have more control at the resource level, but also be able to stay away from the complex operation and maintenance at the resource level.

Against this background and demand, Alibaba Cloud's functional computing innovation has introduced a command line operation function for function instances in the Serverless scenario, supporting users to log in to the function instance from the console interface, or use tools to execute specified commands on the instance. This article describes the usage methods and scenarios of this function in detail.

Example Exec function positioning and usage

The instance command line operation function provides a consistent usage experience with K8S Pod Exec and Docker Container Exec, and supports executing specific commands in the real running environment of function instances.

At the same time, due to the extreme flexibility and pay-as-you-go characteristics of Serverless, there are some essential differences between the Exec function of instances in the Serverless scenario and K8S and Docker:

1. Exec operations can only be performed on surviving instances (including reserved resident instances and on-demand active instances). If the on-demand instance idle timeout is released, it cannot be executed again;

2. InstanceExec requests do not occupy the concurrency of the instance. Therefore, even if the instance concurrency of the function is set to 1, both InvokeFunction and InstanceExec operations can be executed simultaneously

3. An operation of InstanceExec is treated as an InvokeFunction request call. As long as the websocket connection established by the InstanceExec request is not disconnected from the function instance, the function instance will always be active, and the same billing rules apply to the Invoke Function. The user can set the idleTimeout parameter of InstantceExec to actively disconnect the client after a specified amount of idle time

The instance command line operation function supports logging in to the instance on the console, using the Serverless Dev tool to execute commands, or using the SDK call interface to execute commands.

Console Login Instance

On the console of the official function calculation website, on the Function Details - Monitoring Indicators - Instance Indicators page, you can log in to the instance on the far right.

Click "Login Instance" and the interface will be brought up to a terminal interface. You can immediately log in to the instance and execute the command to troubleshoot the problem.

On the Function Details - Monitoring Indicators - Instance Indicators page, click the instance ID to enter the instance details page of the function. There is a login instance button at the top right of the interface. Click to enter the instance.

SDK call

Taking the Golang SDK as an example, the invocation methods of other SDKs are mostly similar.

The SDK encapsulates the InstanceExec API. When calling the interface, it is necessary to use OnStdout and OnStderr to pass in two callback functions. The callback function defines the specific logic for processing the data returned by the Exec channel; At the same time, you can use the returned execConn to input a stdin message for transmission to the remote Exec channel.

Applicable scenarios

1. Troubleshoot online issues

In some daily scenarios, instance command line operations can bring more efficient and convenient troubleshooting methods that are more consistent with user habits.

The user Xiaowang is a Serverless Xiaobai user. After writing a program and deploying it to the function calculation, he found that the environment variables set in the function were not effective. If further troubleshooting is required, he needs to modify the code, print the log, redeploy, and view the log, using such a cumbersome troubleshooting method. Now with the help of example command line operations, Xiao Wang can directly use a command: s exec {instance_id} ENV to locate the problem in one step.

Instance command line operations provide a convenient login experience that can help users solve application problems in complex scenarios. In some cases, users have been unable to specifically locate problems through function logs and monitoring indicators, and need to use tools such as coredump, tcpdump, and jmap for in-depth troubleshooting.

For example, user Xiao Li found that his online program recently encountered some functional errors, and the error reported was that the connection to a remote service timed out. Xiao Li suspects that the network link between the function instance and the remote service is unstable. He wants to enter the instance and investigate and analyze the network situation between the instance and the remote service. He can follow these steps:

• After logging into the internal instance, first install the tcpdump tool. You need to execute two commands, apt-get update and apt-get install tcpdump:

After installation, execute the tcpdump command to capture packets from remote service IP requests, and save the capture results in the tcpdump.cap file:

After capturing packets, upload the tcpdump.cap file to your own OSS using the OSS command line tool ossutil64, and then download it locally. You can analyze it using the analysis tool wireshark.

2. Program performance optimization

Many times, developers need to use various profiling tools to analyze performance, resource usage, and other issues. For example, the utilization of resources such as the CPU and memory of the application instance does not meet expectations; Application performance is lower than expected, bottlenecks are found through the profiling tool, and so on. Through example command line operations, developers can easily run various profiling tools provided by the language and framework to optimize program performance and resource usage.

Taking Gaode autonomous travel running on functional computing as an example, its peak TPS will reach hundreds of thousands of levels. As a real-time online application, the service can accept request delays of tens of milliseconds. Considering the cost pressure, before going online, they expect to measure the maximum TPS that a single instance can withstand and the corresponding call delay, in order to evaluate the number of instances required.

However, in the pressure test, Gaud found that the average/long tail delay of a single instance did not meet expectations. When the TPS of a single instance reached 300 TPS, the request delay would rise linearly. They want to determine if there is a performance bottleneck in their application, or if there is a problem with the performance of the function calculation runtime? With the help of instance command line operations, they can log in to the inside of the instance. After in-depth analysis through profiling, they found performance issues, and finally optimized the program performance to meet the online standard.

Let's take the custom runtime as an example: The demo sample program is written using Golang and deployed to function calculations.

• After logging into the instance, download the Golang installation package:

• Unzip and install go:

• Execute the go tool approve command and generate an analysis file:

/root/pprof/pprof.bootstrap.samples.cpu.001.pb.gz

• Finally, use the OSS command line tool ossutil64 to run/ The ossutil64 cp command uploads the analysis file to your own OSS Bukcet, and then downloads it to the user's local location for visual analysis.

summary

The introduction of the instance command line function hopes to eliminate the "last mile" of users using Serverless and directly present the real function running environment to users. Thereafter, Serverless will no longer be a "black box", and users can more trust and rely on the Serverless platform to expand more business scenarios and scales.