Higress actual combat

Preface

On November 15, in the live broadcast of Higress Open Source Development History and Demo Demo Demo, we demonstrated how Higress's Wasm plug-in can be configured and effective for Ingress resources. This paper reviews the Demo on that day and explains the underlying principle and mechanism.

Before Demo runs in this article, Higress needs to be installed in the K8s cluster and the following quick start configuration takes effect:

https://github.com/alibaba/higress/releases/download/v0.5.2/quickstart.yaml

The function to be implemented in this Demo is a Mock response function, which needs to return HTTP responses according to the configured content. This article will be introduced in the following ways:

• Writing code: code logic analysis

• Effective plug-in: describes how to compile, package, deploy and take effect

• Test plug-in function: explain how the global granularity and routing/domain name granularity are effective

• Principle of plug-in effectiveness: review the overall process and explain the principle of plug-in effectiveness

• Three revolutionary features: introduce the changes brought by Wasm plug-in mechanism to the development of gateway plug-ins

Three functions can be seen in the above code:

• main: The plug-in defines the plug-in context through the main function, including the plug-in name, the function used to resolve the configuration, and the function used to process the request/response

• parseConfig: This function mounts the ParseConfigBy specified in SetCtx to the plug-in configuration resolution phase. The three parameters passed in are:

• json: The configuration of the incoming plug-in will be uniformly serialized into a json dictionary object, providing parseConfig for parsing

• config: parseConfig outputs the resolved plug-in configuration to this MyConfig object

• log: provides a log output interface

• onHttpRequestHeaders: proxywasm.SendHttpResponse called in the function is used to directly return HTTP responses. This function is mounted to the execution phase of the parsing request header through ProcessRequestHeadersBy specified in SetCtx. Other mounting methods include:

• ProcessRequestBodyBy: Attach to the execution phase of the resolution request body

• ProcessResponseHeadersBy: the execution phase of attaching to the construction response Header

• ProcessResponseBodyBy: the execution phase of attaching to the construction response body

The three parameters passed in are:

• ctx: used to obtain the request context, such as scheme/method/path. Through ctx, you can set a custom context, which can be accessed across execution phases

• config: provides parsed custom configurations

• log: provides a log output interface

The plug-in function implemented by 30 lines of code is relatively simple. Here are some examples of relatively complex functions:

https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions

Here are the detailed usage documents of the plug-in sdk:

https://higress.io/zh-cn/docs/user/wasm-go.html

This plug-in sdk is implemented based on the proxy wasm go sdk of the Tetrate community. If you pay attention to the lower level details, you can view:

https://github.com/tetratelabs/proxy-wasm-go-sdk

https://github.com/alibaba/higress/blob/main/plugins/wasm-go/pkg/wrapper

It can be seen that Higress's wasm go sdk encapsulates the details of plug-in context processing through the generic feature introduced in Go 1.18, thereby reducing the amount of code required for plug-in development. Developers only care about the logic of configuration resolution and request response processing.

Effective plug-in

After the code is written, there are three steps to make the plug-in logic effective:

1. Compile: compile the go code into a Wasm format file

2. Image push: package the Wasm file into a Docker image and push it to the image warehouse

3. Distribution configuration: create WasmPlugin resource on K8s

compile

Compile the Go file main.go above into plugin.wasm

tinygo build -o plugin.wasm -scheduler=none -target=wasi main.go

Image push

Write Dockerfile

FROM scratch

COPY plugin.wasm ./

Build and push Docker images (Higress's official image warehouse is used in this example)

Create this resource through kubectl

kubectl apply -f wasmplugin.yaml

Test plug-in functions

Take effect of this modified configuration:

kubectl apply -f wasmplugin.yaml

The results are as follows:

• The request/foo will return the HTTP reply "hello foo" (matching to the first rule)

• The request/bar will return the HTTP response "hello bar" (matching to the second rule)

• Request www.example.com will return the HTTP response "hello world" (matching the third rule)

• Request www.abc.com will return the HTTP response "hello hierarchy" (no matching rule, use global configuration)

Plug in validation principle

Here is a brief description of the plug-in's effective mechanism:

1. The user compiles the code into a wasm file

2. The user builds the wasm file into a docker image

3. The user pushes the Docker image to the image warehouse

4. User creates WasmPlugin resource

5. Changes from Istio watch to WasmPlugin resources

6. The xDS proxy process in the Higress Gateway obtains the configuration from Istio and finds the image address of the plug-in

7. The xDS proxy pulls images from the image warehouse

8. The xDS proxy extracts the wasm file from the image

9. The invite process in the Higress Gateway obtains the configuration from the xDS proxy and finds the local path of the wasm file

10. invoke Load wasm file from local file

Here, the enable obtains the configuration and loads the wasm file using the ECDS (Extension Config Discovery Service) mechanism. The update of the wasm file and direct hot loading will not cause any connection interruption, and the business traffic will be completely lossless.

Three revolutionary features

The Wasm plug-in mechanism above brings three revolutionary features to the development of gateway custom plug-ins.

Feature 1: plug-in life cycle and gateway decoupling

This feature mainly benefits from Istio's WasmPlugin mechanism design. You can communicate with K8s Nginx

You can see that Nginx Ingress loads the custom plug-in, and needs to mount the lua file into the pod, or load it when building the image. In this way, the life cycle of the plug-in is bound to the gateway. When the plug-in logic is updated, a new version needs to be released, and the gateway also needs to release a new version or redeploy.

Using the WasmPlugin mechanism, the plug-in needs to release a new version. You only need to build the image of the plug-in itself and distribute it to take effect, and you can manage the version of the plug-in based on the image tag. In this way, the plug-in changes do not need to redeploy the gateway, and the ECDS mechanism of Envoy is also completely lossless to traffic.

Feature 2: High performance multilingual support

Based on Wasm's ability, plug-ins can be written in multiple languages, which is more friendly to developers. Another way to implement the multilingual development plug-in is the external process/service plug-in based on RPC and gateway process communication. This mode will have additional IO overhead, and the additional process/service will also bring additional operation and maintenance complexity. At present, we are concerned about the performance of the Wasm plug-in. From our test data, the instruction execution performance is indeed different from that of the native C++language, but the performance is the same as Lua, and far better than that of the external plug-in.

Feature 3: safety sandbox

Envoy currently supports a variety of Wasm runtimes, such as V8, WAMR, and wasmtime. These runtimes all provide the security sandbox capability. That is, the logic of accessing null pointers and not capturing exceptions appears in the Wasm plug-in, and the host process of Envoy will not crash. In addition, you can configure to perform Fail Open processing after the plug-in logic is abnormal, skip the plug-in execution logic, and minimize the impact on the business.

Open source community

Thanks to the previous work of the Istio/Envoy community, which enables Higress to enable WasmPlugin for Ingress resources and enhances the customization and expansion capability of the Ingress Controller.

Special thanks to the proxy wasm go sdk implemented by the Tetrate community. Higress encapsulates the wasm go sdk on this basis, reducing the threshold for plug-in development.

Higress has done some Bugfix work on Istio/Envoy's Wasm capability, which has been incorporated into the upstream community. Some subsequent feature capabilities will continue to feed back the upstream community.