Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Higress: A Best Practice for Knative Ingress Gateway

Higress: A Best Practice for Knative Ingress Gateway

This article describes the implementation of the network layer capabilities of Knative.

By Weiji Zhao (Zhaowei)

In traditional application development, it is necessary to manage the underlying infrastructure, server, and network configuration. However, with the rise of cloud-native serverless architecture, these infrastructure details are abstracted and automated. Developers can now shift their focus from server configuration, expansion, monitoring, and maintenance to the development of business logic and application functionality. Serverless architecture brings benefits such as high efficiency, elastic and serverless management, pay-as-you-go service, fast deployment and iteration, and scalability. It reduces the complexity of development and operations, improves development efficiency, and enhances application quality.

Knative Serving is a Kubernetes-based Serverless open source platform for building and managing modern, scalable, traffic-driven, and serverless applications. It provides various features to support the deployment of serverless services, including automatic scaling of pods based on HTTP traffic triggers, service versioning, automatic traffic management, and fault recovery.

1
Source: https://knative.dev/docs/serving/architecture/

The overall architecture of Knative is shown in the figure above. Components such as Controller and DomainMapping are responsible for managing the lifecycle of KnativeCRD resources. Elasticity is provided by core components such as Activator, Autoscaler, and Queue-Proxy. Network and routing capabilities are delivered by various types of Ingress gateways.

This article focuses on the implementation of the network layer capabilities of Knative. These capabilities rely on Knative Ingress CRD and other network layer components. Knative utilizes components like Contour, Istio, and Kourier as its network layer components, offering limited network capabilities such as basic routing, authentication, and TLS to meet fundamental routing and security requirements. Higress is a cloud-native gateway that seamlessly integrates security, traffic, and microservices. Leveraging Higress as the traffic ingress for Knative services enhances traffic governance, security protection, observability, and scalability capabilities.

2

How Does the Knative Network Layer Work?

Next, let's take Net-Istio as an example to introduce the process of using Knative Serving to implement service publishing through the network layer. Net-istio network layer separates the data plane from the control plane. The data plane uses Envoy to process network traffic. The control plane manages and configures the data plane and supports dynamic configuration and management of gateways.

3

When a KService is deployed, the Knative Serving Controller parses the routes in the KService and generates the corresponding KIngress resource. KIngress is a CRD (Custom Resources Definition) of Knative. Its resources contain all the information required for service disclosure. Example:

apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
  annotations:
    networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev # Specify istio as the network layer.
  name: hello
  namespace: default
spec:
  httpOption: Enable # Specify whether to enable HTTPS redirection.
  rules:
  - hosts:
    - hello.default.example.com
    http:
      paths:
      - splits: #Split specifies that the traffic is distributed to different service revisions according to the percentage.
        - appendHeaders:
            Knative-Serving-Namespace: default
            Knative-Serving-Revision: hello-00001
          percent: 100
          serviceName: hello-00001
          serviceNamespace: default
          servicePort: 80
visibility: ExternalIP  # Specify whether the service is accessible only within the cluster or disclosed externally. 
tls:  # Specify the certificate and key used to disclose the Kservice over
  - hosts:
    - hello.default.example.com
    secretName: route-ecbe0df2-101a-4aa4-8cf9-f2e98773fcdf
    secretNamespace: default

In this example, Istio is used as the network layer. The Net-Istio component monitors the Kingress resources in the cluster. Each time the Kingress is created, deleted, or modified, Net-Istio parses the Kingress resources and converts the route configurations of the Kingress resources to resources such as the virtual services and gateways of Istio. Istio monitors VirtualService and sends route configurations to Envoy on the data plane through xDS. This way, the routing configurations are transmitted.

After receiving the EDS data of the route destination cluster, the Envoy forwards the request to the activator pod or revision pod based on the IP address of the endpoint associated with the service. When the Knative Controller is in the cold start or scale-down state, it sets the endpoint associated with the service as the activator pod IP. When it is in a steady state, the endpoint associated with the service is set as the revision pod IP deployed by the user.

After the working principle of the Knative network layer is clarified, we can find that the actual functions of the control plane in the Knative network layer are as follows:

  1. Monitor and obtain KIngress resources generated by Knative Services.
  2. Map the KIngress resource to the Envoy configuration information on the data plane.
  3. Disclose the configuration information to the Envoys that it manages.

Different control planes of the Knative network layer convert and deliver route configurations through their mechanisms to implement basic routing capabilities. However, in more complex application scenarios, existing network layers may fail to fully meet requirements such as security, authentication, observability, fine-grained traffic governance, and scalability. One solution is to continue to integrate components such as security gateways, traffic gateways, and observability platforms on top of the Knative network layer. However, the design of multi-layer gateways undoubtedly consumes more operating resources and increases O&M costs.

Adapt Higress to Knative Serving

To expand Knative's adaptability to various scenarios, designing multi-layer gateways is not the best choice. As a next-generation cloud-native gateway that integrates security, traffic, and microservices, Higress provides better traffic governance, security protection, observability, and scalability and ensures better stability and security when used as the traffic ingress of Knative services. Higress can be adapted to Knative Serving in two ways and has enhanced control plane capabilities.

Solution 1: Higress + KIngress

4

The principle of this solution is similar to other Knative network layers. It involves monitoring and updating KIngress, converting route configurations, and pushing configurations on the data plane. Higress is compatible with all features of the Knative Serving network layer, such as traffic division between different service revisions, TLS, timeout, retry, traffic marking, and automatic endpoint discovery. This ensures that Knative services can easily utilize Higress as their traffic ingress.

It is worth noting that in this solution, Higress does not rely on other custom resources. It only depends on KIngress resources managed by Knative Serving and common resources of Kubernetes clusters, such as endpoints and secrets, providing a more native adaptation to Knative.

Thanks to the integration with the microservice technology stack and the WASM extension mechanism, Higress can offer more powerful capabilities for Knative services, including authentication, throttling, WAF protection, and observability. Serverless service developers can shift their focus from implementing basic capabilities to developing their business logic. Higress has launched a plugin market to meet various requirements, including basic security, throttling, and authentication. It also provides custom plugin interfaces to help users better adapt to their Serverless applications.

Solution 2: Higress + Istio CRD

Higress can use its compatibility with Istio CRD to replace Istio as the control plane of Knative gateways. The following figure shows the implementation of solution 2. It is easy to see that this solution is based on the Higress' compatibility with Istio CRD. The transformation from route configuration to Istio CRD is completed through net-istio-controller. Higress Controller parses Istio CRD resources and delivers the configured data plane to complete the delivery of route configuration.

5

Solution Comparison

Both solutions can implement the capabilities of the Knative network layer and are compatible with most of the capabilities of Higress in terms of traffic governance, security protection, observability, and scalability. Istio is an excellent service mesh solution. However, if service mesh is not required in application scenarios, Istio CRD will bring additional complexity and resource consumption to the cluster. The Higress + KIngress solution is not dependent on other custom resources and is adapted to Knative in a more native manner without sacrificing capabilities.

The Practice of Connecting Higress to Knative Services

Prerequisites

  1. kubectl [1], Helm [2], and Knative CLI (kn) [3] are installed.
  2. A Kubernetes cluster whose version is Kubernetes v1.24 or later is created. For the convenience of demonstration, I'll conduct practice on the local Kubernetes cluster in this article.
  3. Install the Knative CRD and Knative Serving components. For more information, see Knative Installation Guide [4].
  4. Install Higress [5].

Note: Knative CRD is checked when the Higress Controller is deployed. Make sure that the Knative CRD is installed before you install Higress.

Solution 1: Higress + KIngress

Configure adaptation items between Knative and Higress.

1.  Configure Knative to use Higress as the Ingress:

kubectl patch configmap/config-network \
  --namespace knative-serving \
    --type merge \
      --patch '{"data":{"ingress-class":"higress"}}'

2.  The default routing policy of Knative is based on the domain name. You need to set the Knative domain name:

kubectl edit configmap config-domain -n knative-serving

Modify the default domain name by editing the data item in the ConfigMap. In this practice, change the default domain name to example.com. After the modification takes effect, all Knative services and routes automatically update their domain names.

apiVersion: v1
data:
  example.com: ""
kind: ConfigMap

3.  Confirm the RBAC permission of Higress on Knative resources:

kubectl get clusterrole higress-controller-higress-system -o yaml

The RBAC permission list should contain the following fields. If not, run the kubectl edit command to manually add them:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
rule:
 ...
- apiGroups:
  - networking.internal.knative.dev
  resources:
  - ingresses
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - networking.internal.knative.dev
  resources:
  - ingresses/status
  verbs:
  - get
  - patch
  - update

After you complete the preceding configurations, you can use Higress to seamlessly connect to the Knative service.

Function verification of the compatibility of Higress with the Knative network layer

1.  Deploy the Knative service

kn service create hello \
--image ghcr.io/knative/helloworld-go:latest \
--port 8080 \
--env TARGET=World
#Expected Output
Service hello created to latest revision 'hello-00001' is available at URL: 
http://hello.default.${LOADBALANCER_IP}.example.com
# ${LOADBALANCER_IP} is a Higress Gateway IP. If you deploy a local Kubernetes cluster, this parameter is empty.

2.  Verify Knative AutoScaling

Send a request to the Hello service:

# When the Knative service is deployed locally, the LOADBALANCER_IP is 127.0.0.1, which is simulated by using No DNS.
curl -H "host: hello.default.example.com" http://${LOADBALANCER_IP}
#Expected Output
Hello World!

Run the following command to observe the AutoScaling process:

kubectl get pod -l serving.knative.dev/service=hello -w

You can observe that the scale-up process is triggered when requests are forwarded to the Hello service. If no requests are sent for a period of time, the number of pods is automatically scaled down to 0:

NAME                                      READY   STATUS              RESTARTS   AGE
hello-00001-deployment-689c99c59b-6f5fw   0/2     Pending             0          0s
hello-00001-deployment-689c99c59b-6f5fw   0/2     ContainerCreating   0          0s
hello-00001-deployment-689c99c59b-6f5fw   1/2     Running             0          1s
hello-00001-deployment-689c99c59b-6f5fw   2/2     Running             0          1s
hello-00001-deployment-689c99c59b-6f5fw   2/2     Terminating         0          62s
hello-00001-deployment-689c99c59b-6f5fw   1/2     Terminating         0          91s
hello-00001-deployment-689c99c59b-6f5fw   0/2     Terminating         0          92s

3.  Verify Knative Traffic Splitting

The Traffic Splitting feature defined by KIngress manages traffic division rules among different Knative service revisions. This feature can be used for blue-green release and canary deployment of Knative services. A revision is an instant snapshot of the application code and configuration. Each time you change the configuration of the Knative service, a new revision is created. When a new revision is released, Higress divides traffic among different versions of the Knative services based on the routing rules in KIngress.

Create a new revision of the Hello service:

kn service update hello --env TARGET=Knative
#Expected Output   
Service 'hello' created to latest revision 'hello-00002' is available at URL:
http://hello.default.${LOADBALANCER_IP}.example.com

Set the traffic percentage for different revisions, where the traffic percentage for hello-00001 is 50% and the traffic percentage for hello-00002 is 50%.

kn service update hello --traffic hello-00001=50  --traffic @latest=50

If you send multiple requests to the Hello service, you can observe that the traffic is divided into two different revisions, and the ratio meets the set traffic ratio:

#Expected Output   
Hello Knative!
Hello World!
Hello Knative!
Hello World!

Enhance Knative network layer capabilities by using the Higress plug-in

As mentioned above, Higress can provide more powerful capabilities for Knative services, such as authentication, throttling, WAF protection, and observability. In this section, we will use key-auth and key-rate-limit plug-ins in the Higress plug-in market to briefly demonstrate the authentication and throttling capabilities provided by Higress as the Knative network layer.

1.  Verify the throttling capability of Higress for the Knative service.

Set throttling rules and deploy the key-rate-limit plug-in. For more information, see Key-based Throttling [6].

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: key-rate-limit
  namespace: higress-system
spec:
  defaultConfig:
    _rules_:
# Rule 2: Match by domain name to take effect.
    - _match_domain_:
      - "hello.default.example.com"
      limit_by_header: x-ca-key
      limit_keys:
      - key: 123456
        query_per_minute: 2
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-rate-limit:1.0.0

The specific meaning of the above configuration is as follows: The rate of requests that contain the x-ca-key: 123456 request header is limited to three times per minute. Requests that exceed three times within one minute cannot access the Knative service. This configuration takes effect for the Knative service domain name hello.default.example.com. We verify it with the following requests:

curl -H "Host: hello.default.example.com" http://127.0.0.1 -H "x-ca-key: 123456" 
Hello World!    # When the Knative service is accessed, return 200.
curl -H "Host: hello.default.example.com" http://127.0.0.1 -H "x-ca-key: 123456" 
Hello Knative!  # When the Knative service is accessed, return 200.
curl -H "Host: hello.default.example.com" http://127.0.0.1 -H "x-ca-key: 123456" 
rate_limited   # Trigger throttling and return 429

2.  Verify the authentication capability of Higress for the Knative service.

Set authentication rules and deploy the key-auth plug-in. For more information, see Key-based Authentication [7].

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: key-auth
  namespace: higress-system
spec:
  defaultConfig:
    consumers:
    - credential: 2bda943c-ba2b-11ec-ba07-00163e1250b5
      name: consumer1
    - credential: c8c8e9ca-558e-4a2d-bb62-e700dcc40e35
      name: consumer2
    keys:
    - apikey
    in_query: true
    _rules_:
    - _match_domain_:
      - "hello.default.example.com"
      allow:
      - consumer2
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-auth:1.0.0

The specific meaning of the above configuration is as follows: In the user group consumer, only consumer2 has access permission. This configuration takes effect for the Knative service domain name "hello.default.example.com". We verify it with the following requests:

curl -H "Host: hello.default.example.com" http://127.0.0.1
# If no APIKey is provided in the request, return 401.
curl -H "Host: hello.default.example.com" http://127.0.0.1/?apikey=2bda943c-ba2b-11ec-ba07-00163e1250b5
# Consumer1 has no access permissions and returns 403
curl -H "Host: hello.default.example.com" http://127.0.0.1/?apikey=c8c8e9ca-558e-4a2d-bb62-e700dcc40e35
Hello World!  # Consumer2 has access permissions and returns 200

Solution 2: Higress + Istio CRD

In addition to the preceding solution, Higress can use Istio CRD to connect to Knative services based on its compatibility with Istio CRDs. The methods are as follows:

Step 1. Install Istio CRD

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm install istio-base istio/base -n istio-system --create-namespace

If you enable Istio CRD, you must update the deployment parameters of Higress:

helm upgrade higress -n higress-system --set global.enableIstioAPI=true higress.io/higress --reuse-values

Step 2. Install relevant network layer components

Run the following command to install the Knative Istio Controller (net-istio-controller).

kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.10.1/net-istio.yaml

Step 3. Configure Istio Config to point to Higress

The Knative community provides the configuration of using a non-default gateway. For more information, see Install Istio for Knative-Knative [8]. Procedure:

Modify the svc in the config-istio to higress-gateway:

kubectl edit configmap config-istio -n knative-serving

Add the following configuration in the code:

data:
  gateway.knative-serving.knative-ingress-gateway: higress-gateway.higress-system.svc.cluster.local
  local-gateway.knative-serving.knative-local-gateway: higress-gateway.higress-system.svc.cluster.local

Update the gateway instance knative-local-gateway and knative-ingress-gateway in the namespace Knative-serving:

kubectl edit gw knative-local-gateway -n knative-serving
kubectl edit gw knative-ingress-gateway -n knative-serving

Replace the label under the selector of the gateway instance Knative-local-gateway and Knative-ingress-gateway with the higress-gateway label. The default configurations of the higress-gateway are as follows:

spec:
  selector:
    app: higress-gateway
    higress: higress-system-higress-gateway

After completing the above configurations, you can also use Higress to implement the basic capabilities of the Knative network layer.

If you think that Higress[9] is helpful, please give us a star!

Reference

[1] kubectl
https://kubernetes.io/docs/tasks/tools/
[2] Helm
https://helm.sh/
[3] Knative CLI (kn)
https://knative.dev/docs/getting-started/quickstart-install/#install-the-knative-cli
[4] Knative Installation Guide
https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#install-the-knative-serving-component
[5] Higress
https://higress.io/en-us/docs/ops/deploy-by-helm/
[6] Key-based Throttling
https://higress.io/en-us/docs/plugins/key-rate-limit/
[7] Key-based Authentication
https://higress.io/en-us/docs/plugins/key-auth/
[8] Install Istio for Knative-Knative
https://knative.dev/docs/install/installing-istio/#configuring-the-installation
[9] github: Higress
https://github.com/alibaba/higress

Computing Open Source Kubernetes cloud native Knative Best Practice Ingress Gateway Higress
0 0 0
Share on

Read previous post:

Getting Started with OpenKruiseGame

Read next post:

RocketMQ 5.0 Architecture Analysis: How to Support Diversified Scenarios Based on Cloud-native Architecture

Alibaba Cloud Native Community

503 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

503 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All