OpenKruise v1.7: SidecarSet Supports Native Kubernetes Sidecar Containers

By Liheng

Preface

OpenKruise is an open-source cloud-native application automation management suite provided by Alibaba Cloud and is currently an incubating project hosted by the Cloud Native Computing Foundation (CNCF). It embraces containerization and cloud-native technologies that have been accumulated by Alibaba Group over the years. OpenKruise is a standard extension component based on Kubernetes that is widely used in Alibaba Group's production environment. It is also a technical concept and best practice that closely follows upstream community standards and adapts to large-scale Internet scenarios.

In August 2024, OpenKruise released its latest version, v1.7 (ChangeLog). This article provides an overview of the core features of the new release.

Upgrade the Kubernetes Dependency to 1.28

OpenKruise references the Kubernetes codebase as workload fields, such as PodTemplateSpec. Upgrading the Kubernetes dependency to 1.28 brings the following benefits:

SidecarSets, CloneSets, and Advanced StatefulSets can be configured with native Kubernetes Sidecar containers.
It lays the groundwork for supporting in-place VPA for resources in CloneSet and Advanced StatefulSet, which will be supported in future versions, maximizing the benefits from the Kubernetes community.
The updateStatefulSet method in the Advanced StatefulSet controller has been refactored based on the upstream community version, improving code readability.

🔔 Note: Upgrading the Kubernetes dependency to 1.28 does not mean that OpenKruise can only be deployed in Kubernetes clusters with a version of 1.28 or later. OpenKruise still supports deployment in Kubernetes clusters with a version of 1.18 or later. However, if you want to use the advanced Kubernetes features mentioned above, you will need to upgrade OpenKruise to version 1.7.

SidecarSet supports native Kubernetes Sidecar containers

Kubernetes, starting from version 1.28, supports Sidecar Containers through the initContainers[x].restartPolicy=Always approach. This approach offers the following advantages:

• The Sidecar container is guaranteed to be ready before the main business container starts. For example, a log collection container is ready, and even if the main business container crashes on startup, the logs can still be collected timely.

• For Job-type Pods, after the main container completes its execution, the Sidecar container will also exit automatically without blocking the completion of the Job.

Configuration Example:

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: test-sidecarset
spec:
  selector:
    matchLabels:
      app: sample
  updateStrategy:
    type: NotUpdate
  initContainers:
  - name: sidecar
    image: nginx:alpine
    restartPolicy: Always
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "sleep 10"]

🔔 Note: The current version only supports automatic injection of Sidecar containers and does not yet support in-place upgrade capabilities.

Additionally, if your Kubernetes version is 1.28 or earlier and you have the requirement for Sidecar containers to be ready before the main container or for Sidecar containers in Job-type Pods to exit promptly, you can use Job Sidecar Terminator and Container Launch Priority features provided by OpenKruise. You can achieve these effects by setting the environment variables KRUISE_TERMINATE_SIDECAR_WHEN_JOB_EXIT and KRUISE_CONTAINER_PRIORITY in the Sidecar container as follows:

apiVersion: v1
kind: Pod
spec:
  template:
    spec:
      containers:
      - name: sidecar
        env:
        - name: KRUISE_TERMINATE_SIDECAR_WHEN_JOB_EXIT
          value: "true"
        - name: KRUISE_CONTAINER_PRIORITY
          value: "1"
      - name: main
        image: centos:6.7

Advanced StatefulSets supports starting ordinals

By default, Pod ordinals start at 0. However, you can set a custom starting ordinal by configuring the .spec.ordinals.start field. To use this feature, you need to enable the FeatureGate StatefulSetStartOrdinal=true.

spec.ordinals.start: If the .spec.ordinals.start field is configured, Pods will be assigned ordinals ranging from .spec.ordinals.start to .spec.ordinals.start + .spec.replicas - 1. For example, if replicas=5 and ordinals.start=3, the Pod ordinals will be [3, 7].

apiVersion: apps.kruise.io/v1beta1
kind: StatefulSet
metadata:
  name: sample
spec:
  replicas: 5
  ordinals:
    start: 3
  serviceName: fake-service
  selector:
    matchLabels:
      app: sample
  template:
    metadata:
      labels:
        app: sample
    spec:
      containers:
      - name: main
        image: nginx:alpine

Image prefetch supports the aliyun-acr-credential-helper component

Starting from Kubernetes v1.20, the kubelet can dynamically retrieve credentials for a container image registry using exec plugins. For more information, see the community documentation. It is recommended to use this capability in the following scenarios:

• API calls to a cloud provider service are required to retrieve authentication information for a registry.

• Credentials have short expiration times and requesting new credentials frequently is required.

• Storing registry credentials on disk or in imagePullSecrets is not acceptable.

OpenKruise also supports a similar approach for image prefetch. The steps are as follows:

1. Install the cloud provider's credential provider plugin on the node, such as AWS.

2. The configuration file for the credential provider plugin is as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: credential-provider-config
  namespace: kruise-system
data:
  CredentialProviderPlugin.yaml: |
    apiVersion: kubelet.config.k8s.io/v1
    kind: CredentialProviderConfig
    providers:
    # The name of the providers must be the same as that of the aliyun-acr-credential-helper component.
    - name: ecr-credential-provider
      matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
      - "*.dkr.ecr.*.amazonaws.com.cn"
      - "*.dkr.ecr-fips.*.amazonaws.com"
      - "*.dkr.ecr.us-iso-east-1.c2s.ic.gov"
      - "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
      defaultCacheDuration: "12h"
      apiVersion: credentialprovider.kubelet.k8s.io/v1
      env:
      - name: AWS_PROFILE
        value: temp

3. The credential provider plugin needs authentication to request temporary image registry credentials from the cloud provider. The following is how AWS shares the credentials file:

The AWS Worker machine already contains the credentials file ($HOME/.aws/credentials)
Directly configure the $HOME/.aws directory on the kruise-daemon for authentication:

helm install kruise https://... --set installation.createNamespace=false --set daemon.credentialProvider.enable=true --set daemon.credentialProvider.hostPath=/etc/eks/image-credential-provider --set daemon.credentialProvider.configmap=credential-provider-config --set daemon.credentialProvider.awsCredentialsDir=/root/.aws

4. Create an ImagePullJob and use the above plugin for image registry authentication to complete the image prefetch.

Support third-party tools (Cert-Manager) to inject Kruise Webhook CA

OpenKruise requires a CA certificate to enable webhook authentication. By default, Kruise generates a self-signed CA certificate. Additionally, it is possible to inject and manage certificates through third-party tools, which can meet the strict and unified credential management requirements of some enterprises. Example:

1. Install a third-party CA management tool, such as Cert-Manager.

2. Create issuer and certificate resources:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kruise-webhook-certs
  # consistent with installation.namespace
  namespace: kruise-system
spec:
  # where to store the certificates
  # cert-manager would generate a secret kruise-system/kruise-webhook-certs with the certificates
  # DO NOT CHANGE THE SECRET NAME SINCE KRUISE READ CERTS FROM THIS SECRET
  secretName: kruise-webhook-certs
  dnsNames:
  - kruise-webhook-service.kruise-system.svc
  - localhost
  issuerRef:
    name: selfsigned-kruise
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-kruise
  namespace: kruise-system
spec:
  selfSigned: {}

3. OpenKruise needs to enable featureGates=EnableExternalCerts=true and inject the corresponding cert annotations into the webhookConfiguration and CRD.

helm install kruise https://... --set featureGates="EnableExternalCerts=true" --set-json externalCerts.annotations='{"cert-manager.io/inject-ca-from":"kruise-system/kruise-webhook-certs"}'

Structured logs

Logs are a crucial aspect of observability and an essential tool for debugging. However, OpenKruise logs are traditionally unstructured strings, making them difficult to parse automatically and challenging for any reliable subsequent processing, analysis, or querying.

Starting with OpenKruise 1.17, we have added support for structured logs, which natively support (key, value) pairs and object references. To maintain backward compatibility, structured logs will still be output as strings, where the string contains a representation of these "key"="value" pairs. You can also set helm install ... --set manager.loggingFormat=json to output the logs in JSON format. The following sample shows the InfoS call:

klog.V(3).InfoS("SidecarSet updated status success", "sidecarSet", klog.KObj(sidecarSet), "matchedPods", status.MatchedPods,
"updatedPods", status.UpdatedPods, "readyPods", status.ReadyPods, "updateReadyPods", status.UpdatedReadyPods)

You will get the following log output:

I0821 14:22:35.587919       1 sidecarset_processor.go:280] "SidecarSet updated status success" sidecarSet="test-sidecarset" matchedPods=1 updatedPods=1 readyPods=1 updateReadyPods=1

Or, by setting helm install ... --set manager.loggingFormat=json, the logs will be output in JSON format:

{
  "ts": 1724239224606.642,
  "caller": "sidecarset/sidecarset_processor.go:280",
  "msg": "SidecarSet updated status success",
  "v": 3,
  "sidecarSet": {
    "name": "test-sidecarset"
  },
  "matchedPods": 1,
  "updatedPods": 1,
  "readyPods": 0,
  "updateReadyPods": 0
}

Outlook

We currently have two future releases planned:

release 1.8: CloneSet/Advanced StatefulSet will support PVC and in-place resource configuration changes, and SidecarSet will support in-place upgrades for native Kubernetes Sidecar containers.
release 1.9: Kruise API will be upgraded to v1beta1, introducing Liveness Probe features and a minimal deployment solution for Kruise components.

Welcome to participate in the development and version planning of Kruise. Additionally, we highly encourage you to join our OpenKruise open-source community through Github, Slack, and other channels.

Community

OpenKruise v1.7: SidecarSet Supports Native Kubernetes Sidecar Containers

Preface

Upgrade the Kubernetes Dependency to 1.28

SidecarSet supports native Kubernetes Sidecar containers

Advanced StatefulSets supports starting ordinals

Image prefetch supports the aliyun-acr-credential-helper component

Support third-party tools (Cert-Manager) to inject Kruise Webhook CA

Structured logs

Outlook

Read previous post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Cloud-Native Applications Management Solution

Managed Service for Prometheus

Container Service for Kubernetes

ACK One