All Products
Search
Document Center

Simple Log Service:Create and use an event center

Last Updated:Aug 30, 2024

Events record the status changes of Kubernetes clusters. For example, an event can record a status change when you create, run, or delete a pod, or when a component exception occurs. K8s Event Center collects all events in Kubernetes clusters in real time. This allows you to perform various operations on the events. For example, you can store, query, analyze, and visualize event data and configure alerts for the events. This topic describes how to create and use an event center in K8s Event Center.

Kubernetes is designed based on the state machine. Events are generated due to transitions between different states. Typically, Normal events are generated when the state machine changes to expected states and Warning events are generated when the state machine changes to unexpected states.

ACK provides out-of-the-box event monitoring solutions that target containerization scenarios. ACK uses NPD and kube-eventer provided by NPD to monitor container events.

image
  • NPD is a tool to diagnose Kubernetes nodes. NPD detects node exceptions, generates node events, and works with kube-eventer to raise alerts upon these events and enable closed-loop management of alerts. NPD generates node events when the following exceptions are detected: Docker engine hangs, Linux kernel hangs, outbound traffic exceptions, and file descriptor exceptions. For more information, see NPD.

  • kube-eventer is an open source event emitter that is maintained by ACK. kube-eventer sends Kubernetes events to sinks such as DingTalk, Simple Log Service, and EventBridge. kube-eventer also provides filter conditions to filter different levels of events. You can use kube-eventer to collect events in real time, trigger alerts upon specific events, and asynchronously archive events. For more information, see kube-eventer.

  • NPD is a tool to diagnose Kubernetes nodes. NPD detects node exceptions, generates node events, and works with kube-eventer to raise alerts upon these events and enable closed-loop management of alerts. NPD generates node events when the following exceptions are detected: Docker engine hangs, Linux kernel hangs, outbound traffic exceptions, and file descriptor exceptions. For more information, see NPD.

  • kube-eventer is an open source event emitter that is maintained by ACK. kube-eventer sends Kubernetes events to sinks such as DingTalk, Simple Log Service, and EventBridge. kube-eventer also provides filter conditions to filter different levels of events. You can use kube-eventer to collect events in real time, trigger alerts upon specific events, and asynchronously archive events. For more information, see kube-eventer.

Prerequisites

A Kubernetes cluster is created, such as a Container Service for Kubernetes (ACK) cluster and an ACK Serverless cluster.

Billing

If the following conditions are met, you are not charged when you use an event center:

  • The data retention period of the Logstore that is associated with your event center is 90 days, which is the default value.

  • The amount of data that is written to your event center per day is less than 256 MB, which is equivalent to approximately 250,000 events.

Examples:

  • If the default data retention period is specified for the associated Logstore and 1,000 events are generated in your Kubernetes cluster per day, you can use your event center for free.

  • If you change the data retention period to 105 days and 1,000 events are generated in your Kubernetes cluster per day, you are charged for the data that is stored in the associated Logstore after the data is stored for 90 days. The fee is calculated based on the billable item of the storage space occupied by log data. For more information about the billable item, see Billable items of pay-by-feature.

Step 1: Deploy the kube-eventer and node-problem-detector components

ACK cluster

If you use an ACK cluster, you need to only deploy the ack-node-problem-detector component because the component is integrated with the features of the kube-eventer and node-problem-detector components. For more information, see Event monitoring. If you use an ACK Serverless cluster, you must deploy the kube-eventer component.

NPD works with third-party plug-ins to detect node exceptions and generate cluster events. A Kubernetes cluster also generates events when the status of the cluster changes. For example, when a pod is evicted or an image pull operation fails, a related event is generated. The Kubernetes event center of Simple Log Service collects, stores, and visualizes cluster events. It allows you to query and analyze these events, and configure alerts. You can sink cluster events to the Kubernetes event center of Simple Log Service by using the following methods.

  • If you have selected Install node-problem-detector and Create Event Center when you create the cluster, proceed to Step 2 to view the Kubernetes event center. For more information about how to install the ack-node-problem-detector component when you create a cluster, see Create an ACK managed cluster.

  • If you did not select Install node-problem-detector and Create Event Center when you create the cluster, perform the following steps to manually install it.

    1. Log on to the ACK console.

    2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Operations > Add-ons.

    3. On the Logging and Monitoring tab, find and install ack-node-problem-detector.

Other Kubernetes cluster

  1. Deploy the kube-eventer component.

    1. Install the kubectl tool. For more information, see Connect to an ACK cluster by using kubectl.

    2. Create a configuration file named eventer.yaml by using the following sample code:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          name: kube-eventer
        name: kube-eventer
        namespace: kube-system
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: kube-eventer
        template:
          metadata:
            labels:
              app: kube-eventer
            annotations:
              scheduler.alpha.kubernetes.io/critical-pod: ''
          spec:
            dnsPolicy: ClusterFirstWithHostNet
            serviceAccount: kube-eventer
            containers:
              - image: registry.cn-hangzhou.aliyuncs.com/acs/kube-eventer:v1.2.5-cc7ec54-aliyun
                name: kube-eventer
                command:
                  - "/kube-eventer"
                  - "--source=kubernetes:https://kubernetes.default"
                  ## .send to sls
                  ## --sink=sls:https://{endpoint}?project={project}&logStore=k8s-event&regionId={region-id}&internal=false&accessKeyId={accessKeyId}&accessKeySecret={accessKeySecret}
                  - --sink=sls:https://cn-beijing.log.aliyuncs.com?project=k8s-xxxx&logStore=k8s-event&regionId=cn-beijing&internal=false&accessKeyId=xxx&accessKeySecret=xxx
                env:
                  # If TZ is assigned, set the TZ value as the time zone
                  - name: TZ
                    value: "Asia/Shanghai"
                volumeMounts:
                  - name: localtime
                    mountPath: /etc/localtime
                    readOnly: true
                  - name: zoneinfo
                    mountPath: /usr/share/zoneinfo
                    readOnly: true
                resources:
                  requests:
                    cpu: 10m
                    memory: 50Mi
                  limits:
                    cpu: 500m
                    memory: 250Mi
            volumes:
              - name: localtime
                hostPath:
                  path: /etc/localtime
              - name: zoneinfo
                hostPath:
                  path: /usr/share/zoneinfo
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        name: kube-eventer
      rules:
        - apiGroups:
            - ""
          resources:
            - events
          verbs:
            - get
            - list
            - watch
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: kube-eventer
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: kube-eventer
      subjects:
        - kind: ServiceAccount
          name: kube-eventer
          namespace: kube-system
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: kube-eventer
        namespace: kube-system

      Parameter

      Type

      Required

      Description

      endpoint

      string

      Yes

      The Simple Log Service endpoint. For more information, see Endpoints.

      project

      string

      Yes

      The Simple Log Service project.

      logStore

      string

      Yes

      The Simple Log Service Logstore.

      internal

      string

      Yes for self-managed Kubernetes clusters

      If you use a self-managed Kubernetes cluster, set the value to false.

      regionId

      string

      Yes for self-managed Kubernetes clusters

      The ID of the region where the Simple Log Service project resides. For more information, see Endpoints.

      accessKeyId

      string

      Yes for self-managed Kubernetes clusters

      The AccessKey ID. We recommend that you use the AccessKey ID of a Resource Access Management (RAM) user. For more information, see AccessKey pair.

      accessKeySecret

      string

      Yes for self-managed Kubernetes clusters

      The AccessKey secret. We recommend that you use the AccessKey secret of a RAM user. For more information, see AccessKey pair.

    3. Run the following command to apply the eventer.yaml configuration file to your cluster:

      kubectl apply -f eventer.yaml

      Expected output:

      deployment.apps/kube-eventer created
      clusterrole.rbac.authorization.k8s.io/kube-eventer created
      clusterrolebinding.rbac.authorization.k8s.io/kube-eventer created
      serviceaccount/kube-eventer created
  2. Deploy the node-problem-detector component.

    For more information, visit GitHub.

Step 2: Create an event center

Note

After you create an event center, Simple Log Service automatically creates a Logstore named k8s-event in the specified project. Related dashboards are also created for the event center.

  1. Log on to the Simple Log Service console.

  2. In the Log Application section, click the Intelligent O&M tab. Then, click K8s Event Center.

  3. On the Event Center Management page, click Add in the upper-right corner.

  4. In the Create Event Center panel, configure the parameters and click Next.

    • If you select Select Existing Project, you can select an existing project from the Project drop-down list to manage the resources of your event center. The resources include a Logstore and the related dashboards.

    • If you select Select Kubernetes Cluster from Container Service for Kubernetes, you can select an existing Kubernetes cluster from the K8s Cluster drop-down list. If you use this method to create an event center, Simple Log Service automatically creates a project whose name is in the k8s-log-{cluster-id} format to manage the resources of your event center. The resources include a Logstore and the related dashboards.

Step 3: Use an event center

After you create an event center in K8s Event Center and deploy the kube-eventer and node-problem-detector components, you can use the event center. For example, you can use the event center to view event statistics, query event details, view the lifecycle of a pod, view node events, view core component events, configure alerts, perform custom query operations, and update the version of K8s Event Center.

In the left-side navigation pane of the K8s Event Center page, find the event center that you want to manage and click the k8s事件中心-002 icon. Then, you can perform the following operations.

image

Operation

Description

View event statistics

The Event Overview tab displays the statistics of core events. The statistics include the total number of events, difference between the number of error events within the current day and the preceding day, statistics of alerts, trends of error events, and details of pod OOM events.

Note

If a pod OOM event is recorded, you can view only the node on which the event occurs, process name, and process ID. The pod in which the event occurs cannot be identified. However, you can query the pod restart event that occurs around the time of the pod OOM event. This way, you can identify the pod.

Query event details

The Event Details tab displays the details of the events that are returned by using different filter conditions, such as the event type, event destination, host, namespace, and name.

View the lifecycle of a pod

The Pod Lifecycle tab displays the details of the events that occur within the lifecycle of a pod. You can filter for important pod events by event level.

View node events

The Node Event tab displays the details of node events. You can view the lifecycle of a node and the events that occur on the node.

View core component events

The Event Core tab displays the details of core component events. The events include NLC.Task.RestartECS.Fail and NLC.Task.URL.Mode.Unimplemented.

Configure alerts

On the page that appears after you click Alert Configuration, you can configure alerts for your event center. For more information, see Configure alerts.

Perform custom query operations

On the page that appears after you click Custom Query, you can execute custom query statements.

All events in an event center are stored in a Logstore. You can use all features of the Logstore. For example, you can execute custom query statements, consume event data, create custom reports, or configure custom alerts. For more information, see Query and analyze logs.

If you want to access a project that is specified for an event center, you can obtain the name of the project by using one of the following methods:

  • Obtain the name of the project from the URL of the page that appears after you click Custom Query. The URL is in the https://sls.console.aliyun.com/lognext/app/k8s-event/project/k8s-log-xxxx/logsearch/k8s-event format. The field that follows the project field indicates the name of the project. Example: k8s-log-xxxx.

  • In the list of event centers on the Event Center Management page, find the event center and view the project name.

Update the version of K8s Event Center

On the page that appears after you click Version Update, you can update the version of K8s Event Center.

Delete an event center

On the K8s Event Center > Event Center Management page, find the event center that you want to delete and click the k8s事件中心 icon in the Actions column.

Sample log

The following example shows a collected sample log:

hostname:  cn-hangzhou.i-***********"
level:  Normal
pod_id:  2a360760-****
pod_name:  logtail-ds-blkkr
event_id:  {  
   "metadata":{  
      "name":"logtail-ds-blkkr.157b7cc90de7e192",
      "namespace":"kube-system",
      "selfLink":"/api/v1/namespaces/kube-system/events/logtail-ds-blkkr.157b7cc90de7e192",
      "uid":"2aaf75ab-****",
      "resourceVersion":"6129169",
      "creationTimestamp":"2019-01-20T07:08:19Z"
   },
   "involvedObject":{  
      "kind":"Pod",
      "namespace":"kube-system",
      "name":"logtail-ds-blkkr",
      "uid":"2a360760-****",
      "apiVersion":"v1",
      "resourceVersion":"6129161",
      "fieldPath":"spec.containers{logtail}"
   },
   "reason":"Started",
   "message":"Started container",
   "source":{  
      "component":"kubelet",
      "host":"cn-hangzhou.i-***********"
   },
   "firstTimestamp":"2019-01-20T07:08:19Z",
   "lastTimestamp":"2019-01-20T07:08:19Z",
   "count":1,
   "type":"Normal",
   "eventTime":null,
   "reportingComponent":"",
   "reportingInstance":""
}

Field

Type

Description

hostname

String

The hostname of the server on which the event occurred.

level

String

The level of the log. Valid values: Normal and Warning.

pod_id

String

The unique identifier of the pod. This field is available only if the event type is related to the pod.

pod_name

String

The name of the pod. This field is available only if the event type is related to the pod.

event_id

JSON

The details of the event. The value of this field is a JSON string.

FAQ

Why does no data exist in my event center?

After you deploy an event center, new events are automatically collected to the event center. You can click Custom Query to search for the events. We recommend that you set the time range in the upper-right corner to 1 Day. Data may not be found in the event center due to the following reasons:

  • After you deploy the event center, no events are generated in the associated Kubernetes cluster.

    You can run the kubectl get events --all-namespaces command to check whether new events are generated in a cluster.

  • Invalid values are specified for the parameters that are used to deploy the kube-eventer and node-problem-detector components.

    • If you use an ACK cluster, perform the following steps:

      1. Log on to the ACK console.

      2. On the Clusters page, find the cluster that you want to manage and click the name of the cluster.

      3. In the left-side navigation pane, choose Applications > Helm.

      4. On the Helm page, find ack-node-problem-detector and click Update in the Actions column.

      5. Check and modify the parameter settings. For more information, see Step 1: Deploy the kube-eventer and node-problem-detector components.

    • If you use a self-managed Kubernetes cluster, follow the instructions provided in Collect Kubernetes events.

How do I view the logs of a container in which an event occurs?

  • If you use an ACK cluster, perform the following steps:

    1. Log on to the ACK console.

    2. On the Clusters page, find the cluster that you want to manage and click the name of the cluster.

    3. In the left-side navigation pane, choose Workloads > Pods.

    4. Set the Namespace parameter to kube-system.

    5. On the Pods page, find the pod that you want to manage and click Logs in the Actions column.

  • If you use a self-managed Kubernetes cluster, find the kube-system namespace and the file whose name is prefixed with eventer-sls to view the pod logs.