This topic describes how to use the eventer component to collect events from Kubernetes to Simple Log Service.
Kubernetes is designed based on the state machine. Events are generated due to transitions between different states. Typically, Normal events are generated when the state machine changes to expected states and Warning events are generated when the state machine changes to unexpected states.
ACK provides out-of-the-box event monitoring solutions that target containerization scenarios. ACK uses NPD and kube-eventer provided by NPD to monitor container events.
NPD is a tool to diagnose Kubernetes nodes. NPD detects node exceptions, generates node events, and works with kube-eventer to raise alerts upon these events and enable closed-loop management of alerts. NPD generates node events when the following exceptions are detected: Docker engine hangs, Linux kernel hangs, outbound traffic exceptions, and file descriptor exceptions. For more information, see NPD.
kube-eventer is an open source event emitter that is maintained by ACK. kube-eventer sends Kubernetes events to sinks such as DingTalk, Simple Log Service, and EventBridge. kube-eventer also provides filter conditions to filter different levels of events. You can use kube-eventer to collect events in real time, trigger alerts upon specific events, and asynchronously archive events. For more information, see kube-eventer.
NPD is a tool to diagnose Kubernetes nodes. NPD detects node exceptions, generates node events, and works with kube-eventer to raise alerts upon these events and enable closed-loop management of alerts. NPD generates node events when the following exceptions are detected: Docker engine hangs, Linux kernel hangs, outbound traffic exceptions, and file descriptor exceptions. For more information, see NPD.
kube-eventer is an open source event emitter that is maintained by ACK. kube-eventer sends Kubernetes events to sinks such as DingTalk, Simple Log Service, and EventBridge. kube-eventer also provides filter conditions to filter different levels of events. You can use kube-eventer to collect events in real time, trigger alerts upon specific events, and asynchronously archive events. For more information, see kube-eventer.
Prerequisites
A Kubernetes cluster is created, such as a Container Service for Kubernetes (ACK) cluster and an ACK Serverless cluster.
Billing
If the following conditions are met, you are not charged when you use an event center:
The data retention period of the Logstore that is associated with your event center is 90 days, which is the default value.
The amount of data that is written to your event center per day is less than 256 MB, which is equivalent to approximately 250,000 events.
Examples:
If the default data retention period is specified for the associated Logstore and 1,000 events are generated in your Kubernetes cluster per day, you can use your event center for free.
If you change the data retention period to 105 days and 1,000 events are generated in your Kubernetes cluster per day, you are charged for the data that is stored in the associated Logstore after the data is stored for 90 days. The fee is calculated based on the billable item of the storage space occupied by log data. For more information about the billable item, see Billable items of pay-by-feature.
Step 1: Deploy the kube-eventer and node-problem-detector components
ACK cluster
If you use an ACK cluster, you need to only deploy the ack-node-problem-detector component because the component is integrated with the features of the kube-eventer and node-problem-detector components. For more information, see Event monitoring. If you use an ACK Serverless cluster, you must deploy the kube-eventer component.
NPD works with third-party plug-ins to detect node exceptions and generate cluster events. A Kubernetes cluster also generates events when the status of the cluster changes. For example, when a pod is evicted or an image pull operation fails, a related event is generated. The Kubernetes event center of Simple Log Service collects, stores, and visualizes cluster events. It allows you to query and analyze these events, and configure alerts. You can sink cluster events to the Kubernetes event center of Simple Log Service by using the following methods.
If you have selected Install node-problem-detector and Create Event Center when you create the cluster, proceed to Step 2 to view the Kubernetes event center. For more information about how to install the ack-node-problem-detector component when you create a cluster, see Create an ACK managed cluster.
If you did not select Install node-problem-detector and Create Event Center when you create the cluster, perform the following steps to manually install it.
Log on to the ACK console.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose .
On the Logging and Monitoring tab, find and install ack-node-problem-detector.
Other Kubernetes cluster
Deploy the kube-eventer component.
Install the kubectl tool. For more information, see Connect to an ACK cluster by using kubectl.
Create a configuration file named eventer.yaml by using the following sample code:
apiVersion: apps/v1 kind: Deployment metadata: labels: name: kube-eventer name: kube-eventer namespace: kube-system spec: replicas: 1 selector: matchLabels: app: kube-eventer template: metadata: labels: app: kube-eventer annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: dnsPolicy: ClusterFirstWithHostNet serviceAccount: kube-eventer containers: - image: registry.cn-hangzhou.aliyuncs.com/acs/kube-eventer:v1.2.5-cc7ec54-aliyun name: kube-eventer command: - "/kube-eventer" - "--source=kubernetes:https://kubernetes.default" ## .send to sls ## --sink=sls:https://{endpoint}?project={project}&logStore=k8s-event®ionId={region-id}&internal=false&accessKeyId={accessKeyId}&accessKeySecret={accessKeySecret} - --sink=sls:https://cn-beijing.log.aliyuncs.com?project=k8s-xxxx&logStore=k8s-event®ionId=cn-beijing&internal=false&accessKeyId=xxx&accessKeySecret=xxx env: # If TZ is assigned, set the TZ value as the time zone - name: TZ value: "Asia/Shanghai" volumeMounts: - name: localtime mountPath: /etc/localtime readOnly: true - name: zoneinfo mountPath: /usr/share/zoneinfo readOnly: true resources: requests: cpu: 10m memory: 50Mi limits: cpu: 500m memory: 250Mi volumes: - name: localtime hostPath: path: /etc/localtime - name: zoneinfo hostPath: path: /usr/share/zoneinfo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-eventer rules: - apiGroups: - "" resources: - events verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-eventer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-eventer subjects: - kind: ServiceAccount name: kube-eventer namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: kube-eventer namespace: kube-system
Parameter
Type
Required
Description
endpoint
string
Yes
The Simple Log Service endpoint. For more information, see Endpoints.
project
string
Yes
The Simple Log Service project.
logStore
string
Yes
The Simple Log Service Logstore.
internal
string
Yes for self-managed Kubernetes clusters
If you use a self-managed Kubernetes cluster, set the value to false.
regionId
string
Yes for self-managed Kubernetes clusters
The ID of the region where the Simple Log Service project resides. For more information, see Endpoints.
accessKeyId
string
Yes for self-managed Kubernetes clusters
The AccessKey ID. We recommend that you use the AccessKey ID of a Resource Access Management (RAM) user. For more information, see AccessKey pair.
accessKeySecret
string
Yes for self-managed Kubernetes clusters
The AccessKey secret. We recommend that you use the AccessKey secret of a RAM user. For more information, see AccessKey pair.
Run the following command to apply the eventer.yaml configuration file to your cluster:
kubectl apply -f eventer.yaml
Expected output:
deployment.apps/kube-eventer created clusterrole.rbac.authorization.k8s.io/kube-eventer created clusterrolebinding.rbac.authorization.k8s.io/kube-eventer created serviceaccount/kube-eventer created
Deploy the node-problem-detector component.
For more information, visit GitHub.
Step 2: Create an event center
After you create an event center, Simple Log Service automatically creates a Logstore named k8s-event
in the specified project. Related dashboards are also created for the event center.
Log on to the Simple Log Service console.
In the Log Application section, click the Intelligent O&M tab. Then, click K8s Event Center.
On the Event Center Management page, click Add in the upper-right corner.
In the Create Event Center panel, configure the parameters and click Next.
If you select Select Existing Project, you can select an existing project from the Project drop-down list to manage the resources of your event center. The resources include a Logstore and the related dashboards.
If you select Select Kubernetes Cluster from Container Service for Kubernetes, you can select an existing Kubernetes cluster from the K8s Cluster drop-down list. If you use this method to create an event center, Simple Log Service automatically creates a project whose name is in the
k8s-log-{cluster-id}
format to manage the resources of your event center. The resources include a Logstore and the related dashboards.
Step 3: Use an event center
After you create an event center in K8s Event Center and deploy the kube-eventer and node-problem-detector components, you can use the event center. For example, you can use the event center to view event statistics, query event details, view the lifecycle of a pod, view node events, view core component events, configure alerts, perform custom query operations, and update the version of K8s Event Center.
In the left-side navigation pane of the K8s Event Center page, find the event center that you want to manage and click the icon. Then, you can perform the following operations.
Operation | Description |
View event statistics | The Event Overview tab displays the statistics of core events. The statistics include the total number of events, difference between the number of error events within the current day and the preceding day, statistics of alerts, trends of error events, and details of pod OOM events. Note If a pod OOM event is recorded, you can view only the node on which the event occurs, process name, and process ID. The pod in which the event occurs cannot be identified. However, you can query the pod restart event that occurs around the time of the pod OOM event. This way, you can identify the pod. |
Query event details | The Event Details tab displays the details of the events that are returned by using different filter conditions, such as the event type, event destination, host, namespace, and name. |
View the lifecycle of a pod | The Pod Lifecycle tab displays the details of the events that occur within the lifecycle of a pod. You can filter for important pod events by event level. |
View node events | The Node Event tab displays the details of node events. You can view the lifecycle of a node and the events that occur on the node. |
View core component events | The Event Core tab displays the details of core component events. The events include NLC.Task.RestartECS.Fail and NLC.Task.URL.Mode.Unimplemented. |
Configure alerts | On the page that appears after you click Alert Configuration, you can configure alerts for your event center. For more information, see Configure alerts. |
Perform custom query operations | On the page that appears after you click Custom Query, you can execute custom query statements. All events in an event center are stored in a Logstore. You can use all features of the Logstore. For example, you can execute custom query statements, consume event data, create custom reports, or configure custom alerts. For more information, see Query and analyze logs. If you want to access a project that is specified for an event center, you can obtain the name of the project by using one of the following methods:
|
Update the version of K8s Event Center | On the page that appears after you click Version Update, you can update the version of K8s Event Center. |
Sample log
The following example shows a collected sample log:
hostname: cn-hangzhou.i-***********"
level: Normal
pod_id: 2a360760-****
pod_name: logtail-ds-blkkr
event_id: {
"metadata":{
"name":"logtail-ds-blkkr.157b7cc90de7e192",
"namespace":"kube-system",
"selfLink":"/api/v1/namespaces/kube-system/events/logtail-ds-blkkr.157b7cc90de7e192",
"uid":"2aaf75ab-****",
"resourceVersion":"6129169",
"creationTimestamp":"2019-01-20T07:08:19Z"
},
"involvedObject":{
"kind":"Pod",
"namespace":"kube-system",
"name":"logtail-ds-blkkr",
"uid":"2a360760-****",
"apiVersion":"v1",
"resourceVersion":"6129161",
"fieldPath":"spec.containers{logtail}"
},
"reason":"Started",
"message":"Started container",
"source":{
"component":"kubelet",
"host":"cn-hangzhou.i-***********"
},
"firstTimestamp":"2019-01-20T07:08:19Z",
"lastTimestamp":"2019-01-20T07:08:19Z",
"count":1,
"type":"Normal",
"eventTime":null,
"reportingComponent":"",
"reportingInstance":""
}
Field | Type | Description |
hostname | String | The hostname of the server on which the event occurred. |
level | String | The level of the log. Valid values: Normal and Warning. |
pod_id | String | The unique identifier of the pod. This field is available only if the event type is related to the pod. |
pod_name | String | The name of the pod. This field is available only if the event type is related to the pod. |
event_id | JSON | The details of the event. The value of this field is a JSON string. |
FAQ
Why does no data exist in my event center?
After you deploy an event center, new events are automatically collected to the event center. You can click Custom Query to search for the events. We recommend that you set the time range in the upper-right corner to 1 Day. Data may not be found in the event center due to the following reasons:
After you deploy the event center, no events are generated in the associated Kubernetes cluster.
You can run the
kubectl get events --all-namespaces
command to check whether new events are generated in a cluster.Invalid values are specified for the parameters that are used to deploy the kube-eventer and node-problem-detector components.
If you use an ACK cluster, perform the following steps:
Log on to the ACK console.
On the Clusters page, find the cluster that you want to manage and click the name of the cluster.
In the left-side navigation pane, choose
.On the Helm page, find ack-node-problem-detector and click Update in the Actions column.
Check and modify the parameter settings. For more information, see Step 1: Deploy the kube-eventer and node-problem-detector components.
If you use a self-managed Kubernetes cluster, follow the instructions provided in Collect Kubernetes events.
How do I view the logs of a container in which an event occurs?
If you use an ACK cluster, perform the following steps:
Log on to the ACK console.
On the Clusters page, find the cluster that you want to manage and click the name of the cluster.
In the left-side navigation pane, choose
.Set the Namespace parameter to kube-system.
On the Pods page, find the pod that you want to manage and click Logs in the Actions column.
If you use a self-managed Kubernetes cluster, find the kube-system namespace and the file whose name is prefixed with eventer-sls to view the pod logs.