Open-source Prometheus monitoring - Container Service for Kubernetes

Prometheus is an open-source monitoring tool tailored for cloud-native applications. This topic explains how to implement Prometheus monitoring within an Alibaba Cloud Container Service for Kubernetes (ACK) cluster.

Background information

This topic explores efficient monitoring of system components and resource entities within a Kubernetes cluster. Monitored objects fall into the following categories:

Resource monitoring involves tracking the resource consumption of nodes and applications, including the utilization of nodes, clusters, and pods in a Kubernetes environment.
Application monitoring focuses on internal application metrics, such as real-time user count and application-level business monitoring and alerting via ports.

Monitored objects include the following:

System components, such as Kubernetes cluster built-ins like the API server, controller-manager, and etcd, with monitoring configured in the configuration file.
Static resource entities, including node resource status and kernel events, with monitoring specified in the configuration file.
Dynamic resource entities, such as deployments, daemon sets, and pods, which are abstract workload entities in Kubernetes. Prometheus monitoring solutions are applicable for these entities.
Custom applications that require unique monitoring data and metrics need specific configurations, achieved by combining port exposure with Prometheus monitoring solutions.

Procedure

Deploy the Prometheus monitoring solution.
1. Log on to the ACK console. In the left-side navigation pane, choose Marketplace > Marketplace.
2. Navigate to the App Catalog page, select the App Directory tab, search for and select ack-prometheus-operator.
3. On the ack-prometheus-operator page, select One-click Deployment.
4. In the Create panel, choose the cluster and namespace, then select Next.
5. Configure the required parameters on the Parameter Configuration page, then click Confirm.
  To view the deployment results:
  1. Execute the following command to map Prometheus from the cluster to local port 9090.
```
kubectl port-forward svc/ack-prometheus-operator-prometheus 9090:9090 -n monitoring
```
  2. Open localhost:9090 in a browser to access Prometheus.
  3. In the menu bar, choose Status > Targets to view all scraping tasks. If the status for all tasks is UP, it indicates that scraping tasks are functioning correctly.
Examine and present data aggregation.
1. Execute the following command to map Grafana from the cluster to local port 3000.
```
kubectl -n monitoring port-forward svc/ack-prometheus-operator-grafana 3000:80
```
2. Open localhost:3000 in a browser, select the appropriate dashboard, and view the aggregated data.
Inspect alert rules and set up silent alerts.
- To view alert rules
  Access in a browser, and select Alerts from the menu bar to view current alert rules.
  - Red indicates alerts are active.
  - Green signifies normal status.
- To configure silent alerts
  Execute the following command and open localhost:9093 in a browser. Choose Silenced to set up silent alerts.
```
kubectl --namespace monitoring port-forward svc/alertmanager-operated 9093
```

The steps above provide an overview of deploying the Prometheus monitoring solution. Subsequent sections detail configurations for various scenarios.

Alert configurations

To configure alert notification methods or templates, follow the steps below to modify the Alertmanager's config field.

Configuring Alert Notification Methods
The prometheus-operator supports notifications via DingTalk and email. To enable these features, follow the configuration steps below.
- Configuring DingTalk Alerts
  
  To set up DingTalk alerts, navigate to the ack-prometheus-operator page, select One-click Deployment, and on the Parameter Configuration page, locate the dingtalk field. Set enabled to true and input your DingTalk chatbot's webhook URL in the Token field. In the Alertmanager's config field, locate the receiver field and specify the name of the DingTalk alert receiver (default is Webhook).
  For instance, to configure two DingTalk chatbots, execute the following:
  1. Replacing the DingTalk Token Configuration
    
    Copy the webhook URLs for your DingTalk chatbots and substitute them for dingtalk1 and dingtalk2. Use your actual webhook URLs in place of the example URL: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxx.
  2. Modifying the Receivers
    
    In the Alertmanager's config field, locate the receiver field and input the names of the DingTalk alert receivers, such as webhook1 and webhook2.
  3. Modifying the URL Values
    
    Update the URL values with the actual DingTalk webhook URLs, such as dingtalk1 and dingtalk2.
  Note
  To add multiple DingTalk chatbots, sequentially increment the Webhook values.
- Configuring Email Alerts
  
  To set up email alerts, visit the ack-prometheus-operator page, click One-click Deployment, and on the Parameter Configuration page, fill in the email details in the highlighted area. In the alertmanager's config field, locate the receiver field and enter the name of the email alert receiver (default is mail).
Configuring Alert Notification Templates

Alert templates can be defined in the Alertmanager's templateFiles field. Below is an example.

Mount custom ConfigMap to Prometheus

This section outlines two methods for mounting a ConfigMap to the /etc/prometheus/configmaps/ directory within a pod.

Method 1: Deploy prometheus-operator for the first time

If this is your initial deployment of prometheus-operator, begin with step 1 to deploy the Prometheus monitoring solution. In the Parameter Configuration wizard, locate the field for Prometheus's ConfigMaps and input the name of your custom ConfigMap. Mounting a ConfigMap

Method 2: prometheus-operator is already deployed

If prometheus-operator is already running in your cluster, proceed with the following steps:

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Applications > Helm.
Navigate to the Actions column on the right of the target Helm application and click Update.
In the Update Release panel, enter your custom ConfigMap's name in the fields for Prometheus and Alertmanager ConfigMaps, then click Confirm.

For instance, to define a ConfigMap named special-config containing the Prometheus configuration file, and to use it with the --config.file parameter at pod startup, add the following definition to the Prometheus ConfigMaps field to mount it within the pod. The mount path is /etc/prometheus/configmaps/.

Below is the YAML definition for special-config.

The configuration for the Prometheus ConfigMaps field is detailed below.

Configure Grafana

Mounting Dashboard Configuration
To mount a dashboard file as a ConfigMap to the Grafana pod, navigate to the ack-prometheus-operator page, select One-click Deployment, and on the Parameter Configuration wizard page, locate the extraConfigmapMounts field to configure the mount settings as illustrated below.
Note
- Verify that the dashboard exists as a ConfigMap within the cluster.
  
  The ConfigMap's labels must match those of other ConfigMaps.
- Input the ConfigMap details and mount settings into the extraConfigmapMounts field for Grafana.
- The mount path is /tmp/dashboards/.
- The ConfigMap refers to the name of your defined ConfigMap.
- The name is the JSON file name of your dashboard.
Enabling Dashboard Persistence

To enable data persistence for dashboards in Grafana, proceed as follows:
1. Log on to the ACK console. In the left-side navigation pane, click Clusters.
2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Applications > Helm.
3. Locate ack-prometheus-operator, then click Update on the right side.
4. In the Update Release panel, find the persistence option under the grafana section, and set up Grafana persistence as depicted in the figure below.
To export your dashboard to a local file, save it as a JSON file. For more information, see Export and import Grafana dashboards.

FAQ

What should I do if I don't receive DingTalk alert notifications?
1. Retrieve the webhook URL of your DingTalk chatbot. For more information, see Scenario 3: Use DingTalk for Kubernetes monitoring and alerting.
2. Locate the dingtalk field, set enabled to true, and input the webhook URL of your DingTalk chatbot in the Token field. For more information, see Alert configurations in Dingtalk Alert Configuration.

What should I do if an error occurs during the deployment of prometheus-operator?

The error message displayed is:

Can't install release with errors: rpc error: code = Unknown desc = object is being deleted: customresourcedefinitions.apiextensions.k8s.io "xxxxxxxx.monitoring.coreos.com" already exists

This error is due to the CRDs from a previous deployment not being fully removed upon uninstalling prometheus-operator. Execute the following commands to delete the CRDs and redeploy prometheus-operator:

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com

What should I do if email alert notifications are not being sent?

Email alerts may fail if the smtp_auth_password field contains your logon password rather than the authorization code. Also, verify that the SMTP server endpoint includes a port number.
What should I do if the message The current cluster is temporarily unavailable. Please try again later or submit a ticket for feedback appears when I attempt to update YAML?

This issue typically arises when the Tiller configuration file is too large, leading to cluster unavailability. Remove some comments from the configuration file and mount it to the pod as a ConfigMap. Note that prometheus-operator currently only supports mounting ConfigMaps to prometheus and alertmanager pods. For more information, see Method 2 in Mount custom ConfigMap to Prometheus.
How can I enable features in prometheus-operator after deployment?

To enable specific features post-deployment, navigate to the cluster information page, select Applications > Helm, and click Update next to ack-prometheus-operator. Locate the relevant switches, adjust the settings, and click Confirm to activate the features.
How do I decide between TSDB and Alibaba Cloud disks?

TSDB is available in a limited number of regions, while Alibaba Cloud disks are globally supported. For data retention policies, refer to the configurations below.
What should I do if a Grafana dashboard does not display data correctly?

On the cluster information page, select Applications > Helm, and click Update next to ack-prometheus-operator. Ensure the clusterVersion is set correctly. If your Kubernetes version is below 1.16, set it to 1.14.8-aliyun.1. If it is 1.16 or later, use 1.16.6-aliyun.1.

What should I do if I cannot reinstall ack-prometheus after deleting the ack-prometheus namespace?

Simply deleting the ack-prometheus namespace can leave behind configurations that interfere with reinstallation. Follow these steps to remove residual configurations:

Remove RBAC permissions.

Delete ClusterRoles.

kubectl delete ClusterRole ack-prometheus-operator-grafana-clusterrole
kubectl delete ClusterRole ack-prometheus-operator-kube-state-metrics
kubectl delete ClusterRole psp-ack-prometheus-operator-kube-state-metrics
kubectl delete ClusterRole psp-ack-prometheus-operator-prometheus-node-exporter
kubectl delete ClusterRole ack-prometheus-operator-operator
kubectl delete ClusterRole ack-prometheus-operator-operator-psp
kubectl delete ClusterRole ack-prometheus-operator-prometheus
kubectl delete ClusterRole ack-prometheus-operator-prometheus-psp

Delete ClusterRoleBindings.

kubectl delete ClusterRoleBinding ack-prometheus-operator-grafana-clusterrolebinding
kubectl delete ClusterRoleBinding ack-prometheus-operator-kube-state-metrics
kubectl delete ClusterRoleBinding psp-ack-prometheus-operator-kube-state-metrics
kubectl delete ClusterRoleBinding psp-ack-prometheus-operator-prometheus-node-exporter
kubectl delete ClusterRoleBinding ack-prometheus-operator-operator
kubectl delete ClusterRoleBinding ack-prometheus-operator-operator-psp
kubectl delete ClusterRoleBinding ack-prometheus-operator-prometheus
kubectl delete ClusterRoleBinding ack-prometheus-operator-prometheus-psp

Remove CRDs.

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com