Enable Managed Service for Prometheus for a registered cluster - Container Service for Kubernetes

You can use Managed Service for Prometheus to centrally manage registered Kubernetes clusters that are deployed in different geolocations. This topic describes how to enable Managed Service for Prometheus for a registered cluster.

Prerequisites

An external Kubernetes cluster is connected to a registered cluster in the Container Service for Kubernetes (ACK) console. For more information, see Create a registered cluster in the ACK console and Use onectl to create a registered cluster.
A kubectl client is connected to the registered cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
A route that points to the internal CIDR block of Managed Service for Prometheus is configured if the external Kubernetes cluster is connected to a virtual private cloud (VPC) by using an Express Connect circuit. For more information, see VPC endpoints and the corresponding CIDR blocks of Managed Service for Prometheus.

Step 1: Configure RAM permissions for the ack-arms-prometheus component

Use onectl

Install onectl on your on-premises machine. For more information, see Use onectl to manage registered clusters.

Run the following command to configure RAM permissions for the ack-arms-prometheus component:

onectl ram-user grant --addon arms-prometheus

Expected output:

Ram policy ack-one-registered-cluster-policy-arms-prometheus granted to ram user ack-one-user-ce313528c3 successfully.

Use the console

If the external Kubernetes cluster is registered to ACK over the Internet, you must specify an AccessKey pair in the registered cluster before you install ack-arms-prometheus. This way, ack-arms-prometheus can use the AccessKey pair to access Alibaba Cloud services. If the external Kubernetes cluster is registered to ACK over the internal network, you do not need to specify an AccessKey pair in the registered cluster.

Run the following command to check whether the external Kubernetes cluster is registered to ACK over the internal network:

kubectl -n kube-system get deploy ack-cluster-agent -o=jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="INTERNAL_ENDPOINT")].value}'

If true is included in the output, the external Kubernetes cluster is registered to ACK over the internal network.
If false is included in the output, the external Kubernetes cluster is registered to ACK over the Internet. In this case, perform the following steps to specify an AccessKey pair in the registered cluster:

Specify an AccessKey pair in the registered cluster

Create a RAM user. For more information, see Create a RAM user.

Create a custom policy. For more information, see Create custom policies. The following policy content includes the permissions that are required by ack-arms-prometheus:

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "arms:Describe*",
                "arms:List*",
                "arms:Get*",
                "arms:Search*",
                "arms:Check*",
                "arms:Query*",
                "arms:ListEnvironments",
                "arms:DescribeAddonRelease",
                "arms:InstallAddon",
                "arms:DeleteAddonRelease",
                "arms:ListEnvironmentDashboards",
                "arms:ListAddonReleases",
                "arms:CreateEnvironment",
                "arms:UpdateEnvironment",
                "arms:InitEnvironment",
                "arms:DescribeEnvironment",
                "arms:InstallEnvironmentFeature",
                "arms:ListEnvironmentFeatures"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Attach the custom policy to the RAM user. For more information, see Grant permissions to RAM users.
Create an AccessKey pair for the RAM user. For more information, see Create an AccessKey pair.
Use the AccessKey pair to create a Secret named alibaba-addon-secret in the registered cluster.
Run the following command to create a Secret for ack-arms-prometheus:
```
kubectl -n arms-prom create secret generic alibaba-addon-secret --from-literal='access-key-id=<your AccessKey ID>' --from-literal='access-key-secret=<your AccessKey Secret>'
```
Note
Replace <your AccessKey ID> and <your AccessKey Secret> with the AccessKey pair that you created in the preceding step.

Step 2: Install the ack-arms-prometheus component

Use onectl

Run the following command to install the ack-arms-prometheus component:

onectl addon install arms-prometheus

Expected output:

Addon arms-prometheus, version **** installed.

Use the console

Log on to the ACK console. In the left-side navigation pane, click Cluster.
On the Clusters page, click the name of a cluster and choose Applications > Helm in the left-side navigation pane.
If the arms-prometheus and arms-prom Helm releases are displayed on the Helm page, delete the Helm releases. Then, perform the following steps to re-install arms-prometheus and arms-prom.
In the left-side navigation pane of the details page, choose Operations > Add-ons.
On the Add-ons page, click the Logs and Monitoring tab.
Find the ack-arms-prometheus component and click Install in the lower-right corner. Click OK.

After the component is installed, go to the Managed Service for Prometheus console. Click the Prometheus instance that is named after the registered cluster. On the details page of the Prometheus instance, you can then view monitoring data and create alert rules. For more information, see Managed Service for Prometheus and Create a Prometheus alert rule.

Configure ack-arms-prometheus to collect metrics from port 10250 on cAdvisor

By default, ack-arms-prometheus collects metrics from port 10255 on cAdvisor. If port 10250 on the kubelet in your registered cluster is open, you can perform the following steps to configure ack-arms-prometheus to collect metrics from port 10250 on cAdvisor (Container Advisor):

Create a ServiceMonitor in the registered cluster based on the following YAML template.

Show the YAML file content

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    arms.prometheus.io/discovery: 'true'
  name: arms-prom-cadvisor-10250
  namespace: arms-prom
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

Wait 15 seconds to 1 minute before you start metric collection.

Check whether metrics are collected.
1. Log on to the ARMS console.
2. In the left-side navigation pane, choose Managed Service for Prometheus > Prometheus Instances.
3. In the top navigation bar of the Managed Service for Prometheus page, select the region where the cluster is deployed. Click the name of the Prometheus instance that you want to view to go to the Integration Center page.
4. In the left-side navigation pane, click Service Discovery. On the page that appears, click the Targets tab.
  Check whether a collection task named arms-prom/arms-prom-cadvisor-10250 exists and runs as expected.