ApsaraDB for OceanBase:Monitor ApsaraDB for OceanBase by using Prometheus

Background

OBCloud Exporter is a tool provided by ApsaraDB for OceanBase for processing database data. You can use this tool to authenticate with ApsaraDB for OceanBase and obtain data, and then use Prometheus to monitor the database.

You can directly run Prometheus in the system or deploy it by using Docker.

Prerequisites

1. You have installed Java 8 or later.

2. You have deployed Docker.

Run Prometheus in the system to monitor ApsaraDB for OceanBase

1. Download OBCloud Exporter.

   Note

   At present, you cannot directly download OBCloud Exporter. To download the tool, contact OceanBase Technical Support.

2. Deploy OBCloud Exporter.

   1. Decompress the installation package of OBCloud Exporter to the /opt/obcloud-exporter directory.

      mkdir /opt/obcloud-exporter
      tar -zxvf Obcloud_Exporter.tar.gz -C /opt/obcloud-exporter --strip-components=1

      The following files will be extracted from the installation package:

      • Dockerfile: the Docker container build file.

      • muticloud_metric_config.yaml: the metric configuration file. By default, this file does not need to be modified.

      • application.yaml: the collection configuration file.

      • obcloud-exporter-1.0-SNAPSHOT-jar-with-dependencies.jar: the program of OBCloud Exporter.

      • readme.md: the project description.

   2. Edit the application.yaml configuration file.

      # Monitor name.
      aliyun_monitor:
        # Identity credentials.
        credential:
          access_key_id: xxx
          access_key_secret: xx
          region_id: ap-southeast-1
          # Endpoint.
          end_point: xxx
          # Whether to periodically refresh metrics. The default value is false. 
          metric_meta_auto_refresh: false
        instances:
        		# Instance ID.
          - instance_id: xxx
            # To query data of a specified tenant in the cluster, you must configure tenant_id.
            tenant_id: xxx
            # Instance type, which can be cluster (cluster instance) or tenant (tenant instance or serverless instance). 
            instance_type: tenant
            metrics: 

      The parameters are described as follows:

      • access_key_id and access_key_secret: the AccessKey pair for authentication. You can apply for an AccessKey pair from the corresponding platform. For more information, see Create an AccessKey pair.

      • region_id and end_point: For more information, see Regions (ApsaraDB for OceanBase).

      • metric_meta_auto_refresh: specifies whether to periodically refresh metrics. The default value is false.

      • instances: the instances to be monitored. Separate the cluster or tenant instances with hyphens (-).

      • instance_id: the ID of the cluster instance, serverless instance, or tenant instance.

        Note

        The value cannot be a tenant ID in a cluster.

      • instance_type: Valid values are cluster and tenant. When the value of instance_id is a cluster instance ID, set this parameter to cluster. When the value of instance_id is a tenant instance ID or serverless instance ID, set this parameter to tenant.

      • tenant_id: This parameter is required when you want to query data of a specified tenant in a cluster.

      • metrics: the metrics to be collected. If this parameter is not specified, all metrics are collected. Specify the metrics in the following format:

        instances:
          -instance_id: xxx
            tenant_id: xxx
            instance_type: cluster
            metrics: 
              - sql_all_rt
              - sql_delete_rt

        Note

        If a large number of metrics are collected, the collection speed may be affected. In this case, you can use multiple exporters to collect the metrics.

3. Run the JAR file of OBCloud Exporter.

   java -jar /opt/obcloud-exporter/obcloud-exporter-1.0-SNAPSHOT-jar-with-dependencies.jar

   Verify the deployment result in a browser. Specifically, access the port 9400 at the corresponding IP address in the browser to verify whether monitoring data is received.

   Note

   In practice, use the configured IP address and port number for access.

4. Download Prometheus of the required version from the official website. Here is an example of downloading Prometheus 2.50.1 for Linux:

   wget https://github.com/prometheus/prometheus/releases/download/v2.50.1/prometheus-2.50.1.linux-amd64.tar.gz

5. Deploy Prometheus.

   1. Decompress the installation package of Prometheus to the /opt/prometheus directory.

      mkdir -p /opt/prometheus
      tar -zxvf prometheus-2.50.1.linux-amd64.tar.gz -C /opt/prometheus --strip-components=1

   2. Move the prometheus.yml configuration file to the working directory and edit the file as needed.

      mv /opt/prometheus/prometheus.yml /usr/local/bin/
      vi /usr/local/bin/prometheus.yml

      Modify the following parameters:

      global:
        scrape_interval: 30s
        scrape_timeout: 20s
      
      scrape_configs:
        - job_name: "nodes"
          static_configs:
          - targets: ['localhost:9400']

   3. Create the service file and data storage directory of Prometheus.

      mkdir /var/lib/prometheus/
      sudo vim /etc/systemd/system/Prometheus.service

      Enter the save the following content.

      Note

      The User and Group parameters are optional. For information about how to configure a username and a user group, see the next step.

      [Unit]
      Description=Prometheus
      Wants=network-online.target
      After=network-online.target
      
      [Service]
      User=prometheus
      Group=prometheus
      Restart=on-failure
      ExecStart=/opt/prometheus/prometheus \
            --config.file /usr/local/bin/prometheus.yml \
            --storage.tsdb.path /var/lib/prometheus/
      
      [Install]
      WantedBy=multi-user.target

   4. (Optional) Configure a username and a user group. If the username and user group specified in the previous step already exist, or no username and user group are specified, you can skip this step.

      sudo groupadd prometheus
      sudo useradd -r -g prometheus prometheus
      sudo chown -R prometheus:prometheus /opt/prometheus /var/lib/prometheus
      sudo chmod -R 755 /opt/prometheus /var/lib/prometheus

6. Start the Prometheus service.

   # Reload systemd configurations.
   sudo systemctl daemon-reload
   
   # Start the Prometheus service.
   sudo systemctl start prometheus
   
   # Enable the Prometheus service to start at boot.
   sudo systemctl enable prometheus
   
   # Check the service status.
   sudo systemctl status prometheus

7. Check whether Prometheus has started.

    sudo netstat -ntlp | grep 9090

Deploy Prometheus by using Docker to monitor ApsaraDB for OceanBase

1. Download OBCloud Exporter.

   Note

   At present, you cannot directly download OBCloud Exporter. To download the tool, contact OceanBase Technical Support.

2. Deploy OBCloud Exporter.

   1. Decompress the installation package of OBCloud Exporter to the /opt/obcloud-exporter directory.

      mkdir /opt/obcloud-exporter
      tar -zxvf Obcloud_Exporter.tar.gz -C /opt/obcloud-exporter --strip-components=1

      The following files will be extracted from the installation package:

      • Dockerfile: the Docker container build file.

      • muticloud_metric_config.yaml: the metric configuration file. By default, this file does not need to be modified.

      • application.yaml: the collection configuration file.

      • obcloud-exporter-1.0-SNAPSHOT-jar-with-dependencies.jar: the program of OBCloud Exporter.

      • readme.md: the project description.

   2. Edit the application.yaml configuration file.

      # Monitor name.
      aliyun_monitor:
        # Identity credentials.
        credential:
          access_key_id: xxx
          access_key_secret: xx
          region_id: ap-southeast-1
          # Endpoint.
          end_point: xxx
          # Whether to periodically refresh metrics. The default value is false. 
          metric_meta_auto_refresh: false
        instances:
        		# Instance ID.
          - instance_id: xxx
            # To query data of a specified tenant in the cluster, you must configure tenant_id.
            tenant_id: xxx
            # Instance type, which can be cluster (cluster instance) or tenant (tenant instance or serverless instance). 
            instance_type: tenant
            metrics: 

      The parameters are described as follows:

      • access_key_id and access_key_secret: the AccessKey pair for authentication. You can apply for an AccessKey pair from the corresponding platform. For more information, see Create an AccessKey pair.

      • region_id and end_point: For more information, see Regions (ApsaraDB for OceanBase).

      • metric_meta_auto_refresh: specifies whether to periodically refresh metrics. The default value is false.

      • instances: the instances to be monitored. Separate the cluster or tenant instances with hyphens (-).

      • instance_id: the ID of the cluster instance, serverless instance, or tenant instance.

        Note

        The value cannot be a tenant ID in a cluster.

      • instance_type: Valid values are cluster and tenant. When the value of instance_id is a cluster instance ID, set this parameter to cluster. When the value of instance_id is a tenant instance ID or serverless instance ID, set this parameter to tenant.

      • tenant_id: This parameter is required when you want to query data of a specified tenant in a cluster.

      • metrics: the metrics to be collected. If this parameter is not specified, all metrics are collected. Specify the metrics in the following format:

        instances:
          -instance_id: xxx
            tenant_id: xxx
            instance_type: cluster
            metrics: 
              - sql_all_rt
              - sql_delete_rt

        Note

        • For more information, see Monitoring metrics.

        • If a large number of metrics are collected, the collection speed may be affected. In this case, you can use multiple exporters to collect the metrics.

3. Start OBCloud Exporter by using Docker.

   1. In the /opt/obcloud-exporter directory, build the Docker container obcloud-exporter.

      cd /opt/obcloud-exporter
      docker build -t obcloud-exporter:1.0 .

   2. Start the Docker container directly or by using an external configuration file.

      1. Start the Docker container directly.

         docker run -itd -p9400:9400 --name obcloud-exporter obcloud-exporter:1.0

      2. Start the Docker container by using an external configuration file.

         docker run -itd -p9400:9400 -v /opt/obcloud-exporter/application.yaml:/app/application.yaml --name obcloud-exporter obcloud-exporter:1.0

4. Verify the deployment result in a browser. Specifically, access the port 9400 at the corresponding IP address in the browser to verify whether monitoring data is received.

   Note

   In practice, use the configured IP address and port number for access.

5. Create the prometheus.yml configuration file of Prometheus.

   vi /usr/local/bin/prometheus.yml

   Edit the configuration file.

   global:
     scrape_interval: 30s
     evaluation_interval: 30s
   
   scrape_configs:
     - job_name: obcloud-exporter-test
       static_configs:
           # Local server and port number.
         - targets: ['obcloud-exporter:9400']
           labels:
             instance: obcloud-exporter

6. Start the Prometheus service.

    docker run  -d \
      -p 9090:9090 \
      -v /usr/local/bin/prometheus.yml:/etc/prometheus/prometheus.yml  \
      --link obcloud-exporter  \
      --name prometheus \
      prom/prometheus

7. Check whether Prometheus has started.

   sudo netstat -ntlp | grep 9090

View database monitoring data through Prometheus

Visit the Prometheus service URL http://localhost:9090/targets in a browser to view corresponding monitoring data.

Monitoring metrics

When you configure OBCloud Exporter, specify the monitoring metrics supported by ApsaraDB for OceanBase, so that the metrics will be collected by OBCloud Exporter. When OBCloud Exporter is running, the system synchronizes these monitoring metrics, including cluster host monitoring metrics and tenant-level monitoring metrics, to OBCloud Exporter. For more information, see cluster monitoring metrics and tenant monitoring metrics.

When the metric_meta_auto_refresh parameter in the application.yaml configuration file is set to false, supported monitoring metrics are those returned at initial access. When the metric_meta_auto_refresh parameter is set to true, the list of supported monitoring metrics is refreshed every 10 minutes.