×
Community Blog How to Migrate from Self-managed Open-source Prometheus to Alibaba Cloud Managed Service for Prometheus

How to Migrate from Self-managed Open-source Prometheus to Alibaba Cloud Managed Service for Prometheus

This article discusses migration solutions in various typical scenarios of Alibaba Cloud Managed Service for Prometheus.

By Yingchuan

Introduction

Prometheus has become the industry standard in the field of observability metrics. Due to various factors, some users still use "self-managed open-source Prometheus, self-managed open-source Thanos, and self-managed Grafana" to implement metric monitoring and alerting for infrastructure and business applications. Alibaba Cloud Managed Service for Prometheus is fully integrated with the open-source Prometheus ecosystem. It supports observations for a wide range of components and provides high-performance, high-availability, high-scalability, low-cost, and easy-to-maintain metric observations. To facilitate migration from self-managed open-source Prometheus to Alibaba Cloud Managed Service for Prometheus, this article describes migration solutions in various typical scenarios.

Typical Deployment Scenarios of Self-managed Open-source Prometheus and Thanos

1

Currently, typical deployment scenarios of self-managed open-source Prometheus and Thanos include:

  • Open-source Prometheus installed in a Kubernetes cluster:

    • Installed in an ACK cluster.
    • Installed in Container Service clusters of other cloud vendors.
    • Installed in a self-managed Kubernetes cluster in a self-managed IDC.
  • Open-source Prometheus installed in a non-Kubernetes cluster:

    • Installed on Alibaba Cloud ECS.
    • Installed on ECS of other cloud vendors.
    • Installed on a VM in a self-managed IDC.

To meet long-term monitoring requirements, users usually deploy one or more sets of self-managed Thanos on the basis of self-managed open-source Prometheus to store multiple self-managed open-source Prometheus datasets in a centralized and long-term manner.

Pain Points of Self-managed Open-source Prometheus and Thanos

For self-managed open-source Prometheus and Thanos, enterprises usually face the following major problems:

• For various components used in business systems, you need to install the exporter and configure dashboards and alert rules, which is a heavy workload. In addition, the dashboards and alert rules of the open-source Grafana are not professional enough, and the dashboards and alert rules that are optimized based on the principles and best practices of observation components are not available.

• Multiple self-managed open-source Prometheus applications must be installed in different container clusters or VPCs due to different enterprise departments or business systems, resulting in high deployment costs and complex O&M.

• Self-managed open-source Prometheus cannot handle metrics with large traffic when container clusters or ECS instances are on a large scale.

• The introduction of self-managed Thanos for centralized and long-term storage increases the complexity of the entire metric observability system. In addition, Thanos Receiver is required in non-Kubernetes scenarios, which makes the entire Thanos deployment and O&M more complex and costly.

The Capability Framework of Alibaba Cloud Managed Service for Prometheus

Alibaba Cloud Managed Service for Prometheus is fully integrated with the open-source Prometheus ecosystem. It supports observations for a wide range of components and provides a variety of out-of-the-box preset dashboards and fully managed hybrid cloud and multi-cloud Prometheus services. In addition to supporting Alibaba Cloud Container Service, self-managed Kubernetes, and Remote Write, Alibaba Cloud Managed Service for Prometheus also provides metric observation capabilities for hybrid cloud and multi-cloud ECS applications. It also supports multi-instance aggregated observation capabilities to implement unified query of Prometheus metrics, unified Grafana data sources, and unified alerts.

2

Comparison Between Self-managed Open-source Prometheus and Alibaba Cloud Managed Service for Prometheus

Comparison Item Open-source Prometheus Alibaba Cloud Managed Service for Prometheus
Deployment and O&M costs Prometheus, Grafana, and AlertManager must be deployed on self-paid Elastic Compute Service (ECS) instances in multiple VPCs, multi-container clusters, multi-cloud clusters, and multi-IDC clusters, which requires high deployment and O&M costs. Prometheus, Grafana, and Alert Center are integrated, fully managed, O&M-free, and out-of-the-box.
Availability, performance, data capacity, and retention period Single node, low collection and query performance, small data capacity, and short retention period. High availability, high collection and query performance, large data capacity, and long retention period.
Components Manually deploy various open-source exporters, create collection jobs and dashboards, and configure alert rules. Provide one-click access to common components (automatic deployment of exporter, collection job, and creation of dashboards and alert rules).
Grafana dashboard The open-source Grafana dashboard is usually simple. In most cases, the collected metrics are displayed directly, lacking in-depth optimization based on the principles and best practices of monitoring components. Professional dashboard templates for common components are provided to help you quickly and accurately understand the running status of components and troubleshoot issues.
Alert item Alert item templates for common components are not available. You must learn and configure alert items by yourself. Professional and flexible alert item templates based on the monitoring practices of each component are provided. You can configure the core and recommended alert items of each component in a white-screen manner.
Global aggregate query Self-managed Thanos implements "physical aggregation", which has high deployment and O&M costs and low flexibility. Provide global aggregation instances capability. This feature supports high-performance dynamic aggregation query, dashboard display, and alert of data from multiple Prometheus instances.

Technical Solution of Migrating Self-managed Open-source Prometheus to Alibaba Cloud Managed Service for Prometheus

The migration from self-managed open-source Prometheus to Alibaba Cloud Managed Service for Prometheus consists of three phases: metric collection, visualized analysis, and alert configuration. The following sections discuss the solutions for these three phases in different self-managed Prometheus deployment scenarios.

Step 1: Migrate Metric Collection

Metric collection means Prometheus regularly pulls and saves metric data from the target monitoring component or its corresponding exporter based on the configuration of the collection job.

Alibaba Cloud Container Service Scenario: Container Cluster Monitoring

3

Migration Requirements

• After Alibaba Cloud Managed Service for Prometheus is configured and verified, the migration can be completed after n days (the number of days for which the existing self-managed open-source Prometheus Service is stored). This way, Alibaba Cloud Managed Service for Prometheus can collect complete monitoring data.

Migration Procedures

Log on to the console of Alibaba Cloud Managed Service for Prometheus. In the Integration Center, select Kubernetes Cluster Monitor to create the Kubernetes Environment instance.

4

In the Prometheus console, choose Integration Management. On the Integration Management page, configure the integration of each monitoring component in the container cluster. This way, Alibaba Cloud Managed Service for Prometheus can generate collection jobs, capture monitoring data, and generate default dashboard and default alert rules.

5

[Optional] For user-defined collection tasks (ServiceMonitor, PodMonitor, and custom jobs), you can configure custom collection rules in the corresponding environment instance of Integration Management.

6

• Verify whether the container cluster dashboard and the default dashboard of each component of the container cluster on Alibaba Cloud Managed Service for Prometheus are normal.

7

Self-managed Kubernetes Scenario: Container Cluster Monitoring

8

This scenario is similar to the Self-managed Kubernetes Scenario: Container Cluster Monitoring, except that you need to first register the self-managed Kubernetes cluster as the registered cluster of Alibaba Cloud Container Service.

Alibaba Cloud ECS Scenario: ECS Monitoring

9

Migration Requirements

• After Alibaba Cloud Managed Service for Prometheus is configured and verified, the migration can be completed after n days (the number of days for which the existing self-managed open-source Prometheus Service is stored). This way, Alibaba Cloud Managed Service for Prometheus can collect complete monitoring data.

Migration Procedures

• Log on to the console of Alibaba Cloud Managed Service for Prometheus. On the Integration Center page, click Host Monitor, and follow the wizard to create an ECS Environment instance. Then, deploy the exporter and collection configuration for the ECS Host Monitor.

10

• In the Integration Management of the console, configure the integration of each component to be monitored in the VPC so that Alibaba Cloud Managed Service for Prometheus can generate collection jobs, capture monitoring data, and generate default dashboard and default alert rules.

• [Optional] For user-defined collection tasks, you can configure custom collection rules in the corresponding environment instance of Integration Management.

• Verify whether the default dashboard of each component on Alibaba Cloud Managed Service for Prometheus is normal.

Self-managed IDC Scenario: ECS Monitoring

11

Migration Requirements

• After Alibaba Cloud Managed Service for Prometheus is configured and verified, the migration can be completed after n days (the number of days for which the existing self-managed open-source Prometheus Service is stored). This way, Alibaba Cloud Managed Service for Prometheus can collect complete monitoring data.

• You need to connect the self-managed IDC to an Alibaba Cloud VPC through methods such as the VPC leased line.

• You must install Node/Process Exporter on each VM of the self-managed IDC.

Migration Procedures

• Log on to the console of Alibaba Cloud Managed Service for Prometheus. On the Integration Center page, click Host Monitor (self-service installation and IP domain selection), and follow the wizard to create an ECS Environment instance. Then, generate the collection configuration for the Host Monitor.

• In the Integration Management of the console, configure the integration of each component to be monitored in the VPC so that Alibaba Cloud Managed Service for Prometheus can generate collection jobs, capture monitoring data, and generate default dashboard and default alert rules.

[Optional] For user-defined collection tasks, you can configure custom collection rules in the corresponding environment instance of Integration Management.

• Verify whether the default dashboard of each component on Alibaba Cloud Managed Service for Prometheus is normal.

General Remote Write Scenario: General-purpose Instance

12

Migration Requirements

• After Alibaba Cloud Managed Service for Prometheus is configured and verified, the migration can be completed after n days (the number of days for which the existing self-managed open-source Prometheus Service is stored). This way, Alibaba Cloud Managed Service for Prometheus can collect complete data.

Migration Procedures

• Log on to the console of Alibaba Cloud Managed Service for Prometheus, choose Instances, click Create Instance, and select General-purpose Instance to create an instance.

13

Modify the configuration of self-managed open-source Prometheus to Remote Write monitoring data to Alibaba Cloud Managed Service for Prometheus.

On the Metrics Center of the console of Alibaba Cloud Managed Service for Prometheus, query and verify metrics from the self-managed open-source Prometheus Remote Write.

Step 2: Migrate Visualized Analysis

In the migration process of the above-mentioned metric collection, Alibaba Cloud Managed Service for Prometheus has built-in default, professional, and out-of-the-box dashboards for container clusters and common components (such as MySQL and Redis).

• For container/Kubernetes and ECS monitoring scenarios:

In the console of Alibaba Cloud Managed Service for Prometheus, you can view the default dashboard of each component on the Query Dashboards page.

For a custom dashboard, you must create a Grafana Pro Edition and then migrate self-managed Grafana or import the specified dashboard.

14

• For general-purpose instances in Remote Write scenarios: To continue using the existing self-managed Grafana, you only need to change the data source of Grafana to the corresponding instance data source of Alibaba Cloud Managed Service for Prometheus.

For the scenario that you want to aggregate and display the monitoring data of multiple containers/Kubernetes/ECS/IDC, you can use the global aggregation instance of Alibaba Cloud Managed Service for Prometheus.

15
16

Step 3: Migrate Alert Configurations

Similar to the migration of visualized analysis, Alibaba Cloud Managed Service for Prometheus has also created professional and out-of-the-box alert rules for container clusters and common components (such as MySQL and Redis).

• For integration scenarios of containers/Kubernetes and Elastic Compute Service (ECS) instances, log on to the console of Alibaba Cloud Managed Service for Prometheus. On the Integrated Addons page of Integration Management, click Alerts of the corresponding component to view the default alert rules generated for the component.

17

For custom alert rules, you can create and manage them on the Prometheus Alert Rules tab of the Prometheus Monitor page in the console of Alibaba Cloud Managed Service for Prometheus. You can also use the Prometheus Alert Rules Template to import existing Prometheus alert rules.

18

• For the scenario that you want to aggregate and alert the monitoring data of multiple containers/Kubernetes/ECS/IDC, you can use the global aggregation instance of Alibaba Cloud Managed Service for Prometheus.

19

About Alibaba Cloud Managed Service for Prometheus

Alibaba Cloud Managed Service for Prometheus is a fully managed observability service built based on open-source Prometheus that is an industry standard for cloud-native observability. By default, it integrates common cloud services and is compatible with mainstream open-source components. The service provides comprehensive observation of businesses, applications, middleware, and systems. In addition, it uses an out-of-the-box Grafana dashboard and intelligent alerting feature to optimize agent performance and system availability. Managed Service for Prometheus is a good choice for quickly building an all-in-one metric monitoring system in your enterprise. Therefore, enterprises can quickly identify and locate problems in businesses, reduce the impact of faults on businesses, and eliminate the workload of system construction and routine maintenance, thus effectively improving the efficiency of O&M and observation.

At the same time, Alibaba Cloud Managed Service for Prometheus, as an important component of Alibaba Cloud Observability Suite (ACOS), is integrated with Grafana and Tracing Analysis to form an observable data layer that supports storage and analysis of metrics and trace data, and integration of heterogeneous data sources. It also allows you to view dashboards, configure the alert rules, and explore data with the standard PromQL or SQL syntax. What's more, it gives data value to different scenarios, such as IT cost management, enterprise risk governance, intelligent O&M, and business continuity assurance, so that observable data can go beyond observation.

0 0 0
Share on

Alibaba Cloud Native

204 posts | 12 followers

You may also like

Comments