Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Best Practice Guide: Writing Monitoring Metrics to Datadog via Alibaba Cloud Prometheus Monitoring Service

Best Practice Guide: Writing Monitoring Metrics to Datadog via Alibaba Cloud Prometheus Monitoring Service

This guide explains how to convert Alibaba Cloud's monitoring metrics to open-source Prometheus data standards and write them to Datadog Metrics throu.

Disclaimer: This is a translated article, all rights reserved to the original author. The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

Background

An international Alibaba Cloud customer recently built its full-stack observability system based on Alibaba Cloud Application Real-Time Monitoring Service (ARMS), Prometheus, and Grafana. However, due to historical reasons, they wanted to write Alibaba Cloud's monitoring metrics to Datadog as well. After researching, they found that Datadog only supports a limited number of integration components of Alibaba Cloud products and does not support other cloud products. Additionally, Datadog's observable data is in a specific format and is incompatible with open-source standards. However, it supports writing Prometheus data to Datadog through the Observability Pipeline. Fortunately, Alibaba Cloud's observability services are fully compatible with the open-source OpenTelemetry and Prometheus data specifications, and Alibaba Cloud Prometheus can access cloud product monitoring metrics. Users can use this best practice guide to convert Alibaba Cloud's monitoring metrics to open-source Prometheus data standards and write them to Datadog Metrics through the Alibaba Cloud Prometheus monitoring service.

Prerequisites

  1. Activate the Alibaba Cloud Prometheus monitoring service
  2. Activate the Alibaba Cloud CloudMonitor Enterprise cloud monitoring service
  3. Activate the Datadog Pipelines service

Procedure

1. Create a Prometheus for Cloud Service Instance

  • Log in to the Alibaba Cloud console and go to the Alibaba Cloud Prometheus monitoring service control panel
  • Choose the Shanghai region (which has optimal network connectivity for global monitoring metrics in CloudMonitor)
  • Create a new Prometheus for Cloud Service instance

1
2

To select the cloud product to connect, for Alibaba Cloud products, Prometheus supports two data sources: Enterprise Cloud Monitor and Cloud Service Self-Monitoring. However, the current best practice only supports data from Enterprise Cloud Monitor. The following example demonstrates how to connect Alibaba Cloud ECS monitoring data.

Enter the instance name, select Enterprise Cloud Monitor, select Alibaba Cloud ECS, click OK, and wait for the operation to complete.

3

Once the operation is complete, go to the Prometheus monitoring list and click the instance name that was just created. Then, go to the instance details and click Dashboard List. Click any dashboard and confirm that the cloud product monitoring metrics have been written to Prometheus. (There may be a 1-2 minute delay for the data to be written.)

2. Set up an ECS Instance to Prepare for Deploying the Observability Pipeline Worker

Activate an ECS instance in the Singapore region of Alibaba Cloud to deploy the Observability Pipeline service later.

Go to the Alibaba Cloud ECS service console, select the Singapore region, create an ECS instance with 4C8G memory, and allocate a public IPV4 address. Other operating systems can be used, but this example uses Ubuntu.

Note: This tutorial only describes the deployment of Datadog Pipeline single instance to quickly achieve metric writing. It does not consider high availability. Datadog Pipeline also supports high availability. You can hang two ECS instances (both with Datadog Pipeline Worker deployed) after the Alibaba Cloud load balancing service (please refer to the Datadog official website documentation for details) for request load balancing.

After the ECS is successfully set up, log in to the Datadog console to obtain the Observability Pipeline Worker deployment script.

3. Create Observability Pipeline

Log in to the Datadog console and navigate to Integrations - Observability Pipelines and click Create Pipeline to create a new pipeline:

4

After creating the pipeline successfully, click Add Sources on the Pipeline page and select Prometheus Remote Write:

5
6

Enter the component name and listening address. (We recommend using 0.0.0.0, and the port can be customized.) We recommend enabling Auth authentication configuration to ensure data security. Click Save to save the configuration.

7

Click Add Destinations to add a destination source. Select Datadog Metrics:

8
9

Enter the component name and select the source that was configured in the previous step. Set the metric namespace (we recommend using alicloud), choose the region where the data will be written to the Datadog service, and set the Default API Key. (You need to create a Datadog API key and fill in the created key ID. Please refer to the official documentation for information about how to create it.) Click Save to save the configuration.

10

Click Deploy and then Confirm to generate an installation script (do not close the page):

11
12

4. Deploy Observability Pipeline Worker

Log in to the Alibaba Cloud console and go to the ECS list page. Select the ECS instance that was created in the Singapore region and click Remote Connection to enter the online terminal:

13

Follow the Datadog deployment script to complete the deployment of the Observability Pipeline Worker step by step:

Note: When starting the Datadog Pipeline Worker, you may encounter a problem where sensitive information (such as API Key and password) cannot be directly configured in the configuration file. You need to use environment variables.

14

Follow the steps below to set the environment variables:

vim /etc/profile

Set two environment variables and save and exit:

source /etc/profile

Make the environment variables take effect immediately:

15

Restart the Observability Pipeline Worker:

16
17

5. Change the Prometheus Remote Write Configuration to Write Data to the Datadog Observability Pipeline

Note: Before writing Prometheus data to Datadog Pipeline, make sure the configured source port can be accessed publicly. If not, you can configure the ECS security group policy to allow public access.

Go to the Prometheus instance details page that was created earlier, click Settings, and edit the Prometheus.yaml file:

18

19

Set the basic_auth username and password to the ones that were set in the Pipeline Source. Set the URL to the public IP and port of the deployed Pipeline Worker service. Click Save. Go back to the Datadog Pipeline Worker configuration page. If you see the following message, the data has been successfully written.

20

Wait 1-2 minutes, go to the Datadog Metrics Explorer page, and search for the alicloud namespace metric. You can see that the ECS-related monitoring metrics have been written to Datadog. You can use Datadog's dashboard capabilities to create custom monitoring dashboards.

21

ARMS monitoring CloudMonitor Tutorials Monitoring Service Prometheus Observability Datadog
0 2 1
Share on

Read previous post:

Analysis of Alibaba Cloud Container Network Data Link (5): Terway ENI-Trunking

Read next post:

Alibaba Cloud Was Named a Leader in the Forrester Wave™: FaaS, Q2 2023

Alibaba Cloud Native

206 posts | 12 followers

You may also like

Comments

Alibaba Cloud Native

206 posts | 12 followers

Related Products

More Posts by Alibaba Cloud Native

See All