Using Prometheus and Blackbox Exporter to Monitor Services on Alibaba Cloud

By Alex Muchiri.

In this tutorial, we will explore ways to monitor the activity of a service running on an Alibaba Cloud ECS server. The tools for the job include Prometheus, Alertmanager and Blackbox Exporter. Follow this link to access the latest Prometheus client download. Prometheus is open source and is used by developers and system administrators to track system metrics, monitor services, and record that information in a time-series database. It makes use of an Alertmanager for alerting receivers, which may also duplicate or group different or similar alert types. The advantage of using the manager is its compatibility with messaging services such as email.

While Prometheus is a useful tool all by itself, it becomes even handier when integrated with an exporter. Your infrastructure can be separated into various components including web and database servers, messaging systems or APIs. An exporter such as Blackbox Exporter can have alerts configured for all these individual parts. Blackbox Exporter uses several techniques to run request metrics including HTTP, HTTPS, DNS, TCP, and ICMP. For each request, there are detailed data points such as the success of requests and response times. We shall use both tools to see how well a NGINX server running on Alibaba Cloud responds to our requests. We will also create email alerts if the server fails a response test.

What You Will Need for This Tutorial

To complete the setup guide in this tutorial, you should have the following:

An Alibaba Cloud Elastic Compute Service (ECS) instance that has Ubuntu 18.04 installed on it.
A NGINX server that is installed with Ubuntu.
A server block listening through a known port. For this, we will use a Django app running on port 8000.
Prometheus installed and running on your server. Steps to do this are given below.
An SMTP email server

We will all of these set up in this tutorial. Note that it is also possible to configure Alertmanager to send notifications to Slack.

Installing Prometheus

Prometheus is available by default in Ubuntu 18.04 (Bionic Beaver), but in some versions, which are outdated, you may not have it installed. The first step is to update the package list with the following commands:

sudo apt-get update
sudo apt-get upgrade

Next, you'll want to run the command below to check what version is available in your system.

sudo apt-cache policy prometheus

The output should look like this:

prometheus:
  Installed: (none)
  Candidate: 2.1.0+ds-1
  Version table:
     2.1.0+ds-1 500
        500 <http://ke.archive.ubuntu.com/ubuntu bionic/universe> amd64 Packages

Now check if Prometheus is installed by executing the command below:

sudo systemctl status prometheus

Below is the expected output.

Unit prometheus.service could not be found.

Next, we will need to install Prometheus on our Alibaba Cloud instance by running the command below.

sudo apt-get install prometheus

If Prometheus is unavailable in your package repository, you will need to fetch the key first. You can do so with this command.

wget https://s3-eu-west-1.amazonaws.com/deb.robustperception.io/41EFC99D.gpg | sudo apt-key add -

Then, update the package list.

sudo apt-get update -y

Then you can install Prometheus.

sudo apt-get install prometheus

Once the installation process is complete, activate Prometheus by running the commands below:

sudo systemctl start prometheus
sudo systemctl enable prometheus

Confirm that the service has been created successfully and that it is in good state by running the following command:

sudo systemctl status prometheus

Below is the expected output.

- prometheus.service - Monitoring system and time series database
   Loaded: loaded (/lib/systemd/system/prometheus.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-08-13 19:04:20 UTC; 22s ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 8976 (prometheus)
    Tasks: 4 (limit: 2312)
   CGroup: /system.slice/prometheus.service
           ??9078 /usr/bin/prometheus

Now that we have Prometheus installed, let's now move on to installing Blackbox Exporter.

Installing Blackbox Exporter

Now you'll want to download Blackbox Exporter using the commands below.

cd ~
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.14.0/blackbox_exporter-0.14.0.linux-amd64.tar.gz

Extract the binaries with the following command.

tar -xvf blackbox_exporter-0.14.0.linux-amd64.tar.gz

Create a Blackbox Exporter user.

sudo useradd --no-create-home --shell /bin/false blackbox_exporter

Copy the extracted binaries to the new user's bin directory.

sudo mv blackbox_exporter-0.14.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter

Then remove the downloaded file.

sudo rm -r blackbox_exporter-0.14.0.linux-amd64.tar.gz

Next, run the command below to change directory ownership.

sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter

Now let's create a new configuration file for Blackbox Exporter. To do with the following commands:

sudo mkdir /etc/blackbox_exporter
sudo nano /etc/blackbox_exporter/blackbox.yml

Copy the configurations below to your file:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET

To proceed, you'll need to create a systemd service file. Create one with the following command:

sudo nano /etc/systemd/system/blackbox_exporter.service

Place the lines below into the file:

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target

Next, update Blackbox Exporter's configuration file permissions.

chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml

Reload the systemctl daemon.

sudo systemctl daemon-reload

Start the Blackbox Exporter service.

sudo systemctl start blackbox_exporter

Confirm that the service launched succesfully and is in a good status.

systemctl status blackbox_exporter

Below is the expected output.

  blackbox_exporter.service - Blackbox Exporter
   Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-08-13 19:12:40 UTC; 18s ago
 Main PID: 10084 (blackbox_export)
    Tasks: 6 
   CGroup: /system.slice/blackbox_exporter.service
           └─10084 /usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

Aug 13 00:02:40 ip-172-31-41-126 systemd[1]: Started Blackbox Exporter.
Aug 13 00:02:40 ip-92-101-100-68 blackbox_exporter[10084]: level=info ts=2019-08-13T19:12:40.5229204Z caller=main.go:213 msg="Starting blackbox_exporter" version="(version=0.14.0, branch=HEAD, revision=bb
Aug 13 00:02:40 ip-92-101-100-68 blackbox_exporter[10084]: level=info ts=2019-08-13T19:12:40.52553523Z caller=main.go:226 msg="Loaded config file"
Aug 13 00:02:40 ip-92-101-100-68 blackbox_exporter[10084]: level=info ts=2019-08-13T19:12:40.525695324Z caller=main.go:330 msg="Listening on address" address=:9115

Run the below command to start the service at system boot:

sudo systemctl enable blackbox_exporter

Configure Prometheus

Now, we will monitor a simple Django application running on port 8000 using Prometheus and Blackbox Exporter. We will create an additional configuration in the Prometheus configuration file under the assumption that the app is already running on the respective port.

sudo nano /etc/prometheus/prometheus.yml

Add the lines below in the file.

 - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://localhost:8000
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Restart Prometheus with the following command.

sudo systemctl restart prometheus

Create the alert rules for Prometheus with the following command.

sudo nano /etc/prometheus/alert.rules.yml

Append the lines below in the file.

groups:
- name: alert.rules
  rules:
  - alert: EndpointDown
    expr: probe_success == 0
    for: 10s
    labels:
      severity: "critical"
    annotations:
      summary: "Endpoint  down"

Update the permissions by running the command below.

sudo chown prometheus:prometheus /etc/prometheus/alert.rules.yml

Verify that the rules have been created successfully by running the command below.

sudo promtool check rules /etc/prometheus/alert.rules.yml

Below is the expected output.

Checking /etc/prometheus/alert.rules.yml
  SUCCESS: 1 rules found

Now restart Prometheus with the following command.

sudo systemctl restart prometheus

Configuring Alertmanager

In this section, we will install and configure Alertmanager to send alerts to an email based on Prometheus metrics to notify about endpoints that go down.

For this, to start off, create a new Alertmanager user and update directory permissions with the following commands:

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool

Then, download Alertmanager.

sudo wget https://github.com/prometheus/alertmanager/releases/download/v0.17.0/alertmanager-0.17.0.linux-amd64.tar.gz

Extract the downloaded file. You can do this simply by running the command below.

sudo tar -xvf alertmanager-0.17.0.linux-amd64.tar.gz

Move the extracted binaries to the /usr/local/bin/ directory with the following command.

sudo mv alertmanager-0.17.0.linux-amd64/alertmanager /usr/local/bin/
sudo mv alertmanager-0.17.0.linux-amd64/amtool /usr/local/bin/

Remove downloaded file to save some disk space.

sudo rm -r alertmanager-0.17.0.linux-amd64.tar.gz

Next, create a configuration for Alertmanager.

sudo mkdir /etc/alertmanager

In your favorite editor, create a configuration file with the following command:

sudo nano /etc/alertmanager/alertmanager.yml

Append the following lines into the file.

global:
  smtp_smarthost: 'smtp.domain.net:587'
  smtp_from: 'AlertManager <mailer@localhost.com>'
  smtp_require_tls: true

route:
  group_by: ['instance', 'alert']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: team-1

receivers:
  - name: 'team-1'
    email_configs:
      - to: 'alerts@email.com'

Note that for the above lines, if you don't want TLS or your email server does not support TLS, you may leave the value as false.

Now you'll want to update the file permissions by running the command below.

sudo chown alertmanager:alertmanager -R /etc/alertmanager

Now we need to create a systemd service for Alertmanager. Using your favorite editor, add a configuration file.

sudo nano /etc/systemd/system/alertmanager.service

Append the lines below in the file.

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --web.external-url http://0.0.0.0:9093

[Install]
WantedBy=multi-user.target

Integrate Prometheus using the configuration below.

sudo nano /etc/prometheus/prometheus.yml

Append the lines below in the file.

...
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
...

Next, include the following lines in your /etc/systemd/system/prometheus.service. In your favorite editor, run the command below:

sudo nano /etc/systemd/system/prometheus.service

Append and replace the lines below into the file:

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --web.external-url Error! Hyperlink reference not valid.

[Install]
WantedBy=multi-user.target

Update the systemd file by changing the lines below:

sudo nano /etc/systemd/system/prometheus.service 
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.external-url Error! Hyperlink reference not valid.

Reload daemon and restart the services with the following commands:

systemctl daemon-reload
systemctl restart prometheus
systemctl restart alertmanager

Next, confirm both Alertmanager and Prometheus are properly installed with the following commands.

systemctl status alertmanager
systemctl status prometheus

If all is well, enable alertmanager start at boot with this command.

sudo systemctl enable alertmanager

Conclusion

In this tutorial, we have covered how to configure Prometheus with Blackbox Exporter and Alertmanager to send email alerts. While in most cases Alibaba Cloud will rarely run down, sometimes services running on the platform may experience errors. Nonetheless by implementing the things we have done in this tutorial you can stay on top of things and keep your system up and running as quickly as possible in the event of outages.

Do you have an Alibaba Cloud account? Sign up for an account and try over 40 products for free. This deal is worth up to $1300. Get Started with Alibaba Cloud to learn more.

The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

Community

Using Prometheus and Blackbox Exporter to Monitor Services on Alibaba Cloud

What You Will Need for This Tutorial

Installing Prometheus

Installing Blackbox Exporter

Configure Prometheus

Configuring Alertmanager

Conclusion

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

CloudMonitor

Server Load Balancer

Web Hosting Solution

Managed Service for Prometheus