Aug 4, 2021

[Prometheus] Trying Alertmanager Locally

#Monitoring

What is Alertmanager?

Alertmanager sends alerts (e.g., email/Slack) when Prometheus metrics satisfy configured conditions.

Installation

You can spin up Prometheus with Docker Compose using this repo: → Install docker-compose

GitHub - vegasbrianc/prometheus: A docker-compose stack for Prometheus monitoring
A docker-compose stack for Prometheus monitoring. Contribute to vegasbrianc/prometheus development by creating an account on GitHub.
GitHub - vegasbrianc/prometheus: A docker-compose stack for Prometheus monitoring

Start with: 

$ git clone https://github.com/vegasbrianc/prometheus
$ cd prometheus
$ docker-compose up

Ports:

Prometheushttp://localhost:9090/
Alertmanagerhttp://localhost:9093/

Slack notifications

Create a Slack app for Prometheus, enable Incoming Webhooks, and get the webhook URL.

Incoming Webhooks | Slack Developer Docs
Incoming Webhooks は他のアプリから Slack にメッセージを投稿するためのとてもシンプルな方法です。Incoming Webhook を作成すると、メッセージとその他のオプションを含む JSON ペイロードを送るための一意な URL が払い出されます。

Set username, channel, and api_url in alertmanager/config.yml: 

route:
 receiver: 'slack'

receivers:
 - name: 'slack'
   slack_configs:
       - send_resolved: true
         username: '<username>'
         channel: '<channel name>'
         api_url: '<webhook url>'

Alertmanager configuration

Define alert rules like this: 

groups:
- name: <rule name>
  rules:
  - alert: <alert name>
    expr: <PromQL expression>
    for: <duration>
    severity: <severity>
    annotations:
      description: "Alert description"
      summary: "Summary"
alertAlert name
exprPromQL expression
forHow long the condition must hold (s, m, h, d)
severitySeverity (e.g., critical, warning)
annotationsDescription and summary

In this repo, rules live in prometheus/alert.rules.

To force an example alert, set expr to 1 as below: 

groups:
- name: rule-example
  rules:
  - alert: alert-example
    expr: 1
    for: 1m

After editing, restart the Docker containers to apply changes. After a short wait, http://localhost:9090/alerts shows the alert as active.

Image

Status is PENDING or FIRING; FIRING means the alert is firing.

Alertmanager shows fired alerts:

Image

Slack also receives the notification:

Image

References