This page looks best with JavaScript enabled

PMM2 Mattermost and AlertManager

 ·  🎃 kr0m

In a previous article, I explained how to send AlertManager notifications via Telegram. However, if we are paranoid (which is never a bad thing) about whether Telegram is spying on us, we can set up a Mattermost server and send notifications to it instead. We can find the plugin link on the Mattermost website.

Since we are setting up the entire PMM2 monitoring infrastructure with Docker, we will also set up our Mattermost server on it. The first step is to install DockerCompose.
https://github.com/docker/compose/releases/

wget https://github.com/docker/compose/releases/download/1.26.0-rc4/docker-compose-Linux-x86_64 -O /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Clone the Mattermost-Docker repository:

git clone https://github.com/mattermost/mattermost-docker.git
cd mattermost-docker

Mattermost consists of three parts: app db, app, and web. But before building the images, we define our net as external and use it in the construction of the apps. In the web app, we also change the port so that it does not conflict with that of pmm-server:

vi docker-compose.yml

networks:
  pmm-net:
    external: true

  db:
    networks:
      - pmm-net
      
  app:
    networks:
      - pmm-net

  web:
    networks:
      - pmm-net
    ports:
      - "180:80"
      - "1443:443"

Build the images:

docker-compose build
docker images

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
mattermost-docker_web        latest              68d72a5edf18        3 minutes ago       21.4MB
mattermost-docker_app        latest              cdc0bf418ca8        3 minutes ago       295MB
mattermost-docker_db         latest              4b411ff22962        4 minutes ago       589MB

Create the necessary directories for Mattermost:

mkdir -pv ./volumes/app/mattermost/{data,logs,config,plugins,client-plugins}
chown -R 2000:2000 ./volumes/app/mattermost/

We deploy Mattermost CTs:

docker-compose up -d
docker ps

CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                    PORTS                                        NAMES  
a7453fab24b8        mattermost-docker_db       "/entrypoint.sh post…"   43 minutes ago      Up 43 minutes (healthy)   5432/tcp                                     mattermost-docker_db_1  
db5b18e6d358        mattermost-docker_app      "/entrypoint.sh matt…"   43 minutes ago      Up 43 minutes (healthy)   8000/tcp                                     mattermost-docker_app_1  
e993144658fb        mattermost-docker_web      "/entrypoint.sh"         43 minutes ago      Up 43 minutes (healthy)   0.0.0.0:180->80/tcp, 0.0.0.0:1443->443/tcp   mattermost-docker_web_1

We access the web interface:
http://mattermost.alfaexploit.com:180/signup_email

We create a team:


We now have our running Mattermost:

We configure the theme, user profile, and more.

We create a private alert notification channel:


We create a bot account that the plugin will use to publish alerts, we access the System Console:

Integrations > Bot Accounts > Enable Bot Account Creation

We save and click on the link *Integrations > Bot Accounts,*Add Bot Account:



To integrate Alertmanager notifications into Mattermost, we need to download a plugin:

wget https://github.com/cpanato/mattermost-plugin-alertmanager/releases/download/v0.0.1/com.cpanato.alertmanager-0.0.1.tar.gz

By default, Mattermost does not allow uploading plugins, so we modify the configuration and restart the CT:

vi ./volumes/app/mattermost/config/config.json

    "PluginSettings": {
        "Enable": true,
        "EnableUploads": true,
docker stop mattermost-docker_app_1 && docker start mattermost-docker_app_1

We install the plugin.

System Console > Plugins > Management > Upload Plugin:

We enable and configure the plugin:

System Console > Plugins > AlertManager
Team: sysadmins
Channel: alerts
User: alertmanager
Token: GENERATE
AlertManager URL: http://pmm-alertmanager:9093

We configure AlertManager to send notifications to Mattermost via webhook:

webhook: https://SITEURL/plugins/com.cpanato.alertmanager/api/webhook?token=TOKEN

If we have followed the previous article, the configuration will be as follows:

vi prometheusConf/alertmanager.conf

global:

route:

  # Default receiver por defecto: Alertmanager parece ignorar este parámetros, como lo ignora lo definimos mas abajo como un receiver vacío
  receiver: 'default'

  # No agrupes las alertas, de este modo los resolved se ven claros en Telegram
  #group_by: [node_name]

  # Tiempo de espera desde que se recibe por primera vez la alerta de grupo hasta que se envia(solo aplicable si se agrupan alertas)
  #group_wait: 30s

  # Tiempo de espera entre notificaciones de una alarma de grupo previamente recibida(solo aplicable si se agrupan alertas)
  #group_interval: 6m

  # Cada cuanto reenviar las alertas, a esto hay que sumarle el tiempo del alert de Prometheus
  # Si el alert tiene "for: 6m" y repeat_interval 1m -> la alerta se enviará cada 6+1=7m
  repeat_interval: 1m

  # Additional receivers
  routes:

  # Telegram: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'telegram'
    continue: true

  # Mattermost: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'mattermost'
    continue: true

  # Asterisk: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'asterisk-api'
    continue: true

inhibit_rules:
- source_match:
    alertname: 'BrokenNodeExporter'
  target_match_re:
    severity: '.*'
  equal: ['node_name']

receivers:
- name: 'default'

- name: 'telegram'
  webhook_configs:
  - url: http://pmm-telegrambot:9087/alert/CHATID
    send_resolved: true

- name: 'mattermost'
  webhook_configs:
  - url: http://mattermost.alfaexploit.com:180/plugins/com.cpanato.alertmanager/api/webhook?token=RxjZNyWPMo0rIZSTvq4mKe2zL4c9L1QQ
    send_resolved: true
    
- name: 'asterisk-api'
  webhook_configs:
  - url: http://pmm-asterisk.alfaexploit.com:4444/alertSysAdmins
    send_resolved: false

Restart the CT:

docker stop pmm-alertmanager && docker start pmm-alertmanager

Trigger an alarm and see the notification in Mattermost:

If we want to send notifications to different teams/channels, we must first create them, then edit the plugin to have another name and re-import it.

Unzip the plugin:

tar xvzf com.cpanato.alertmanager-0.0.1.tar.gz

x com.cpanato.alertmanager/  
x com.cpanato.alertmanager/plugin.json  
x com.cpanato.alertmanager/server/  
x com.cpanato.alertmanager/server/dist/  
x com.cpanato.alertmanager/server/dist/plugin-darwin-amd64  
x com.cpanato.alertmanager/server/dist/plugin-windows-amd64.exe  
x com.cpanato.alertmanager/server/dist/plugin-linux-amd64  
x com.cpanato.alertmanager/assets/  
x com.cpanato.alertmanager/assets/.gitkeep  
x com.cpanato.alertmanager/assets/alertmanager-1.png  
x com.cpanato.alertmanager/assets/alertmanager-3.png  
x com.cpanato.alertmanager/assets/alertmanager-2.png
cp -rp com.cpanato.alertmanager com.cpanato.alertmanager2

Edit it so that it does not overwrite the previous one:

vi com.cpanato.alertmanager2/plugin.json

 "id": "com.cpanato.alertmanager2",
 "name": "AlertManager2",

Re-generate the compressed file:

tar czvf com.cpanato.alertmanager2-0.0.1.tar.gz com.cpanato.alertmanager2

Import it as we did before, now we see a second plugin:

Reconfigure alertmanager:

vi prometheusConf/alertmanager.conf

global:

route:

  # Default receiver por defecto: Alertmanager parece ignorar este parámetros, como lo ignora lo definimos mas abajo como un receiver vacío
  receiver: 'default'

  # No agrupes las alertas, de este modo los resolved se ven claros en Telegram
  #group_by: [node_name]

  # Tiempo de espera desde que se recibe por primera vez la alerta de grupo hasta que se envia(solo aplicable si se agrupan alertas)
  #group_wait: 30s

  # Tiempo de espera entre notificaciones de una alarma de grupo previamente recibida(solo aplicable si se agrupan alertas)
  #group_interval: 6m

  # Cada cuanto reenviar las alertas, a esto hay que sumarle el tiempo del alert de Prometheus
  # Si el alert tiene "for: 6m" y repeat_interval 1m -> la alerta se enviará cada 6+1=7m
  repeat_interval: 1m

  # Additional receivers
  routes:

  # Telegram: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'telegram'
    continue: true

  # Mattermost: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'mattermost'
    continue: true

  # Mattermost2: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'mattermost2'
    continue: true

  # Asterisk: Solo alarmas criticas
  - match:
      severity: critical
    receiver: 'asterisk-api'
    continue: true

inhibit_rules:
- source_match:
    alertname: 'BrokenNodeExporter'
  target_match_re:
    severity: '.*'
  equal: ['node_name']

receivers:
- name: 'default'

- name: 'telegram'
  webhook_configs:
  - url: http://pmm-telegrambot:9087/alert/-379448070
    send_resolved: true

- name: 'mattermost'
  webhook_configs:
  - url: http://grafanakr0m:180/plugins/com.cpanato.alertmanager/api/webhook?token=RxjZNyWPMo0rIZSTvq4mKe2zL4c9L1QQ
    send_resolved: true

- name: 'mattermost2'
  webhook_configs:
  - url: http://grafanakr0m:180/plugins/com.cpanato.alertmanager2/api/webhook?token=SkgI49ezMRRResXAdQ-wCtFZ9RAMSmzB
    send_resolved: true

- name: 'asterisk-api'
  webhook_configs:
  - url: http://pmm-asterisk.alfaexploit.com:4444/alertSysAdmins
    send_resolved: false

Restart AlertManager:

docker stop pmm-alertmanager && docker start pmm-alertmanager

If we trigger an alarm, we will see it in both teams/channels:

We can also check the alerts from Mattermost:

/alertmanager alerts - to list the existing alerts  
/alertmanager silences - to list the existing silences  
/alertmanager expire_silence <SILENCE_ID> - to expire the specified silence  
/alertmanager status - to list the version and uptime of the Alertmanager instance  
/alertmanager help - to get this help

NOTE: The bot only highlights the channel where it sent the alert, it does not show a list.

DEBUG

We can see the logs of the CTs with:

docker logs -f mattermost-docker_db_1
docker logs -f mattermost-docker_app_1
docker logs -f mattermost-docker_web_1

The app logs are located under the directory volumes/app/APP_NAME/logs/, for example, to see the logs of the mattermost app:

tail -f mattermost-docker/volumes/app/mattermost/logs/mattermost.log

If you liked the article, you can treat me to a RedBull here