How to set up notification recipients

Webhook, Email and Telegram - section “Alerts” → “Recipients”.

Recipients are configured in the GUI: Notifications → Recipients. For recipient types, routes, and setup procedures, see Configuring alerting.

After creating a recipient, choose which alerts to send on that channel: Alerts → Routes (see Routes).

Webhook #

Webhook is used to send messages to notification aggregation services, service-desk, task-manager.

----BEGIN CERTIFICATE---- and ----END CERTIFICATE----

Webhook configuration example #

Step 1. Download demo-webhook #

Load the manifests of the Deployment, Service, VMRule objects into the management cluster using manifest import.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-webhook
  namespace: monitoring
  labels:
    app: demo-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-webhook
  template:
    metadata:
      creationTimestamp:null
      labels:
        app: demo-webhook
    spec:
      terminationGracePeriodSeconds: 30
      automountServiceAccountToken: false
      containers:
      - name: demo-webhook
        image: quay.io/mgoerens/demo-webhook-receiver-alertmanager:0.0.1
        imagePullPolicy: IfNotPresent
        env:
        - name: PORT
          value: "8080"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
        resources:
          limits:
            cpu: 10m
            memory: 30Mi
          requests:
            cpu: 10m
            memory: 30Mi
        securityContext:
          capabilities:
            drop:
              - ALL
          allowPrivilegeEscalation: false

apiVersion: v1
kind: Service
metadata:
  name: demo-webhook
  namespace: monitoring
  labels:
    app: demo-webhook
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: demo-webhook

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
  labels:
    app: demo-webhook
    role: user
  name: example-alert
  namespace: victoria-metrics
spec:
  groups:
    - concurrency: 1
      interval: 1m
      labels:
        app: demo-webhook
        ruletype: alert
      name: example
      rules:
        - alert: ExampleAlert
          expr: vector(1)
      type: prometheus

Step 2: Create a webhook recipient #

In the cluster for which you need to configure alerting, go to Cluster → Alerts →Recipients, create a recipient with type webhook. In the URL field, enter the value: http://demo-webhook.monitoring.svc/webhook. Enter demo-webhook as the recipient name. Leave the remaining fields blank. Save your changes.

Step 3. Set up a route #

In the Cluster → Alerts → Routes section, configure sending alerts via this channel. If necessary, select notification labels that will serve as a filter for sending through a given channel.

Step 4. Check the logs #

In the namespace monitoring of the management cluster, go to view the pod named demo-webhook. Open the container log viewer. If configured correctly, the logs will contain messages from triggered alerts in the cluster.

Telegram #

Step 1. Prepare the bot and recipient #

To set up sending alerts to telegram you need to:

1. Create a telegram bot.

2. Determine the recipient: this can be sent to a user or to a chat group of users.

Step 2. Configure the recipient in the GUI #

Once the bot and recipients are defined, in the GUI, configure the recipient:

Additionally, you can customize the template in the Notification template in text format field. For example, to show trigger time, severity, and description:

{{ define "telegram.message" }}
{{ if eq .Status "firing" }}🔥{{ else }}✅{{ end }} *[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}*

{{ range .Alerts }}
*Severity:* {{ .Labels.severity | title }}
*Summary:* {{ .Annotations.summary }}
*Time:* {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }}

{{ if .Annotations.description }}*Description:* {{ .Annotations.description }}
{{ end -}}
{{ if .Annotations.runbook }}[Runbook]({{ .Annotations.runbook }})
{{ end -}}
{{ end }}
{{ template "telegram.message" . }}

Then the notification will look like:

Send credentials (OPTIONAL): Username/Password pair or Bearer token.

If you need to set up a secure connection, you must additionally enter:

Once you’ve completed setting up a recipient, you need to configure what alerts will be sent on that channel. To do this, in the Cluster section → Alerts → Routes, configure the sending of alerts via this channel. If necessary, select notification labels that will serve as a filter for sending through a given channel.

Email #

Depending on your email provider, you will need to fill out information to set up an email alert. For example, Gmail requires authorization data.

Example of setting up sending to maildev #

Step 1: Prepare maildev configuration #

Prepare the maildev configuration file:

image:
  registry: mirror.gcr.io
  repository: maildev/maildev
  pullPolicy: IfNotPresent
# tag: "2.2.1"

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

ingress:
  annotations:
    cert-manager.io/cluster-issuer: <your parameter value>
  enabled: true
  className: "nginx"

  hosts:
    - host: <your parameter value>
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls:
    - secretName: maildev-ingress-tls
      hosts:
        - <your parameter value>

Step 2. Install maildev #

Install maildev with the resulting configuration file in the namespace monitoring of the management cluster.

helm upgrade --install -n monitoring maildev https://github.com/christianknell/helm-charts/releases/download/maildev-1.3.4/maildev-1.3.4.tgz -f CONFIGURATION-FILE-NAME

If the setting is correct, a Pod with the maildev prefix will appear in the monitoring namespace of the management cluster.

Step 3. Create an email recipient #

In the cluster for which you want to configure sending an alert to email, in the Cluster → Alerts → Recipients section, create a recipient with the “email” type.

Fill in the sender, recipient. Enter the SMTP server: maildev-smtp.monitoring.svc:1025.

If the setting is correct, then maildev will be available at the address specified as the host in the configuration file (https://maildev.management-cluster-name.<base_domain>/#/).

Step 4. Configure the route and check #

In the Cluster → Alerts → Routes section, configure sending alerts via this channel. If necessary, select notification labels that will serve as a filter for sending through a given channel.

If the setting turned out to be correct, then you will see a notification in maildev that looks like this: