How to set up alerting based on logs
VMRule and VMAlert for VictoriaLogs.
In the Shturval platform, logs for client clusters and the management cluster are stored in VictoriaLogs. When needed, configure alerting based on cluster logs using custom resources VMRule and VMAlert. Step-by-step instructions with examples follow.
Prerequisites
- Cluster logs are available on the Logs tab cluster dashboard: the local log collection module (shturval-log-collector, Vector) is enabled in the cluster, shturval-metrics-collector is running in the management cluster.
- Alertmanager is configured in the management cluster to deliver notifications (Email, Telegram, Webhook, etc.).
Step 1: Create a VMRule
VMRule sets the rule: under what condition in the logs the alert is triggered.
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
labels:
cluster: clustername
role: user
vmalert-type: vlogs
name: vlogs-rule
namespace: clustername
spec:
groups:
- headers:
- 'ProjectID: 4'
interval: 5m
name: vlogsgroup
type: vlogs
rules:
- alert: vlogsrule
annotations:
description: 'SSC {{ index $labels "objectRef.name" }} was updated by {{ index $labels "user.username" }} in the last {{$for}}'
expr: objectRef.resource:"shturvalserviceconfigs" verb:~"patch|update" | stats by(objectRef.name, objectRef.resource, user.username) count() as updateSSC | filter updateSSC:>1
labels:
source: vlogs
| Parameter | Description |
|---|---|
| metadata.labels.cluster | Name of the cluster where log alerting is required |
| metadata.name | Rule name |
| metadata.namespace | Namespace of the cluster (cluster name) |
| groups.name | Rule group name |
| groups.headers | ProjectID in VictoriaLogs. Only one project can be specified |
| groups.type | vlogs |
| rules.alert | Alert name |
| rules.expr | Expression for notification based on logs (field expr in the rule) |
According to the established rule in expr, notifications will be sent when services (SSC) change in the cluster.
Step 2: Create VMAlert
VMAlert evaluates rules and sends alerts to Alertmanager.
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
labels:
cluster: clustername
name: vlogs
namespace: clustername
spec:
datasource:
url: http://vlogs-logs.clustername:9428
evaluationInterval: 30s
externalLabels:
cluster: clustername
extraArgs:
rule.defaultRuleType: vlogs
notifiers:
- url: http://vmalertmanager-shturval-metrics-collector.victoria-metrics:9093
remoteRead:
url: http://shturval-monitoring-vmselect.victoria-metrics:8481/select/monitoring_id:3/prometheus
remoteWrite:
url: http://shturval-monitoring-vminsert.victoria-metrics:8480/insert/monitoring_id:3/prometheus
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
ruleSelector:
matchLabels:
vmalert-type: vlogs
cluster: clustername
ruleNamespaceSelector: {}
selectAllByDefault: false
| Parameter | Description |
|---|---|
| metadata.namespace | Namespace of the cluster in which alerting is configured based on logs |
| metadata.labels.cluster | Name of the cluster in which log alerting is configured |
| spec.datasource.url | Cluster log source URL: http://vlogs-logs.<cluster_name>:9428 |
| externalLabels.cluster | Name of the cluster in which log alerting is configured |
| spec.remoteRead.url | VictoriaMetricsSelect; instead of monitoring_id specify the value of the cluster annotation with this key |
| spec.remoteWrite.url | VictoriaMetricsInsert; similar to monitoring_id |
| spec.resources | Resources requests and limits (for example: limits cpu 200m, memory 256Mi; requests cpu 100m, memory 128Mi) |
Step 3: Upload to the platform
In the management cluster, import VMRule and VMAlert with manifest import.
Screenshot
Checking alerts
You can check alerts based on logs using a notification on maildev. Set up maildev: create a recipient of type email in the Notifications → Recipients section. More details: example of setting up sending a notification to maildev.