Alert based on cluster logs

In the “Shturval” platform for clusters (client, management) log storage is implemented in VLogs. Cluster logs are available on the Logs tab of the cluster dashboard.

If necessary, you can configure notification (alerting) based on cluster (client, management) logs. To do this:

In the management cluster, by importing manifests for the cluster where log alerting is required, load custom resources VMRule and VMAlert using your parameter values.

VMRule example
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
  labels:
    cluster: <your parameter value>
    role: user
    vmalert-type: vlogs
  name: <your parameter value>
  namespace: <your parameter value>
spec:
  groups:
    - headers:
        - <your parameter value>
      interval: 5m
      name: <your parameter value>
      type: vlogs
      rules:
        - alert: <your parameter value>
          annotations:
            description: 'SSC {{ index $labels "objectRef.name" }} was updated by {{ index $labels "user.username" }} in the last {{$for}}'
          expr: objectRef.resource:"shturvalserviceconfigs" verb:~"patch|update" | stats by(objectRef.name, objectRef.resource, user.username) count() as updateSSC | filter updateSSC:>1
          labels:
            source: vlogs 

According to the established rule in expr, notifications will be sent when there is a change in the services of the cluster.

ParameterDescriptionData typeExample
metadata.clusterName of the cluster where alerting is requiredlogamstringclustername
metadata.nameRule namestringvlogs
groups.nameRule group namestringvlogsgroup
groups.headersProject (ProjectID) in VLogs. Please note! It is possible to specify only one projectarrayProjectID: 4
rules.alertAlert namestringvlogrule
VMAlert Example
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
  annotations:
  labels:
    cluster: <your parameter value> 
  name: vlogs
  namespace: <your parameter value> 
spec:
  datasource:
    url: <your parameter value>
  evaluationInterval: 30s
  externalLabels:
    cluster: <your parameter value>
  extraArgs:
    rule.defaultRuleType: vlogs
  image: {}
  notifiers:
    - url: http://vmalertmanager-shturval-metrics-collector.victoria-metrics:9093
  remoteRead:
    url: <your parameter value>
  remoteWrite:
    url: <your parameter value>
  resources:
    limits:
      cpu: <your parameter value>
      memory: <your parameter value>
    requests:
      cpu: <your parameter value>
      memory: <your parameter value>
  ruleSelector:
    matchLabels:
      vmalert-type: vlogs
      cluster: <your parameter value>clustername
  ruleNamespaceSelector: {}
  selectAllByDefault: false 
ParameterDescriptionData typeExample
metadata.namespaceName of the cluster namespace in which log alerting is configuredstringclustername
metadata.labels.clusterName of the cluster in which log alerting is configuredstringclustername
spec.datasource.urlCluster Log Source URLstringhttp://vlogs-logs.clustername:9428
externalLabels.clusterName of the cluster in which log alerting is configuredstringclustername
spec.remoteRead.urlVictoria Metric Select URL. Instead of monitoring_id, specify the value of the cluster annotation with this keystringhttp://shturval-monitoring-vmselect.victoria-metrics:8481/select/monitoring_id:3/prometheus
spec.remoteWrite.urlVictoria Metrics Insert URL. Instead of monitoring_id, specify the value of the cluster annotation with this keystringhttp://shturval-monitoring-vminsert.victoria-metrics:8480/insert/monitoring_id:3/prometheus
spec.resourcesResources requests and limitsstring[]objectlimits:
  cpu: 200m
    memory: 256Mi
  requests:
  cpu: 100m
    memory: 128Mi
Screenshot

logs-alert

Click Check and Save.

You can check alerts based on logs using a notification on devmail. Instructions for setting up devmail are located in the Email section on the Processing notification rules page.