Collection and processing of logs (ClusterVectorPipeline)

Vector pipelines for collecting and aggregating cluster logs.

ClusterVectorPipeline is a custom resource Kubernetes (API observability.kaasops.io), which specifies the pipeline for collecting and processing logs. Pre-configured ClusterVectorPipeline are created in the cluster along with the installation of the local log collection module.

This page contains a table of pre-configured pipelines, an example of spec and recommendations for custom configurations.

Pre-configured ClusterVectorPipeline #

By default, a set of pre-configured pipelines ClusterVectorPipeline is deployed in the cluster, which collect the main types of logs. Pipelines provide basic coverage:

These pipelines cover typical scenarios: auditing Kubernetes, system events, application events and infrastructure services.

Note: do not manually change the preconfigured system ClusterVectorPipeline - this may disrupt the collection, aggregation and delivery of logs.

ClusterVectorPipeline architecture #

The specification (spec) of the resource ClusterVectorPipeline contains three basic blocks:

• sources - defines data sources;
• transforms - describes the sequence of transformations and filters;
• sinks - sets the endpoints for sending data (VictoriaLogs, external storage, SIEM, syslog, etc.).

Example of a pipeline for logs Kubernetes #

The snippet below illustrates the structure of spec. For the names of the nodes in your cluster, see the ClusterVectorPipeline manifests.

apiVersion: observability.kaasops.io/v1alpha1
kind: ClusterVectorPipeline
metadata:
  name: k8s-logs
  namespace: monitoring
spec:
  # Data source - Kubernetes logs
  sources:
    kubernetes_logs:
      type: kubernetes_logs

  # Chain of transformations
  transforms:
    # Parse the .message field
    message_parser:
      inputs:
        - kubernetes_logs
      source: |
        .message =parse_json(.message) ??
          parse_klog(.message) ??
          .message        
      type: remap

    # Filtering system namespaces
    filter_system_namespaces:
      type: filter
      inputs: [message_parser]
      condition: |
                exists(.kubernetes.namespace_labels."shturval_tech/system-namespace")

    # Filtering user namespaces
    filter_user_namespaces:
      type: filter
      inputs: [message_parser]
      condition: |
                !exists(.kubernetes.namespace_labels."shturval_tech/system-namespace")

Key transformations #

The example above uses platform-specific transformations:

• message_parser:

  • parse it if the field .message has the structure json or klog;
  • otherwise leave it as is.

• filter_system_namespaces and filter_user_namespaces:

  • separate logs by namespace type;
  • focus on the shturval_tech/system-namespace label, which marks system namespaces.

Pipeline Design Guidelines #

When creating your own, ClusterVectorPipeline recommendsXia:

• Use friendly names for pipelines that reflect the source or purpose of the logs.
• Follow a consistent naming style for components (sources, transforms, sinks) to simplify maintenance.
• Build a logical chain of transformations:
  • normalization of timestamps and fields;
  • filtering of unnecessary data;
  • preparing the structure for the target storage.
• Use inputs (inputs) to explicitly describe the sequence (which transform reads data from which component).
• Filter unnecessary data as early as possible to reduce the load on subsequent steps and storage.
• Minimize the number of transformations, leaving only those necessary for your scenarios.
• Avoid duplication of data processing in multiple pipelines if the logic can be expressed in one place.

The ClusterVectorPipeline objects you created can be viewed and edited through the Administration section → Custom Resources by selecting the observability.kaasops.io API group.

Module control mode (Auto / Manual) #

To install the module normally and add log recipients via ShturvalServicePatch, use the Auto mode. In this mode, the configuration supplied with the chart is applied, and user changes to custom values are applied via SSP.

The Manual mode is needed if you create ClusterVectorPipeline manually and want to ensure that these changes are not overwritten by the base configuration. More details can be found in the description of the service shturval-log-collector.