Local log collection module (shturval-log-collector)

About the service #

Purpose #

shturval-log-collector - local log collection module based on Vector: collects logs in the cluster, aggregates and transfers logs to the VictoriaLogs instance in the management cluster.

Vector is an open source tool in Rust. The Vector architecture has three types of components: sources (sources), transforms (transformations) and sinks (receivers). Documentation - vector.dev/docs.

More details: Logging architecture - components and data flows. How to view logs in the platform.

Note: for shturval-log-collector to work correctly, CRD shturval-log-collector-crds must be enabled.

Installation and basic setup #

If installed in a cluster: Services and repositories → Installed services → find Local log collection module → Manage.

If the module is not installed:

1. Services and repositories → Available charts → shturval tab → find the shturval-log-collector-crds chart.
2. If the CRD chart is not yet installed, set shturval-log-collector-crds and wait for the successful status on the Installed services page.
3. On the Available Charts page, select the shturval-log-collector chart and click Install.
4. Select the chart version, namespace logging and control mode Auto.
5. After selecting the chart version, Available configuration parameters for the service (values) will be displayed on the right. If necessary, set parameters in the Service Specification block.

Auto mode is suitable for standard installation and configuration via ShturvalServicePatch: the platform applies the basic chart configuration and applies your patches to customvalues.

Use the Manual mode only for scenarios where you manage logging resources manually, for example, changing system ClusterVectorPipeline. In this mode, automatic application of the supplied configuration is limited, so it is not required for routine addition of log recipients to external systems.

The general procedure is How to deploy a new service.

After installation, check basic operation:

1. Open the cluster → Services and repositories → Installed services → Local log collection module. On the service page, statuses should be successful. For more information about statuses, see the Conditions: ShturvalServiceConfig section.
2. In the Administration section → Custom resources → API group observability.kaasops.io pre-configured ClusterVectorPipeline should be displayed.
3. The Logs tab should be available on the cluster dashboard.

Vector Operator and custom resources #

To configure the local log collection module, use Vector Operator. Supported custom resources:

The basic setting includes the preconfigured ClusterVectorPipeline and Vector.

View in GUI: Administration → Custom Resources→ API group observability.kaasops.io → ClusterVectorPipeline or Vector.

Note: changing the basic setting may lead to errors in log collection, aggregation and analysis. Do not manually change the preconfigured system ClusterVectorPipeline.

A table of pre-configured pipelines, explanations for the spec fields and an example of YAML are on the page Collecting and processing logs (ClusterVectorPipeline).