Configure Vector Telemetry Pipeline to Route Data to Victoria from Kafka

Using Omnia, you can deploy Vector as a high-performance data pipeline tool for collecting, transforming, and routing telemetry data from LDMS and OpenManage Enterprise (OME) sources to VictoriaMetrics and VictoriaLogs. The deployment includes Vector-LDMS and Vector-OME pods as Kafka consumers, vmagent-vector as a dedicated write-buffer for metrics, and vlagent-vector as a log forwarding agent.

Vector provides the following components:

  • Vector-LDMS: Kafka consumer for LDMS metrics, consumes from the ldms topic and routes to VictoriaMetrics via vmagent-vector

  • Vector-OME: Kafka consumer for OME telemetry, consumes from ome.* topics and routes metrics to VictoriaMetrics and logs to VictoriaLogs

  • vmagent-vector: Dedicated vmagent instance as a write-buffer between Vector pods and vminsert. Accepts prometheus_remote_write on port 8429, buffers to disk, and forwards to vminsert. Separate from the existing scraper vmagent to isolate failure domains.

  • vlagent-vector: Dedicated VictoriaLogs forwarding agent deployed as a log write-buffer for Vector pods. Accepts JSON Lines on an HTTP endpoint (port 9427), buffers to disk, and forwards to vlinsert. Required for Vector-OME log/event sinks.

For more details on Vector, see Vector Documentation

Vector enables the following data flows:

  • LDMS metrics: LDMS Store (store_avro_kafka) → Kafka ‘ldms’ topic → Vector-LDMS → vmagent-vector → vminsert → VictoriaMetrics

  • OME metrics: OME → Kafka ‘ome.*’ topics → Vector-OME → vmagent-vector → vminsert → VictoriaMetrics

  • OME logs: OME → Kafka ‘ome.*’ topics → Vector-OME → vlagent-vector → vlinsert → VictoriaLogs

Prerequisites

  • Ensure that the provision.yml playbook has been executed successfully with service_kube_control_plane and service_kube_node in the mapping file.

  • Ensure that Kafka is deployed and operational via Strimzi operator.

  • Ensure that VictoriaMetrics cluster mode is deployed with vminsert, vmstorage, and vmselect components.

  • Ensure that VictoriaLogs cluster mode is deployed with vlinsert, vlstorage, and vlselect components (required for Vector-OME logs).

Steps

  1. Specify the following entries in the software_config.json. If any entry is missing, Omnia skips Vector deployment and logs an informational message. For more information, see Input Parameters for Local Repositories.

{"name": "service_k8s", "version": "1.34.1", "arch": ["x86_64"]}

  1. Configure the telemetry_config.yml to enable Vector telemetry bridges:

    Note

    Vector telemetry bridges are controlled by feature flags in telemetry_config.yml. Set telemetry_bridges > vector_ldms > metrics_enabled to enable Vector-LDMS, and telemetry_bridges > vector_ome > metrics_enabled to enable Vector-OME metrics routing. For Vector-OME logs routing, set telemetry_bridges > vector_ome > logs_enabled.

    telemetry_config.yml

    Parameter

    Mandatory/Optional

    Details

    telemetry_sources > idrac > metrics_enabled

    Mandatory

    • Type: Boolean

    • Enable or disable iDRAC metrics collection from Dell PowerEdge servers

    • Collected metrics: temperature, power, fan speed, storage health, CPU/memory errors

    • Data path:

      • iDRAC Receiver -> ActiveMQ -> KafkaPump -> Kafka ‘idrac’ topic

      • iDRAC Receiver -> ActiveMQ -> VictoriaPump -> vmagent -> victoria_metrics

    • Accepted values: true or false

    • Default value: true

    Note

    If iDRAC telemetry is enabled, mysqldb_user, mysqldb_password, and mysqldb_root_password parameters in the omnia_config_credentials.yml file become mandatory.

    Note

    If you want to deploy only Slurm clusters (slurm_custom), set metrics_enabled to false.

    telemetry_sources > idrac > collection_targets

    Mandatory

    • Collection targets define where iDRAC data is sent before Vector processing

    • Supported values: victoria_metrics, kafka

    • Multiple targets: Can specify both [victoria_metrics, kafka]

    • Default: [victoria_metrics, kafka]

    idrac_telemetry_configurations > mysqldb_storage

    Conditional Mandatory

    • MySQL database storage for iDRAC telemetry

    • Purpose: Storage configuration for iDRAC telemetry MySQL database

    • Accepted values: Must be specified in the form of X[Ki|Mi|Gi|Ti|Pi|Ei]

    • Default value: 1Gi

    • Required when: telemetry_sources > idrac > metrics_enabled is true

    telemetry_sources > ldms > metrics_enabled

    Mandatory

    • Type: Boolean

    • Enable or disable LDMS metrics collection from compute nodes

    • Collected metrics: CPU, memory, network, disk metrics

    • Data path: LDMS samplers → LDMS aggregator → store_avro_kafka → Kafka ‘ldms’ topic

    • Accepted values: true or false

    • Default value: true

    telemetry_sources > ldms > collection_targets

    Mandatory

    • LDMS only supports Kafka collection (no direct victoria_metrics path)

    • Vector-LDMS bridge consumes from Kafka and routes to victoria_metrics

    • Supported values: kafka

    • Default: [kafka]

    telemetry_sources > dcgm > metrics_enabled

    Optional

    • Type: Boolean

    • Enable or disable DCGM (NVIDIA Data Center GPU Manager) metrics collection

    • Collected metrics: GPU temperature, utilization, memory, ECC errors, power

    • Requires: NVIDIA GPU driver installed on compute nodes

    • Accepted values: true or false

    • Default value: true

    telemetry_sources > powerscale > metrics_enabled

    Optional

    • Type: Boolean

    • Enable or disable PowerScale metrics collection from Dell PowerScale (OneFS) storage

    • Collected metrics: Storage metrics from Dell PowerScale clusters

    • Requires: CSM Observability (Karavi) values file configured

    • Data path: CSM Metrics PowerScale → OTEL Collector → vmagent(shared) → victoria_metrics

    • Accepted values: true or false

    • Default value: true

    telemetry_sources > powerscale > logs_enabled

    Optional

    • Type: Boolean

    • Enable or disable PowerScale logs collection

    • Accepted values: true or false

    • Default value: true

    telemetry_sources > powerscale > collection_targets

    Conditional Mandatory

    • PowerScale uses dedicated vmagent(shared) (no Kafka, no Vector)

    • Supported values: victoria_metrics, victoria_logs

    • Default: [victoria_metrics, victoria_logs]

    telemetry_sources > ufm > metrics_enabled

    Optional

    • Type: Boolean

    • Enable or disable UFM (NVIDIA UFM InfiniBand Fabric Manager) metrics collection

    • Collected metrics: IB port state, transmit/receive data, error counters, fabric topology

    • Requires: NVIDIA UFM appliance with Prometheus exporter enabled

    • Data path: UFM Prometheus Exporter → vmagent(shared) → victoria_metrics

    • Accepted values: true or false

    • Default value: false

    telemetry_sources > ufm > logs_enabled

    Optional

    • Type: Boolean

    • Enable or disable UFM syslog logs collection

    • Accepted values: true or false

    • Default value: false

    telemetry_sources > ufm > collection_targets

    Conditional Mandatory

    • UFM uses vmagent(shared) for metrics and VLAgent for logs

    • Supported values: victoria_metrics, victoria_logs

    • Default: [victoria_metrics, victoria_logs]

    telemetry_sources > vast > metrics_enabled

    Optional

    • Type: Boolean

    • Enable or disable VAST (Data Storage) metrics collection

    • Collected metrics: IB port state, transmit/receive data, error counters, fabric topology

    • Requires: VAST appliance with Prometheus exporter enabled

    • Data path: Prometheus Exporter → vmagent(shared) → victoria_metrics

    • Accepted values: true or false

    • Default value: false

    telemetry_sources > vast > logs_enabled

    Optional

    • Type: Boolean

    • Enable or disable VAST syslog logs collection

    • Accepted values: true or false

    • Default value: false

    telemetry_sources > vast > collection_targets

    Conditional Mandatory

    • VAST uses vmagent(shared) for metrics and VLAgent for logs

    • Supported values: victoria_metrics, victoria_logs

    • Default: [victoria_metrics, victoria_logs]

    telemetry_bridges > vector_ldms > metrics_enabled

    Optional

    • Type: Boolean

    • Enable or disable Vector-LDMS bridge (Kafka-to-victoria_metrics bridge for LDMS metrics)

    • Purpose: Consume LDMS metrics from Kafka ‘ldms’ topic, transform NERSC schema to Prometheus format, and write to victoria_metrics

    • Data flow: Kafka ‘ldms’ topic → Vector-LDMS → vmagent-vector → victoria_metrics

    • Requires: telemetry_sources > ldms > metrics_enabled = true

    • Accepted values: true or false

    • Default value: true

    telemetry_bridges > vector_ome > metrics_enabled

    Optional

    • Type: Boolean

    • Enable or disable Vector-OME metrics routing (Kafka-to-Victoria bridge for OME metrics)

    • Data flow: Kafka ‘ome.*’ topics → Vector-OME → vmagent-vector (metrics)

    • Requires: OME to be configured with kafka

    • Accepted values: true or false

    • Default value: true

    telemetry_bridges > vector_ome > logs_enabled

    Optional

    • Type: Boolean

    • Enable or disable Vector-OME logs routing

    • Data flow: Kafka ‘ome.*’ topics → Vector-OME → vlagent-vector (logs)

    • Accepted values: true or false

    • Default value: true

    telemetry_bridges > vector_ome > ome_identifier

    Optional

    • Identifier used by Vector-OME for topic identification and routing

    • Internally used to match topics with the prefix (e.g., ^ome\\..*$)

    • Type: String

    • minLength: 1

    • Default value: ome

    • Note: Change only if your OME Kafka topics use a different prefix

    telemetry_sinks > victoria_metrics > persistence_size

    Conditional Mandatory

    • Storage per vmstorage pod PVC

    • Important: Total VictoriaMetrics storage depends on deployment mode:

      • Single-node mode: Total storage = persistence_size * 1 pod

      • Cluster mode: Total storage = persistence_size * 3 vmstorage pods

    • Example (cluster): 8Gi * 3 = 24Gi total VictoriaMetrics storage

    • Accepted values: Must be specified in the form of X[Ki|Mi|Gi|Ti|Pi|Ei]

    • Default value: 8Gi (results in 24Gi total storage for cluster mode)

    telemetry_sinks > victoria_metrics > retention_period

    Conditional Mandatory

    • Metric retention period in hours

    • Default: 168 (7 days)

    telemetry_sinks > victoria_metrics > additional_metric_remote_write_endpoints

    Optional

    • Additional remote write endpoints for metrics (optional)

    • Purpose: Send metrics to external VictoriaMetrics instances in addition to Omnia-managed VictoriaMetrics

    • Format: List of endpoint objects with ‘url’ field (must start with http:// or https://)

    • TLS: Set ‘tls_insecure_skip_verify: true’ to skip TLS certificate verification

    • Default: [] (empty — only Omnia VictoriaMetrics receives metrics)

    • Example: - url: https://external-metrics-server:8480/insert/0/prometheus/api/v1/write

      tls_insecure_skip_verify: false

    telemetry_sinks > victoria_logs > storage_size

    Conditional Mandatory

    • Storage per vlstorage pod PVC

    • Total storage = storage_size × 3 vlstorage pods

    • Accepted values: Must be specified in the form of X[Ki|Mi|Gi|Ti|Pi|Ei]

    • Default value: 8Gi (results in 24Gi total storage)

    • Sizing formula: (140 MB/day × retention_days × node_count) / 3 replicas

    Warning

    Storage under-provisioning can lead to data loss before the retention period is reached. Calculate storage requirements based on expected log volume and retention needs.

    telemetry_sinks > victoria_logs > retention_period

    Conditional Mandatory

    • Log retention period in hours

    • Type: Integer (hours)

    • Accepted values: 24-8760 (1 day to 1 year)

    • Default: 168 (7 days)

    • Note: Retention is global and applies to all log streams uniformly. Deletion occurs asynchronously during background merge operations.

    Note

    VictoriaLogs does not return an error when log entries with timestamps outside the configured retention window are submitted. Log entries will be automatically removed from VictoriaLogs after the retention period.

    telemetry_sinks > victoria_logs > additional_log_write_endpoints

    Optional

    • Additional remote write endpoints for logs (optional)

    • Purpose: Send logs to external VictoriaLogs instances in addition to Omnia-managed VictoriaLogs

    • Format: List of endpoint objects with ‘url’ field (must start with http:// or https://)

    • TLS: Set ‘tls_insecure_skip_verify: true’ to skip TLS certificate verification

    • Default: [] (empty — only Omnia VictoriaLogs receives logs)

    • Example: - url: https://external-logs-server:9481/internal/insert

      tls_insecure_skip_verify: false

    telemetry_sinks > kafka > persistence_size

    Conditional Mandatory

    • Storage per Kafka pod PVC

    • Total = persistence_size × 6 pods (3 brokers + 3 controllers)

    • Accepted values: Must be specified in the form of X[Ki|Mi|Gi|Ti|Pi|Ei]

    • Default value: 8Gi (results in 48Gi total storage)

    • The default 8Gi size is suitable for small clusters (typically fewer than 5 nodes). For larger clusters, you should increase the persistence_size and adjust log_retention_hours and log_retention_bytes based on expected data volume and cluster size.

    Caution

    Ensure that the Kafka broker settings persistence_size, log_retention_hours, and log_retention_bytes are configured based on your data retention requirements. If the persistent volume reaches its capacity before logs are deleted according to the log retention period configured, Kafka brokers may run out of disk space. For more details on managing Kafka log retention and cleanup policies, see Managing Kafka logs with delete and compact policies.

    telemetry_sinks > kafka > log_retention_hours

    Conditional Mandatory

    • Log retention period in hours

    • Default: 168 (7 days)

    telemetry_sinks > kafka > log_retention_bytes

    Conditional Mandatory

    • Maximum size of Kafka logs (in bytes) before deletion

    • Default: -1 (unlimited)

    telemetry_sinks > kafka > log_segment_bytes

    Conditional Mandatory

    • Maximum size of Kafka log segments (in bytes)

    • Default: 1073741824 (1 GB)

    telemetry_sinks > kafka > topic_partitions

    Conditional Mandatory

    • Topic partitions per source (object format, not array)

    • Format: {topic_name: partition_count}

    • Required when: Source has kafka in collection_targets

    • Allowed topics: idrac, ldms only

    • Default partition counts: idrac=1, ldms=2

    • Example: {idrac: 1, ldms: 2}

    ldms_configurations > agg_port

    Conditional Mandatory

    • Aggregator port on service K8s cluster

    • Valid range: 6001-6100

    • Default: 6001

    ldms_configurations > store_port

    Conditional Mandatory

    • Store daemon port on service K8s cluster

    • The port can be the same as LDMS aggregator port

    • Valid range: 6001-6100

    • Default: 6001

    ldms_configurations > sampler_port

    Conditional Mandatory

    • Sampler port on compute nodes

    • Valid range: 10001-10100

    • Default: 10001

    ldms_configurations > sampler_plugins

    Mandatory

    • Sampler plugins define which metrics to collect from compute nodes

    • Parameters:

      • plugin_name: Name of the LDMS sampler plugin

      • config_parameters: Plugin-specific configuration (as a single string)

      • activation_parameters: Collection schedule in MICROSECONDS

        Format: interval=<microseconds> offset=<microseconds> Example: interval=30000000 (30 seconds)

    • Default plugins:

      • meminfo: Memory usage statistics (free, used, buffers, cached)

      • procstat2: Process statistics (CPU, memory, I/O per process)

      • vmstat: Virtual memory statistics (paging, swapping, memory pressure)

      • loadavg: System load average (1, 5, and 15 minute averages)

      • procnetdev2: Network interface statistics (bytes, packets, errors, drops per interface)

    • Default activation_parameters: interval=30000000 (30 seconds for all plugins except procnetdev2 which includes offset=0)

    powerscale_configurations > otel_collector_storage_size

    Conditional Mandatory

    • PVC size for OTEL Collector metric batching and buffering

    • Accepted values: Must be specified in the form of X[Ki|Mi|Gi|Ti|Pi|Ei]

    • Default value: 5Gi

    powerscale_configurations > csm_observability_values_file_path

    Conditional Mandatory

    Note

    In the values.yaml file, only set karaviMetricsPowerscale -> enabled: true. Set the following parameters to false: karaviMetricsPowerflex -> enabled=false, karaviMetricsPowerstore -> enabled=false, karaviMetricsPowerscaleauthorization.-> enabled=false, karaviMetricsPowermax -> enabled=false.

    Note

    Update the isiAuthType in the values.yaml file based on the current auth type setting. To check the current auth type setting, use the command isi http settings view.

    Note

    For CSI PowerScale health metrics, enable controller > healthMonitor > enabled: true and node > healthMonitor > enabled: true in the CSI PowerScale values.yaml (https://raw.githubusercontent.com/dell/helm-charts/csi-isilon-2.15.0/charts/csi-isilon/values.yaml).

    ufm_configuration > ufm_endpoint

    Conditional Mandatory

    • UFM appliance IP address or hostname

    • Required when: telemetry_sources > ufm > metrics_enabled is true

    • Example: 172.20.44.180 or ufm.example.com

    • Default value: "

    ufm_configuration > ufm_metrics_port

    Optional

    • UFM Prometheus exporter port

    • Default value: 9001 (UFM default Prometheus port)

    ufm_configuration > scrape_interval

    Optional

    • Prometheus scrape interval for UFM metrics

    • Accepted values: Prometheus duration format (e.g., 15s, 30s, 1m)

    • Default value: 30s

    ufm_configuration > scrape_timeout

    Optional

    • Prometheus scrape timeout (must be <= scrape_interval)

    • Accepted values: Prometheus duration format (e.g., 10s, 15s)

    • Default value: 15s

    ufm_configuration > tls_mode

    Optional

    • TLS mode for connecting to UFM Prometheus endpoint

    • Accepted values: self_signed, ca_signed

    • self_signed: Skip TLS verification (insecure_skip_verify=true)

    • ca_signed: Use CA certificate for TLS verification

    • Default value: self_signed

    ufm_configuration > ufm_ca_cert_path

    Optional

    • Path to CA certificate file for UFM TLS verification

    • Required when: tls_mode is ca_signed

    • Must be a valid PEM-format certificate file

    • Default value: " (empty — not used when tls_mode is self_signed)

    ufm_configuration > auth_mode

    Optional

    • Authentication mode for UFM Prometheus endpoint

    • Accepted values: basic, none

    • basic: Use ufm_username/ufm_password from omnia_config_credentials.yml

    • none: No authentication (UFM endpoint is open)

    • Default value: basic

    vast_configuration > vast_endpoint

    Conditional Mandatory

    • VAST appliance IP address or hostname

    • Required when: telemetry_sources > vast > metrics_enabled is true

    • Example: 172.20.44.180 or vast.example.com

    • Default value: "

    vast_configuration > vast_metrics_port

    Optional

    • VAST Prometheus exporter port

    • Default value: 9001 (VAST default Prometheus port)

    vast_configuration > scrape_interval

    Optional

    • Prometheus scrape interval for VAST metrics

    • Accepted values: Prometheus duration format (e.g., 15s, 30s, 1m)

    • Default value: 30s

    vast_configuration > scrape_timeout

    Optional

    • Prometheus scrape timeout (must be <= scrape_interval)

    • Accepted values: Prometheus duration format (e.g., 10s, 15s)

    • Default value: 15s

    vast_configuration > tls_mode

    Optional

    • TLS mode for connecting to VAST Prometheus endpoint

    • Accepted values: self_signed, ca_signed

    • self_signed: Skip TLS verification (insecure_skip_verify=true)

    • ca_signed: Use CA certificate for TLS verification

    • Default value: self_signed

    vast_configuration > vast_ca_cert_path

    Optional

    • Path to CA certificate file for VAST TLS verification

    • Required when: tls_mode is ca_signed

    • Must be a valid PEM-format certificate file

    • Default value: " (empty — not used when tls_mode is self_signed)

    vast_configuration > auth_mode

    Optional

    • Authentication mode for VAST Prometheus endpoint

    • Accepted values: basic, none

    • basic: Use vast_username/vast_password from omnia_config_credentials.yml

    • none: No authentication (VAST endpoint is open)

    • Default value: basic

  2. For Vector-LDMS, ensure that LDMS is configured and the store_avro_kafka plugin is producing to the Kafka ldms topic. Vector-LDMS consumes from this topic.

  3. For Vector-OME, ensure that OME is configured externally and producing to Kafka ome.* topics via the external mTLS listener (port 9094). Run the external_kafka_connect_details.yml playbook to configure OME connectivity.

  4. Run the provision.yml playbook to deploy Vector components:

    cd /opt/omnia/telemetry
ansible-playbook provision.yml

The playbook deploys the following components based on the configured feature flags:

  • vmagent-vector: Dedicated vmagent instance as a write-buffer (deployed when vectro_ome_metrics_enabled or vector_ldms_metrics_enabled is set to true)

  • Vector-LDMS: Kafka consumer for LDMS metrics (deployed when telemetry_bridges > vector_ldms > metrics_enabled = true)

  • Vector-OME: Kafka consumer for OME telemetry (deployed when telemetry_bridges > vector_ome > metrics_enabled = true or telemetry_bridges > vector_ome > logs_enabled = true)

  • vlagent-vector: VictoriaLogs forwarding agent for logs (deployed when telemetry_bridges > vector_ome > logs_enabled = true)

  • Kafka topics and ACLs: For OME topics (deployed when telemetry_bridges > vector_ome > metrics_enabled = true or telemetry_bridges > vector_ome > logs_enabled = true)

  • KafkaUser resources: For Vector-OME mTLS credentials (deployed when telemetry_bridges > vector_ome > metrics_enabled = true or telemetry_bridges > vector_ome > logs_enabled = true)

Note

Vector-LDMS reuses the existing kafkapump KafkaUser for mTLS credentials. Vector-OME requires a new KafkaUser (vector-ome-user) because OME is an external producer with a different security domain.

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.