Integrate OpenManage Enterprise with Omnia Kafka Pipeline for Secure Telemetry Data Streaming

This section describes how to configure OpenManage Enterprise to securely stream metrics into the Service Kubernetes clusters using mutual TLS (mTLS).

Note

To consume OpenManage Enterprise telemetry from Kafka topics and route to VictoriaMetrics and VictoriaLogs using Vector, see Configure Vector Telemetry Pipeline to Route Data to Victoria from Kafka.

Prerequisites

• Ensure that the pod_external_ip_range parameter is set in the omnia_config.yml file for the Service Kubernetes cluster and it is reachable from the OpenManage Enterprise appliance network.

• Ensure Kafka is installed and running in the Service Kubernetes cluster.

• External access to Kafka is available through the following LoadBalancer ports:

  • 9094 for ingesting and querying data.

• Ensure that the nodes are discovered in OpenManage Enterprise before configuring telemetry streaming.

Steps

1. Run the following playbook to retrieve the Kafka connection details and TLS certificates from the Service Kubernetes cluster:

   cd /omnia/utils
   ansible-playbook external_kafka_connect_details.yml
   

   The external_kafka_connect_details.yml playbook performs the following:

   • Retrieves the Kafka LoadBalancer external IP.

   • Extracts the server CA certificate and client certificates/keys from the telemetry namespace.

   • Writes the Kafka endpoint and TLS file locations to /opt/omnia/telemetry/external_kafka_connect_details.yml.

   • Saves the TLS files in /opt/omnia/telemetry/external_kafka/:

   • ca.crt (server certificate)

   • user.crt (client certificate)

   • user.key (client key)

   Note

   If OpenManage Enterprise is installed on a different system than the OIM host, copy ca.crt to that system before uploading it in the UI.

2. Create a client certificate in .pfx format for mTLS by running the following command. Provide a passphrase when prompted:

   cd /opt/omnia/telemetry/external_kafka/
   openssl pkcs12 -export -out user.pfx -inkey user.key -in user.crt
   

   

3. In OpenManage Enterprise, navigate to Configuration > Remote Connectivity, and select Enable.

   

4. In the Kafka Connectivity wizard, select the Enable Kafka Connectivity check box to turn on Kafka integration.

5. In the OME Identifier field, enter a unique identifier to be used as the topic prefix for publishing OpenManage Enterprise metrics.

6. In the Kafka Bootstrap Server field, enter the Kafka external endpoint displayed by the playbook, along with the port number.

   Example:

   <Kafka LoadBalancer External IP>:<Port Number>
   

7. From the Authentication Mode list, select SSL.

8. Under Server Certificate Validation, select the Enable Server Certificate Validation check box, and upload ca.crt from /opt/omnia/telemetry/external_kafka/.

9. Under Client Certificate Configuration, select the Enable Client Certificate for mTLS check box, and upload the client certificate (user.pfx) generated in Step 2. Enter the password or passphrase used to generate the certificate, and click Next.

   

10. On the Data Configuration page, select the metrics to stream to the Omnia Kubernetes Service cluster, and click Next.

11. On the Group Configuration page, select the devices and device groups from which metrics should be collected, and click Next.

12. Navigate to Configuration > Remote Connectivity and verify the following:

    • Under Connectivity, a green check mark next to Connected since indicates successful connectivity between OpenManage Enterprise and the Omnia Service Kubernetes cluster.

    • Under Transfer status, green check marks next to each metric indicate that the selected metrics are being successfully transmitted without errors.

Verify OME Telemetry Data in Kafka

To verify that OME telemetry data is being successfully published to the OME Kafka topics, do the following:

Note

Ensure that the nodes are discovered in OpenManage Enterprise before configuring telemetry streaming.

1. Log in to Service Kubernetes Control plane.

2. Set the required variables using the following command:

   KAFKA_LB_IP=<external IP of bridge-bridge-lb service>
   TOPIC=<OME Topic Name>
   GROUP=ome-consumer-group
   INSTANCE=<a-unique-instance-name>
   

2. Create a Kafka consumer using the following command:

   curl -s -X POST "http://$KAFKA_LB_IP:8080/consumers/$GROUP" \
     -H 'content-type: application/vnd.kafka.v2+json' \
     -d '{
           "name": "'"$INSTANCE"'",
           "format": "json",
           "auto.offset.reset": "earliest"
         }'
   

3. Subscribe the consumer to the telemetry topic using the following command:

   curl -s -X POST "http://$KAFKA_LB_IP:8080/consumers/$GROUP/instances/$INSTANCE/subscription" \
     -H 'content-type: application/vnd.kafka.v2+json' \
     -d '{"topics": ["'"$TOPIC"'"]}'
   

4. Consume messages from the topic using the following command:

   while true; do
     curl -s -X GET "http://$KAFKA_LB_IP:8080/consumers/$GROUP/instances/$INSTANCE/records" \
       -H 'accept: application/vnd.kafka.json.v2+json' | jq '.'
     sleep 2
   done
   

5. (Optional) Cleanup the consumer using the following command:

   curl -s -X DELETE "http://$KAFKA_LB_IP:8080/consumers/$GROUP/instances/$INSTANCE"
   

Note

• From beginning: Ensure "auto.offset.reset": "earliest" when creating the consumer if you want existing data.

• Message format: Use "format": "json" only if producers publish JSON. Otherwise use "binary" and decode base64 payloads.

• Throughput: Adjust polling interval; bridge returns empty array when no new records.

• 404/409 errors: 404 usually means wrong group/instance name; 409 means already subscribed.

Verify OME Telemetry Data in VictoriaMetrics

To verify that OME telemetry data is being successfully routed from Kafka to VictoriaMetrics using Vector, do the following:

1. Log in to the VictoriaMetrics UI.

2. Navigate to the Explore tab.

3. Run the following queries to verify OME metrics are being received:

   • last_over_time({source_subsystem="ome", type="healty"}[24h]): This query fetches health metrics from OME.

     

   • last_over_time({source_subsystem="ome", type="inventory"}[24h]): This query fetches inventory metrics from OME.

     

Note

Note that source_subsystem=ome is coming from the ome_identifier that the user has given in the telemetry_config.yml input file and the suffix after the dot (i.e., health, inventory, auditlogs) is coming from OME.

4. Verify that OME-related metrics are displayed in the results.

Note

Ensure that the Vector-OME bridge is enabled in telemetry_config.yml (telemetry_bridges > vector_ome > metrics_enabled: true) for metrics data to flow from Kafka to VictoriaMetrics.

Verify OME Telemetry Data in VictoriaLogs

To verify that OME telemetry data is being successfully routed from Kafka to VictoriaLogs using Vector, do the following:

1. Log in to the VictoriaLogs UI.

2. Navigate to the Select tab.

3. In the query field, run the following query to filter for OME logs:

   • _msg_topic:ome.auditlogs

   

4. Verify that OME-related logs are displayed in the results.

Note

Ensure that the Vector-OME bridge is enabled in telemetry_config.yml (telemetry_bridges > vector_ome > logs_enabled: true) for logs data to flow from Kafka to VictoriaLogs.

If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.