Skip to main content
Version: NG-2.16

Kafka Data Collector

Introduction​

The kafka data collector O11ySource allows user to read from the configured kafka cluster.

Getting Started​

Compatibility​

vuSmartMaps supports the collecting data from the configured topic from a kafka cluster

Data Collection Method​

vuSmartMaps collects the data using telegraf kafka consumer module.

Prerequisites​

Inputs for Configuring Data Source​

  • Name: Unique identifier for the source
  • Client Kafka Bootstrap Server: Enter the client kafka bootstrap server. Separate multiple broker by comma.
  • Client Kafka Topics: Enter the topics to be consumed. Use comma separated for multiple topics.
  • Destination Kafka Topic: Enter the destination kafka topic

Firewall Requirement​

To collect data from this O11ySource, ensure the following ports are opened:

Source IPDestination IPDestination PortProtocolDirection
vuSmartMaps IPClient IPClient Configured portTCPOutbound

*Before providing the firewall requirements, please update the port based on the customer environment.

Configuring the Target​

To verify that your Kafka broker is reachable from your vuSmartMaps cluster, run the following command. Replace <broker_ip> with the actual IP address of your Kafka broker.


nc -vz <broker_ip> 9092

If the connection is successful, you should see an output similar to:


Connection to <broker_ip> 9092 port [tcp/*] succeeded!

Configuration Steps​

  • Enable the O11ySource.
  • Select the sources tab and press the + button to add a new instance that has to be monitored.
  • Provide the required configurations:
  • Name
  • Client Kafka Bootstrap Server
  • Client Kafka Topics
  • Destination Kafka Topic
  • Click Save to close the data source window.

Note: Scaling horizontally by running multiple Telegraf instances improves throughput. Additionally, increasing the number of partitions on your Kafka topic can further enhance performance by allowing parallel consumption. Please refer to user manual to understand how to run mutliple instances.

Metrics Collected​

The metrics are based on the input data collected.