The image reads, "Open Source Kafka Monitoring" and contains the Kafka logo.

Open Source Monitoring for Kafka

Updated December 2021

A critical component to ensuring Kafka uptime and maintaining peak performance is through monitoring.  Open source monitoring of disk performance, memory usage, CPU, network traffic, and load allow you to identify abnormal metrics in real-time and address potential issues before a performance dip or outage occurs.

In other words, monitoring Apache Kafka is important for both troubleshooting issues and for preventative maintenance.

Furthermore, monitoring Kafka provides assurance to your users that all messages are correctly processed.

 There are several programs available for monitoring Kafka.  Some come with added cost and others are available for free because they are open source.  If you are using Kafka, then you already understand the benefits of open source tools. If you need a refresher check out our post on the benefits of open source tools. 

When setting up Kafka for our clients we almost always use the open source database Elasticsearch and its companion visualization tool Kibana for monitoring.  It is cost effective, has an appealing user interface, and provides the necessary functionality to create a comprehensive and easy-to-use monitoring tool.

Elasticsearch as an Open Source Monitoring Tool for Kafka

If you aren’t familiar with using Elasticsearch as a Kafka monitoring tool, let’s get acquainted.  

#1 Elasticsearch is FREE.  Elasticsearch and its companion tool Kibana are free to download and use.  More information on Elastic’s licensing can be found here.  

#2. Elasticsearch is ADAPTABLE. Elasticsearch works as a multifaceted and adaptable tool that offers horizontally scalable data storage and fast data retrieval.  

#3 Kibana provides attractive DASHBOARDS. Kibana is a sister product to Elasticsearch that delivers customizable and appealing graphics for building dashboards for the Kafka monitoring platform.  

#4 Elasticsearch plays well with ALERTING. Setting up an alerting program for Kafka issues is simplified, as there are several tools that play well with Elasticsearch to provide threshold and Machine Learning based alerting.

Schematic of Information Flow for Open Source Kafka Monitoring

The figure below shows how information flows from the producers to the Kafka cluster(s), and then to the Consumers and Zookeeper.  However, unlike the “normal” use of Kafka where information then flows from the Consumers to your database, here shown as Elasticsearch, information is being sent from Kafka and Zookeeper to database for monitoring the performance of the Kafka cluster(s).

Figure 1. This diagram shows that when monitoring the status of Kafka using Elasticsearch, data is sent to Elasticsearch directly from Kafka.

What metrics should be monitored to track Kafka performance and health?

Both the status of Kafka and the operating system need to be recorded to evaluate system health and performance.  In our article on Kafka Monitoring With Elasticsearch and Kibana we dive into the details of what factors should be included and what “normal” looks like.

Read more about implementing monitoring for Kafka

Have Kafka Questions?

Managed Kafka on your environment with 24/ 7 support.

Consulting support to implement, troubleshoot,
and optimize Kafka.

Fully Managed Kafka
Kafka Consulting Support

Schedule a call with a Kafka solution architect.

Published by

Dattell - Kafka & Elasticsearch Support

Benefit from the experience of our Kafka, Pulsar, Elasticsearch, and OpenSearch expert services to help your team deploy and maintain high-performance platforms that scale. We support Kafka, Elasticsearch, and OpenSearch both on-prem and in the cloud, whether on stand alone clusters or running within Kubernetes. We’ve saved our clients $100M+ over the past six years. Without our guidance companies tend to overspend on hardware or purchase unnecessary licenses. We typically save clients multiples more money than our fees cost in addition to building, optimizing, and supporting fault-tolerant, highly available architectures.

Leave a Reply