Updated December 2021
A critical component to ensuring Kafka uptime and maintaining peak performance is through monitoring. Open source monitoring of disk performance, memory usage, CPU, network traffic, and load allow you to identify abnormal metrics in real-time and address potential issues before a performance dip or outage occurs.
In other words, monitoring Apache Kafka is important for both troubleshooting issues and for preventative maintenance.
Furthermore, monitoring Kafka provides assurance to your users that all messages are correctly processed.
There are several programs available for monitoring Kafka. Some come with added cost and others are available for free because they are open source. If you are using Kafka, then you already understand the benefits of open source tools. If you need a refresher check out our post on the benefits of open source tools.
When setting up Kafka for our clients we almost always use the open source database Elasticsearch and its companion visualization tool Kibana for monitoring. It is cost effective, has an appealing user interface, and provides the necessary functionality to create a comprehensive and easy-to-use monitoring tool.
Elasticsearch as an Open Source Monitoring Tool for Kafka
If you aren’t familiar with using Elasticsearch as a Kafka monitoring tool, let’s get acquainted.
#1 Elasticsearch is FREE. Elasticsearch and its companion tool Kibana are free to download and use. More information on Elastic’s licensing can be found here.
#2. Elasticsearch is ADAPTABLE. Elasticsearch works as a multifaceted and adaptable tool that offers horizontally scalable data storage and fast data retrieval.
#3 Kibana provides attractive DASHBOARDS. Kibana is a sister product to Elasticsearch that delivers customizable and appealing graphics for building dashboards for the Kafka monitoring platform.
#4 Elasticsearch plays well with ALERTING. Setting up an alerting program for Kafka issues is simplified, as there are several tools that play well with Elasticsearch to provide threshold and Machine Learning based alerting.
Schematic of Information Flow for Open Source Kafka Monitoring
The figure below shows how information flows from the producers to the Kafka cluster(s), and then to the Consumers and Zookeeper. However, unlike the “normal” use of Kafka where information then flows from the Consumers to your database, here shown as Elasticsearch, information is being sent from Kafka and Zookeeper to database for monitoring the performance of the Kafka cluster(s).
What metrics should be monitored to track Kafka performance and health?
Both the status of Kafka and the operating system need to be recorded to evaluate system health and performance. In our article on Kafka Monitoring With Elasticsearch and Kibana we dive into the details of what factors should be included and what “normal” looks like.
Have Kafka Questions?
Managed Kafka on your environment with 24/ 7 support.
Consulting support to implement, troubleshoot,
and optimize Kafka.