Monitoring Kafka cluster performance is crucial for diagnosing system issues and preventing future problems.
Additionally, monitoring dispels any doubt in your users’ minds that all messages are being properly processed. Remember, if you lose a user’s data, it can be extremely difficult to regain trust.
Dattell’s engineers are experts in Kafka cluster design, implementation, and support. In this post we will cover key points of consideration when developing a Kafka monitoring tool.
We recommend using Elasticsearch for Kafka monitoring for four reasons:
- Elasticsearch is free.
- Elasticsearch is highly versatile as a single source of truth throughout any organization.
- Kibana works alongside Elasticsearch to provide customized visualizations for tracking Kafka health.
- Several tools work with Elasticsearch to provide threshold and Machine Learning based alerting.
Figure 1a. Diagram of how data moves between Producers, Kafka, Zookeeper, Consumers, Elasticsearch, and Kibana. This figure depicts how data moves in the normal use of Kafka for moving incoming data to the appropriate database(s).
Figure 1b. This diagram shows that when monitoring the status of Kafka using Elasticsearch data is sent to Elasticsearch directly from Kafka.
MONITORING KAFKA STATUS
Kafka performance can be monitored using information from both Kafka and the Operating System.
First we look at how Kafka status is monitored. Key metrics like the status of the partitions and cluster will be found here.
1.) Kafka rate of receiving data and writing to offset per topic
This figure is showing the rate at which Kafka receives data. Kafka works best when there is a somewhat steady flow of data, not consistent, large spikes.
2.) Kafka consumer number behind current data
This figure is showing how far consumers are behind in the queue.
3.) Kafka partitions in sync
This figure is showing the status of replicas in Kafka.
4.) Kafka partition leaders
This figure is showing the number of partition leaders per Kafka instance. These values should be static unless adding or removing topics.
MONITORING OPERATING SYSTEM STATUS
It is also important to monitor the health of the operating system. Kafka relies heavily on the operating system for its page cache, hard drive space, and other system resources. As an example, excessive page cache misses points to the need for more brokers.
1.) Disk IO
This figure is showing disk performance. Disk performance is usually the limiting factor in Kafka, consistently high values here suggest you want to increase the IOPs (input/output performance) of the hard drives or add more Kafka brokers. Skewed values here hint that your cluster should be re-balanced.
We expect to see steady Memory usage and zero Swap usage. Swap usage is a great indicator that your Kafka cluster is performing poorly.
Kafka doesn’t use much CPU unless your batch size is too small.
This figure is showing the rate of network traffic. Skewed values here hint that your cluster should be re-balanced.
Skewed values here hint that your cluster should be re-balanced.
If you have a highly customized installation of Kafka or find you need more information to troubleshoot your cluster, we recommend enabling JMX on the Kafka brokers. Logstash can pull the JMX data, send it to Elasticsearch, and provide you with any specific visualizations or alerts you may need.
Learn more about our effective and personalized Apache Kafka Training.
Data consulting and implementation services from Dattell provide STRATEGY, ENGINEERING, and PERSPECTIVE to support your organization’s data projects. Our services include custom Data Architecture, Business Analytics, Operational Intelligence, Centralized Reporting, Automation, and Machine Learning. Dattell specializes in Apache Kafka and the Elastic Stack for reliable data collection, storage, and real-time display.