Kafka Monitoring with Elasticsearch

Kafka Monitoring With Elasticsearch and Kibana

Updated August 2020

Identifying the right tool for Kafka monitoring is important because monitoring Kafka cluster performance is crucial for diagnosing system issues.  Additionally, monitoring dispels any doubt in your users’ minds that all messages are being properly processed. Remember, if you lose a user’s data, it can be extremely difficult to regain trust.

In this post we will discuss why Elasticsearch and Kibana are ideal Kafka monitoring tools and provide example graphs that detail what exactly should be monitored.

Kafka Monitoring Tool

We recommend using Elasticsearch for Kafka monitoring for four reasons:

  1. Elasticsearch is free.
  2. Elasticsearch is highly versatile as a single source of truth throughout any organization.
  3. Kibana works alongside Elasticsearch to provide customized visualizations for tracking Kafka health.
  4. Several tools work with Elasticsearch to provide threshold and Machine Learning based alerting.

Data Flow for Kafka Monitoring

The following figures show how the flow of data changes depending on if you are using Kafka as your normally would or if you are monitoring Kafka status.

Normal Use of Kafka

Normal use of Kafka
Figure 1a. Diagram of how data moves between Producers, Kafka, Zookeeper, Consumers, Elasticsearch, and Kibana. This figure depicts how data moves in the normal use of Kafka for moving incoming data to the appropriate database(s).

 

Monitoring the Status of Kafka

Monitoring Kafka.jpg

Figure 1b. This diagram shows the flow of data when using Elasticsearch and Kibana as a Kafka monitoring tool. When monitoring the status of Kafka using Elasticsearch, data is sent to Elasticsearch directly from Kafka.

How to Monitor Kafka

Kafka performance can be monitored using information from both Kafka and the Operating System. First we look at how Kafka status is monitored. Key metrics like the status of the partitions and cluster will be found here.

1.) Kafka rate of receiving data and writing to offset per topic

This figure is showing the rate at which Kafka receives data. Kafka works best when there is a somewhat steady flow of data, not consistent, large spikes,

2.) Kafka consumer number behind current data

This figure tracks how far consumers are behind in the queue.

3.) Kafka partitions in sync

This figure is showing the status of replicas in Kafka.

Kafka Partition Replicas vs. Time

4.) Kafka partition leaders

This figure is showing the number of partition leaders per Kafka instance.  These values should be static unless adding or removing topics.

Kafka Partition Leaders vs. Time

MONITORING OPERATING SYSTEM STATUS

It is also important to monitor the health of the operating system.  Kafka relies heavily on the operating system for its page cache, hard drive space, and other system resources. As an example, excessive page cache misses points to the need for more brokers.

1.) Disk IO

This figure is showing disk performance. Disk performance is usually the limiting factor in Kafka, consistently high values here suggest you want to increase the IOPs (input/output performance) of the hard drives or add more Kafka brokers.  Skewed values here hint that your cluster should be re-balanced.


Average System Disk IO

2.) Memory

We expect to see steady Memory usage and zero Swap usage. Swap usage is a great indicator that your Kafka cluster is performing poorly.


Memory Usage vs Time

3.) CPU

Kafka doesn’t use much CPU unless your batch size is too small.


CPU Usage

4.) Networking

This figure is showing the rate of network traffic. Skewed values here hint that your cluster should be re-balanced.


Rate of Network Traffic

5.) Load

Skewed values here hint that your cluster should be re-balanced.

If you have a highly customized installation of Kafka or find you need more information to troubleshoot your cluster, we recommend enabling JMX on the Kafka brokers.  Logstash can pull the JMX data, send it to Elasticsearch, and provide you with any specific visualizations or alerts you may need.


If you have a highly customized installation of Kafka or find you need more information to troubleshoot your cluster, we recommend enabling JMX on the Kafka brokers.  Logstash can pull the JMX data, send it to Elasticsearch, and provide you with any specific visualizations or alerts you may need.

Summary

Elasticsearch and Kibana can be used to monitor Kafka status.  For effective monitoring both Kafka and the Operating System should be tracked.  Kafka monitoring includes tracking the partition offset, consumer group offset, replicas, and partition leaders.  Operating system monitoring includes  tracking disk IO, memory, CPU, networking, and load. 


Have Kafka Questions?

We offer fully managed Kafka with top performance and exceptional support.

We offer a number of support services including RSA, workshops, and training.

Schedule a call with a Kafka solution architect.