Kafka Monitoring With Elasticsearch and Kibana

Monitoring Kafka cluster performance is crucial for diagnosing system issues and preventing future problems.

Additionally, monitoring dispels any doubt in your users’ minds that all messages are being properly processed. Remember, if you lose a user’s data, it can be extremely difficult to regain trust.

Dattell’s engineers are experts in Kafka cluster design, implementation, and support. In this post we will cover key points of consideration when developing a Kafka monitoring tool.

We recommend using Elasticsearch for Kafka monitoring for four reasons:

  1. Elasticsearch is free.
  2. Elasticsearch is highly versatile as a single source of truth throughout any organization.
  3. Kibana works alongside Elasticsearch to provide customized visualizations for tracking Kafka health.
  4. Several tools work with Elasticsearch to provide threshold and Machine Learning based alerting.

Normal use of Kafka
Figure 1a. Diagram of how data moves between Producers, Kafka, Zookeeper, Consumers, Elasticsearch, and Kibana. This figure depicts how data moves in the normal use of Kafka for moving incoming data to the appropriate database(s).

Monitoring Kafka.jpg

Figure 1b. This diagram shows that when monitoring the status of Kafka using Elasticsearch data is sent to Elasticsearch directly from Kafka.

 

Managed Kafka With 99.99% Uptime Guarantee

Dattell’s Kafka as a Service is a fully managed, high-throughput, distributed messaging system built on your cloud instances or on-prem servers, providing enhanced security and reduced latency for your data.

MONITORING KAFKA STATUS

Kafka performance can be monitored using information from both Kafka and the Operating System.

First we look at how Kafka status is monitored. Key metrics like the status of the partitions and cluster will be found here.

1.) Kafka rate of receiving data and writing to offset per topic

This figure is showing the rate at which Kafka receives data. Kafka works best when there is a somewhat steady flow of data, not consistent, large spikes.

monitor1aa

2.) Kafka consumer number behind current data

This figure is showing how far consumers are behind in the queue.

monitor1b

3.) Kafka partitions in sync

This figure is showing the status of replicas in Kafka.

monitor1c

4.) Kafka partition leaders

This figure is showing the number of partition leaders per Kafka instance.  These values should be static unless adding or removing topics.

monitor1d.jpg

Have questions about Kafka or Elasticsearch?

Get in touch with our expert engineers who have assisted hundreds of companies with Apache Kafka, Elasticsearch, and supporting technologies.

MONITORING OPERATING SYSTEM STATUS

It is also important to monitor the health of the operating system.  Kafka relies heavily on the operating system for its page cache, hard drive space, and other system resources. As an example, excessive page cache misses points to the need for more brokers.

1.) Disk IO

This figure is showing disk performance. Disk performance is usually the limiting factor in Kafka, consistently high values here suggest you want to increase the IOPs (input/output performance) of the hard drives or add more Kafka brokers.  Skewed values here hint that your cluster should be re-balanced.

monitor2a

2.) Memory

We expect to see steady Memory usage and zero Swap usage. Swap usage is a great indicator that your Kafka cluster is performing poorly.

monitor2b

3.) CPU

Kafka doesn’t use much CPU unless your batch size is too small.

monitor2c

4.) Networking

This figure is showing the rate of network traffic. Skewed values here hint that your cluster should be re-balanced.

monitor2d

5.) Load

Skewed values here hint that your cluster should be re-balanced.
monitor2e

If you have a highly customized installation of Kafka or find you need more information to troubleshoot your cluster, we recommend enabling JMX on the Kafka brokers.  Logstash can pull the JMX data, send it to Elasticsearch, and provide you with any specific visualizations or alerts you may need.


Kafka as a Service With 99.99% Uptime Guarantee & 24/7 Monitoring

Dattell’s Kafka as a Service is a fully managed, high-throughput, distributed messaging system built on your cloud instances or on-prem servers, providing enhanced security and reduced latency for your data.