How to Simulate Kafka Load for Benchmarking

Testing Kafka under real-world conditions is essential for capacity planning and tuning. In this post, we’ll walk through tools and techniques for simulating production-like workloads against your Kafka cluster.

Use kafka-producer-perf-test.sh

Kafka includes a built-in performance testing tool. It’s great for quick testing of broker throughput and producer configs. Use with acks=all and compression.type=snappy to simulate real use cases.  Check out our post on Kafka acknowledgement settings for more information.

bin/kafka-producer-perf-test.sh \

  –topic test-topic \

  –num-records 1000000 \

  –record-size 100 \

  –throughput 10000 \

  –producer-props bootstrap.servers=localhost:9092

Write custom load generators

Python and Go are popular for building custom test clients.  They are flexible and scriptable, and great for mimicking real production message formats. Use confluent-kafka-python or aiokafka, and add timestamps in headers to track latency.

Test at scale with multiple producers

Use Docker Compose or Kubernetes jobs to spin up concurrent producer containers.  This realistic multi-client testing uncovers bottlenecks.  Set a fixed messages-per-second rate, and capture success/failure metrics and delivery latency.

Simulate consumer load

Use kafka-consumer-perf-test.sh or custom scripts to measure lag and processing throughput, and validate that consumers can keep up under sustained load.  This approach is good for identifying GC pauses and poll loop delays.

Monitor broker metrics during tests

To identify bottlenecks in disk, CPU, memory, and network layers track:

• BytesInPerSec
• UnderReplicatedPartitions
• RequestQueueTimeMs
• JVM GC and thread pool stats
  
  

Summing it up

Don’t wait for a production outage to learn how your Kafka cluster behaves under pressure. Use open source tools to simulate load and benchmark your limits—before your users do.

Need help designing a Kafka benchmark? Get in touch—we’ve helped clients simulate billions of messages per day.