Designing Kafka topics for scalability

Kafka topic design isn’t just about naming—it’s about balancing throughput, retention, and manageability. Here are the patterns we use when designing Kafka topics for high-scale systems.

Partition count strategy

Guideline for partitions: Use 1–2 partitions per consumer for parallelism.

Why partition count is important: More partitions = more throughput, but also more open files and replication work.

Bonus tip: Monitor controller and ISR pressure before scaling beyond 2000 partitions per broker.

Naming conventions

Pattern: <team>.<function>.<environment> (e.g., payments.transactions.prod)

Why naming matters: Helps with ACLs, monitoring, and lifecycle automation.

Use compaction and TTL strategically

Compacted topics: Great for config/state replication.

Delete policy: Use with TTL for logs or metrics.

Why compaction & TTL matters: Reduces storage use and enables recovery from scratch.

Retention planning

Default retention: 7 days is typical, but align with your replay, compliance, or recovery requirements.

Bonus tip: Set tiered storage if using long retention with cloud-native tools.

Isolation for use cases

Isolation for use cases: Separate topics for internal pipelines, external APIs, and operational vs. business events

Why isolation matters: Enables fine-grained monitoring, ACLs, and scaling strategies.

Summing it up

Topic design decisions affect everything from performance to observability. With a good foundation, your Kafka deployment can grow without chaos.

Want help reviewing your Kafka architecture? Let’s talk.

24x7 Kafka Support & Consulting

Visit our Apache Kafka® page for more details on our support services.

Designing Kafka topics for scalability