#3

Exceedingly high overall bandwidth for a single service also merits physical separation.

If a specific subset of the messages in a cluster are the source of the majority of the bandwidth, decrease the overall liability to your message queue by creating a separate cluster dedicated to the high bandwidth messages.

Autoscaling operations become easier as well when working with only a single service.

#4

The fourth case for a separate cluster is when you have critical messages that must be guaranteed.

As mentioned previously, autoscaling operations are more robust when only accounting for a single service.

Additionally, physical separation of your critical data protects the messages from an unrelated, less important service with a bug causing a degradation of your Kafka cluster holding the critical data.

Drawbacks to Physical Separation

The drawback to implementing physically separated clusters is that there are now multiple clusters to monitor and alert on, and multiple clusters to expand and manage.

These perceived drawbacks are minimized by automating the majority of those tasks.

LOGICAL SEPARATION

Logical separation of data is a bit trickier than physical separation because much thought and consideration must be given to the creation of  Kafka topics and which services apply to each topic.

Unless multiple services need the same stream of messages, all of the data a service needs should have its own topic.

You want to keep your topics to a minimum to maintain high performance of the Kafka cluster.

Benefit of Logical Separation

The benefit of logical separation is that missing data is easily retrieved from Kafka because of the organization of information into topics.

Drawback

The drawback to this approach is that for optimization purposes, each topic should have its own dedicated disk to maintain sequential read and write performance.

—

Implementing both forms of physical and logical separation of data will increase the performance of your Kafka clusters, reduce cost, and reduce downtime.