Updated August 2020
Kafka organizes message feeds into categories called topics. Each topic has a name that is unique across the entire Kafka cluster.
Messages are sent to and read from specific topics. In other words, producers write data to topics, and consumers read data from topics.
Kafka topics are multi-subscriber. This means that a topic can have zero, one, or multiple consumers subscribing to that topic and the data written to it.
In Kafka, topics are partitioned and replicated across brokers throughout the implementation. Brokers refer to each of the nodes in a Kafka cluster. The partitions are important because they enable parallelization of topics, enabling high message throughput. See the figure below taken from the Apache Kafka website for a visual of how topics are split into partitions.
Offsets are assigned to each message in a partition to keep track of the messages in the different partitions of a topic. For more information on Kafka Consumer Offsets check out this article.
Create Kafka Topics
Kafka topics can be created either automatically or manually. It is best practice to manually create all input/output topics before starting an application, rather than using auto topic. However, topics do not need to be manually created.
Creating topics automatically is the default setting. You can confirm if this is the case for your implementation by checking that the property auto.create.topics.enable is set to true. With this setting, topics are automatically created when applications produce, consume, or fetch metadata from a not yet existent topic.
For auto topic creation, it’s good practice to check num.partitions for the default number of partitions and default.replication.factor for the default number of replicas of the created topic.
To create topics manually, run kafka-topics.sh and insert topic name, replication factor, and any other relevant attributes. For example:
> bin/kafka-topics.sh –create
–bootstrap-server localhost:9092
–replication-factor 10
–partitions 3
–topic test
View a List of Kafka Topics
To view a list of Kafka topics, run the following command:
> bin/kafka-topics.sh –list
–bootstrap-server localhost:9092
For a more granular view of the topics and partitions:
> bin/kafka-topics.sh –describe
–bootstrap-server localhost:9092
Have Kafka Questions?
We offer fully managed Kafka with top performance and exceptional support.
We offer a number of support services including RSA, workshops, and training.