Kafka Offsets

You might be wondering at this point how consumers can keep track of the messages in the different partitions of a particular topic. This is handled with what are called offsets. Offsets are assigned to each message in a partition. Kafka automatically handles the message ordering, and the sequence for these offsets does not change.

Just like a mailing address includes a country, city, and street number to identify a location, messages within Kafka can be identified using the combination of the topic, partition, and offset. Together, these three pieces of information are unique to a message.

Creating a Kafka Topic

There are two ways to create a Kafka topic.

Auto Topics. Set the property auto.create.topics.enable to true. This parameter is set to true by default. Topics will be automatically created when applications produce, consume, or fetch metadata from a not yet existent topic. It is good practice to check num.partitions for the default number of partitions and default.replication.factor for the default number of replicas of the created topic.

Manual Topics. Topics can be created manually with the Kafka utility. Run kafka-topics.sh and insert topic name, replication factor, and any other relevant attributes.

> bin/kafka-topics.sh –create
–bootstrap-server localhost:9092
–replication-factor 10
–partitions 3
–topic test

It is best practice to manually create all input/output topics before starting an application, rather than using auto topic. However, internal topics do not need to be manually created.

View Topics. To view the list of topics, run the following command:

> bin/kafka-topics.sh –list
–bootstrap-server localhost:9092

Granular View of Topics. For a more granular view of the topics and partitions:

> bin/kafka-topics.sh –describe
–bootstrap-server localhost:9092