Updated May 2022
If you found this post it’s likely because you got the Kafka member_id error. Let’s first cover why the error popped up and then go through two ways to resolve the error.
Reason for the Kafka Member ID Error
When a new consumer joins a group it enters with the member.id set to UNKNOWN_MEMBER_ID, in other words an empty string.
The broker is then tasked with assigning the consumer identity by returning an immutable universally unique identifier (UUID) to the consumer.
Sometimes a consumer join request can fail because of a connection timeout, consumer restarting, or if the max.poll.interval.ms configured on the client is set to infinite. This causes an accumulation of information inside of the group metadata cache. Eventually the broker memory will overload.
To prevent this broker bursting issue, Kafka recognizes and fences invalid join group requests, hence the error.
The specific error message you saw was likely:
join group failed with org.apache.kafka.common.errors.memberidrequiredexception.
Here is a link to the wiki discussing the reasoning for the error.
Resolving the member.id Error
To resolve the error, you will need to ensure that both a consumer group id and a consumer id are set for the consumer process.
Consumer Group ID
The consumer group id must be the same for each process that wants to get a single subset of topic data.
Consumer ID
The consumer id must be unique for each process to get a portion of the single subset of topic data.
There are many different consumer libraries, we’ll just cover two that have the most common config structure.
In Sarama, you can set the consumer id with this setting:
config.ClientID = consumer_name
In Logstash, you can set the consumer id with this setting:
client_id => consumer_name
Bonus Tip
Always use a unique consumer group id per topic/group. Using the same consumer group id across multiple topics can cause issues during consumer rebalancing events.
For more information on Kafka consumer groups check out the rest of blog, starting with our Kafka Consumer Optimization article.
Have Kafka Questions?
Managed Kafka on your environment with 24/ 7 support.
Consulting support to implement, troubleshoot,
and optimize Kafka.