Green background with black text

How to fix the MEMBER_ID Error in Kafka

Updated May 2022

If you found this post it’s likely because you got the Kafka member_id error. Let’s first cover why the error popped up and then go through two ways to resolve the error.

Reason for the Kafka Member ID Error

When a new consumer joins a group it enters with the member.id set to UNKNOWN_MEMBER_ID, in other words an empty string.  

The broker is then tasked with assigning the consumer identity by returning an immutable universally unique identifier (UUID) to the consumer. 

Sometimes a consumer join request can fail because of a connection timeout, consumer restarting, or if the max.poll.interval.ms configured on the client is set to infinite.   This causes an accumulation of information inside of the group metadata cache.  Eventually the broker memory will overload.

To prevent this broker bursting issue, Kafka recognizes and fences invalid join group requests, hence the error.

The specific error message you saw was likely: 
join group failed with org.apache.kafka.common.errors.memberidrequiredexception.

Here is a link to the wiki discussing the reasoning for the error.

Resolving the member.id Error

To resolve the error, you will need to ensure that both a consumer group id and a consumer id are set for the consumer process.  

Consumer Group ID
The consumer group id must be the same for each process that wants to get a single subset of topic data.  

Consumer ID
The consumer id must be unique for each process to get a portion of the single subset of topic data.

There are many different consumer libraries, we’ll just cover two that have the most common config structure.

 In Sarama, you can set the consumer id with this setting:

				
					config.ClientID = consumer_name

 In Logstash, you can set the consumer id with this setting:

				
					client_id => consumer_name

Bonus Tip
Always use a unique consumer group id per topic/group.  Using the same consumer group id across multiple topics can cause issues during consumer rebalancing events.

For more information on Kafka consumer groups check out the rest of blog, starting with our Kafka Consumer Optimization article. 

Have Kafka Questions?

Managed Kafka on your environment with 24/ 7 support.

Consulting support to implement, troubleshoot,
and optimize Kafka.

Fully Managed Kafka
Kafka Consulting Support

Schedule a call with a Kafka solution architect.

Published by

Dattell - Kafka & Elasticsearch Support

Benefit from the experience of our Kafka, Pulsar, Elasticsearch, and OpenSearch expert services to help your team deploy and maintain high-performance platforms that scale. We support Kafka, Elasticsearch, and OpenSearch both on-prem and in the cloud, whether on stand alone clusters or running within Kubernetes. We’ve saved our clients $100M+ over the past six years. Without our guidance companies tend to overspend on hardware or purchase unnecessary licenses. We typically save clients multiples more money than our fees cost in addition to building, optimizing, and supporting fault-tolerant, highly available architectures.