How to Choose a Managed Kafka Provider

Preventative maintenance

Preventative maintenance guided by real-time monitoring should be included with your managed Apache Kafka service.  Expect the provider to monitor for topic offset deltas, consumer group lag, disk cache hit percent, and several other important metrics.  Check out our Kafka monitoring post if you’re interested in learning more about monitoring.  

Response times

For priority 1 issues, 30-minutes should be the guaranteed response time.  Response times for non-emergency issues are also important.  You don’t want to wait for input for hours or days when you are working on a project or your Kafka cluster needs attention.  

Look for down-priority response times as well when choosing a provider. For instance, at Dattell our longest possible wait times for non-production issues are 3-hours.  That’s the longest our clients ever wait for a response from their dedicated engineer, no matter how trivial.

And for production issues our response time is 15 minutes.

Dedicated engineer

Will the managed Kafka provider be connecting you with a dedicated engineer that you will have direct access to work with daily?   Or is the provider connecting you with whichever Kafka engineer happens to be available at the time you need help?  

Consider which approach would provide more personalized service and better outcomes.  

Uptime

Kafka should be the most reliable component of your data architecture.  That’s what makes it so valuable.  The goal of any managed service provider is to provide 100% uptime.  

At Dattell we monitor our clients’ clusters 24×7 to identify emerging issues.  This way we troubleshoot and resolve the issues before they ever lead to data loss or an outage. 

Cluster security

Choose a managed Kafka provider that will address all of your security goals.  Encryption, authorization, and authentication should all be standard.

Keep in mind that security is inherently limited when choosing a hosted Kafka provider.  Running Kafka in your environment with provide you with full authority over access control and other security measures.

Hosted or on your environment

Running Kafka in your environment offers many benefits over third party hosted Kafka.  Several providers, including Dattell, offer fully managed Kafka on your environment, whether on-prem or in the cloud.   

There is less risk exposure when Kafka is running in your environment.  When choosing hosted Kafka, your company’s data is being handed over to a third party.  This third party now makes security decisions, and you lose oversight and overall control. 

You also lose ownership of your Kafka implementation when you use a hosted provider.  If the host makes changes to their service, pricing, versions supported, then you might be able to walk away from the provider, but you’re walking away empty handed.  You need to start from scratch somewhere else. 

Managed Kafka in your environment, in contrast, is owned by you.  If you decide to part ways with your service provider, then you still have a Kafka cluster and can either manage in-house or choose another managed Kafka provider.

Latency considerations

If you run Kafka in your environment, alongside your infrastructure, then you can expect tens of milliseconds of latency.  If you choose a third party hosted option, then latency will increase to hundreds-to-thousands of milliseconds. Read about decreasing network latency for Kafka.

Cost structure

Hosted services are budget friendly for small use cases; the surcharges added onto instance costs are nominal.  For simple project, it would be cheaper to use a host than using a service like ours. 

However, as data volume increases, those surcharges and other fees add up.  For instance, our management fees are half the cost of Confluent when moving 500 Mb/sec.  And with our service you get much more than Confluent.  You get personalized help, 15-minute response times, and training. And we attend regular meetings, expectation-set with senior management, and provide troubleshooting support for the applications that work with Kafka.

Finally, look for a fixed rate approach.  You want to avoid extra, surprise fees.  It makes it hard to budget, and can become much more expensive unexpectedly.