Updated September 2022
One of the attractions of managed services, such as Kafka or Elasticsearch as a Service, is the peace of mind that a competent third party is responsible for ensuring uptime.
In this article we challenge the misconception that managed services need to be hosted on a third party platform to ensure stability and high performance.
Does hosted Kafka provide any benefit over other managed Kafka paradigms?
A commonly held misconception is that hosting on a third party’s environment will give that third party greater authority over ensuring uptime. This simply isn’t true.
The idea that hosted Kafka leads to better uptime falls apart for two reasons.
Firstly, maximizing Kafka uptime has more to do with good communication between the service provider and the customer than any other factor.
For instance, staying informed about current and projected business and technical needs allows a good service provider to anticipate changes that need to be made to the Kafka deployment. Additionally, having comprehensive monitoring in place to review Kafka and operating system performance 24/7 provides critical insight into increased demand and maintenance needs. Neither of these factors are dependent on Kafka being hosted by a third party.
Secondly, a service provider can comprehensively manage Kafka as a service on your environment. This requires limited access to your architecture, only requiring access to Kafka and its supporting technologies such as Zookeeper.
Even in the most sensitive use cases, including high-security government contractors and financial institutions, our customers can provide us with the necessary access to the portion of their cloud or on-prem environment that houses Kafka and its supporting technologies.
What are you giving up when a third party hosts your managed Kafka?
When you hand over Kafka hosting to a third party, you are inherently compromising on three critical factors: speed, data authority / data security, and pricing.
Speed: Building Kafka alongside your infrastructure, that is to say inside your cloud or on-prem environment, guarantees to reduce latency in your system. By allowing a managed Kafka provider to host Kafka in their cloud, you are introducing unnecessary latency.
Data Authority & Security: With Kafka hosted on a third party’s cloud, your company’s and customers’ data, is exposed to additional risk for breach.
At Dattell, because we build managed Kafka services on our customers’ environments, we not only decrease latency, but also increase security by including all the security features you need plus keeping your Kafka cluster off the internet.
Ownership: Your business owns its infrastructure so there no possibility of paying ransom to keep your infrastructure online.
Further, by hosting on your environment, you have an easier time managing your Kafka service in-house as your team grows or changes, or moving to a new managed Kafka provider should your current provider not live up to your expectations.
Pricing: By hosting your Kafka service on their environment, hosted Kafka can end up costing you more in cloud fees because the managed Kafka as a Service provider is incentivized to increase cloud costs and over-build your cluster to charge you more.
Are you okay with your hosted Kafka provider sharing your Kafka cluster?
Some hosted Kafka providers combine multiple companies’ Kafka services into a single cluster. This exposes all of those companies to greater risk of a system crash and decreases the security of your company and customers’ data.
Always ask if this is a practice used by a potential managed Kafka provider.
When is hosted Kafka the right option?
Good question. We have yet to come up with a reason why a hosted solution provides any benefit to a business, other than for the company doing the hosting. Even for small companies without a large or experienced IT team, it still makes sense to keep Kafka hosted on your environment while utilizing a Managed Kafka provider, like Dattell. By hosting Kafka internally you are reducing latency, increasing security, and in most cases decreasing the cost of managed Kafka.
Have Kafka Questions?
Managed Kafka on your environment with 24/ 7 support.
Consulting support to implement, troubleshoot,
and optimize Kafka.