Dattell

Updated September 2020 Apache Kafka is a distributed messaging system that implements pieces of the two traditional messaging models, Shared Message Queues and Publish-Subscribe.  Both Shared Message Queues and Publish-Subscribe models present limitations for handling high throughput use cases.   Apache Kafka provides fault tolerant, high throughput stream processing that can handle even the most complicated … Continue reading Kafka Uses Consumer Groups for Scaling Event Streaming

Updated August 2020 Apache Kafka’s high throughput and high availability make its applications vast.  In this post we dive into eight Kafka case studies.  These accounts are taken from work our Kafka solutions architects have done in the field with our clients. Medical Manufacturing Client automating the drug manufacturing process with multiple machines needs Kafka … Continue reading Kafka Case Studies

Updated August 2020 In this post we will compare Apache Kafka and the Confluent Kafka Platform, describing what they have in common and what sets them apart. What is Confluent Kafka and Apache Kafka? Apache Kafka is an open source message broker that provides high throughput, high availability, and low latency.  Apache Kafka can be … Continue reading Comparing Confluent Kafka and Apache Kafka

Updated September 2020 Taking a break from Kafka optimization posts to get back to the basics of Apache Kafka and define fundamental Kafka concepts. Kafka Definitions:  A Primer for Apache Kafka Fundamentals Kafka Producer.  A Kafka producer is a standalone application, or addition to your application, that sends data to Kafka broker(s). Kafka Broker.  A … Continue reading Kafka Definitions

Updated May 2020 Kafka Consumer’s Role. The role of the Kafka consumer is to read data from Kafka.  Kafka consumer optimization can help avoid errors and increase performance of your application.   While the focus of this blog post is on the consumer, we will also review several broker configurations which affect the performance of consumers. Top … Continue reading Kafka Consumer Optimization

Updated April 2020 Apache Kafka is hugely popular because of its features that guarantee uptime, make it easy to scale, enable Kafka to handle high volumes, and much more.  In this article we will discuss the Top 10 Apache Kafka features to help you evaluate if Kafka is the right technology for your company’s business … Continue reading Top 10 Apache Kafka Features That Drive Its Popularity

Updated August 2019 Kafka organizes message feeds into categories called topics. Each topic has a name that is unique across the entire Kafka cluster. Messages are sent to and read from specific topics.  In other words, producers write data to topics, and consumers read data from topics.  Kafka topics are multi-subscriber.  This means that a … Continue reading What is a Kafka Topic?

In this post we challenge the misconception that managed Kafka services need to be hosted on third party platforms.

Updated July 2020 Kafka’s primary role in many data architecture designs is ensuring that no data is lost.  Databases can fail. Servers can fail. Applications can fail. But a well designed Kafka deployment should provide 24/7, reliable, fault-tolerant message collection and processing. One way to ensure an expertly designed and managed Kafka deployment is to … Continue reading Uptime Guarantees for Managed Kafka as a Service

Updated September 2020 The key to ensuring Kafka uptime and maintaining peak performance is through monitoring.  By reviewing disk performance, memory usage, CPU, network traffic, and load in real-time abnormal metrics or trends can be identified before a performance dip or outage occurs.  Furthermore, monitoring Kafka provides assurance to your users that all messages are correctly … Continue reading Open Source Monitoring for Kafka

Updated June 2020 There are a handful of providers offering Kafka as a Service.  If you are in the market for managed Kafka you might be wondering what factors to consider when choosing a provider.  In this post, we break down the five most important considerations. #1 Is the service fully managed? If the service … Continue reading 5 Factors to Consider When Choosing a Kafka as a Service Provider

Updated September 2020 What is Kafka loading balancing? Load balancing with Kafka is a straightforward process and is handled by the Kafka producers by default.  While it isn’t traditional load balancing, it does spread out the message load between partitions while preserving message ordering. Round-robin approach:  By default, producers choose the partition assignment for each … Continue reading Load Balancing With Kafka

Updated August 2020 One aspect of Kafka that can cause some confusion for new users is the consumer offset.  In this post, we define consumer offset and outline the factors that determine the offset. Defining Kafka Consumer Offset The consumer offset is a way of tracking the sequential order in which messages are received by … Continue reading Understanding Kafka Consumer Offset

Updated March 2019 ZooKeeper is used in distributed systems for service synchronization and as a naming registry.  When working with Apache Kafka, ZooKeeper is primarily used to track the status of nodes in the Kafka cluster and maintain a list of Kafka topics and messages. ZooKeeper was originally developed by Yahoo to address the bugs … Continue reading What is ZooKeeper & How Does it Support Kafka?

Updated September 2020 Many companies leverage both Apache Kafka and the Elastic Stack (Elasticsearch, Logstash, and Kibana) for log and/or event processing.  Kafka is often used as the transport layer, storing and processing data, typically large amounts of data. Kafka stages data before it makes its way to the Elastic Stack.  Logstash transforms the data, … Continue reading Origins of Kafka and Why it Plays Well With Elasticsearch

Updated August 2020 Apache Kafka is a high-throughput, open source message queue used by Fortune 100 companies, government entities, and startups alike. Part of Kafka’s appeal is its wide array of use cases.  In this post we will outline several of Kafka’s uses cases from event sourcing to tracking web activities to metrics and more. … Continue reading Kafka Use Cases

For Apache Kafka performance tuning measure latency and throughput for your Kafka implementation. Latency is the measure of how long it takes Kafka to process a single event. Throughput is the measure of how many events arrive within a particular period of time.

Updated August 2020 There are six key components to securing Kafka. These best practices will help you optimize Kafka and protect your data from avoidable exposure. #1 Encryption By default, data is plaintext in Kafka, which leaves it vulnerable to a man-in-the-middle attack as data is routed over your network. Transport layer security (TLS) and/or … Continue reading Kafka Optimization: Kafka Security Checklist

Updated September 2020 Apache Kafka is a distributed system, running in a cluster with each of the nodes referred to as brokers. Kafka topics are partitioned and replicated across the brokers throughout the entirety of the implementation. These partitions allow users to parallelize topics, meaning data for any topic can be divided over multiple brokers. … Continue reading Kafka Optimization — How many partitions are needed?

In this post, we will define what Kafka topics are and explain how to create them.

There are several message queue programs to choose from: Kafka, RabbitMQ, ActiveMQ, ZeroMQ, Redis, among others. How do you choose which is right for you?

Our team is experienced with implementing and fixing Kafka on a wide-range of systems for an even wider-range of business needs. From our real-world experience with Kafka consulting, we found that there are common questions that many new clients have about the technology.
Here are some quick answers to those questions.

Dattell is partnering with the creators of Kafka and Confluent to strengthen our commitment to improve messaging and data infrastructure for companies of all sizes.

Issues with Apache Kafka performance are directly tied to system optimization and utilization. Here, we compiled the best practices for a high volume, clustered, general use case.

When companies scale, their data handling needs change, and systems that worked a year ago are now over-taxed with the increase in message volume. One particular component of the data handling system, the cluster architecture, should be revisited.