What is a Kafka Topic?

Kafka organizes message feeds into categories called topics. Each topic has a name that is unique across the entire Kafka cluster. Messages are sent to and read from specific topics.  In other words, producers write data to topics, and consumers read data from topics.  Kafka topics are multi-subscriber.  This means that a topic can have … Continue reading What is a Kafka Topic?

Hosted Kafka: Why Managed Kafka in Your Cloud or Data center is a Better Choice Than Hosted Kafka

One of the attractions of a hosted service, such as hosted Kafka, is the peace of mind that a third party is responsible for ensuring uptime.  In this post we challenge that misconception and suggest an improved approach to managed Kafka. Let’s start with a hard truth.  There are big names in managed Kafka as … Continue reading Hosted Kafka: Why Managed Kafka in Your Cloud or Data center is a Better Choice Than Hosted Kafka

Open Source Monitoring for Kafka

The key to ensuring Kafka uptime and maintaining peak performance is through monitoring.  By reviewing disk performance, memory usage, CPU, network traffic, and load in real-time abnormal metrics or trends can be identified before a performance dip or outage occurs. Furthermore, monitoring Kafka provides assurance to your users that all messages are correctly processed. There … Continue reading Open Source Monitoring for Kafka

Load Balancing With Kafka

Load balancing with Kafka is a straightforward process and is handled by the Kafka producers by default.  While it isn’t traditional load balancing, it does spread out the message load between partitions while preserving messaging ordering. By default, producers choose the partition assignment for each incoming message within a topic, following a round-robin approach. As … Continue reading Load Balancing With Kafka

Kafka Use Cases

Apache Kafka is a high-throughput, open source message queue used by Fortune 100 companies, government entities, and startups alike. Part of Kafka’s appeal is its wide array of use cases.  In this post we will outline several of Kafka’s uses cases from event sourcing to tracking web activities to metrics and more. Use Cases for … Continue reading Kafka Use Cases

Kafka Performance Tuning

For performance tuning you will want to measure latency and throughput for your Kafka implementation.  Latency is the measure of how long it takes Kafka to process a single event.  Throughput is the measure of how many events arrive within a particular period of time. To achieve the best balance of latency and throughput, tune … Continue reading Kafka Performance Tuning

Elasticsearch Shards — Definitions, Sizes, Optimizations, and More

Optimizing Elasticsearch for shard size is an important component for achieving maximum performance from your cluster. To get started let’s review a few definitions that are an important part of the Elasticsearch jargon. If you are already familiar with Elasticsearch, you can continue straight to the next section. Defining Elasticsearch Jargon:  Cluster, Replicas, Shards, and … Continue reading Elasticsearch Shards — Definitions, Sizes, Optimizations, and More

Elasticsearch Optimization for Small, Medium, and Large Clusters

The way nodes are organized in an Elasticsearch cluster changes depending on the size of the cluster.  For small, medium, and large Elasticsearch clusters there will be different approaches for optimization. Dattell’s team of engineers are expert at designing, optimizing, and maintaining Elasticsearch implementations and supporting technologies.  Find our more about our Elasticsearch services here. … Continue reading Elasticsearch Optimization for Small, Medium, and Large Clusters

The California Consumer Privacy Act of 2018 (CCPA): How to Prevent Data Breaches

Earlier this year, California passed the California Consumer Privacy Act of 2018, or CCPA for short. Beginning in January 2020, companies will be required to comply with this new law. It places new restrictions on how companies handle personal data, including minimum damages for class action suits in response to data breaches. Dattell is a … Continue reading The California Consumer Privacy Act of 2018 (CCPA): How to Prevent Data Breaches

Is my data valuable?

Yes, your data is valuable. However, like oil in the ground, its value isn’t fully realized until it is cleaned up and processed. And just as crude oil can be valuable for transportation, plastic manufacturing, and heating, company data too can be processed to extract multiple layers of value. In this post, we will discuss the ways in which your data can provide value for your business and customers.

Kafka Monitoring With Elasticsearch and Kibana

Monitoring Kafka cluster performance is crucial for diagnosing system issues and preventing future problems. We recommend using Elasticsearch for Kafka monitoring because Elasticsearch is free and highly versatile as a single source of truth throughout any organization.

Frequently Asked Questions: Apache Kafka

Our team is experienced with implementing and fixing Kafka on a wide-range of systems for an even wider-range of business needs. From our real-world experience with Kafka consulting, we found that there are common questions that many new clients have about the technology.
Here are some quick answers to those questions.

4 Approaches to Data Backup

We outlined the four primary ways for backing up data and their benefits and drawbacks to help you decide on which approach best meets your company’s needs.

Kafka Optimization

Issues with Apache Kafka performance are directly tied to system optimization and utilization. Here, we compiled the best practices for a high volume, clustered, general use case.

When to Consider Physical and Logical Separation With Kafka

When companies scale, their data handling needs change, and systems that worked a year ago are now over-taxed with the increase in message volume. One particular component of the data handling system, the cluster architecture, should be revisited.