Author(s): Neel Shah

Originally published on Towards AI.

1. Kafka’s Core Value Proposition: A Unified Event Streaming Platform

Apache Kafka is frequently compared to message brokers like RabbitMQ or ActiveMQ, but this comparison is incomplete. Kafka is not just a messaging system; it is a distributed event streaming platform that unifies three key capabilities into a single, cohesive system:

1. Publish and Subscribe to Streams of Events: Like a traditional message broker, Kafka provides a mechanism for client applications (Producers) to publish events and for other applications (Consumers) to subscribe to them. This decouples data producers from data consumers, allowing them to evolve independently.
2. Store Streams of Events Durably and Reliably: Unlike traditional brokers that discard messages after they are acknowledged, Kafka stores streams of events in a fault-tolerant, durable way for a configurable duration. This storage layer is not a temporary buffer but a first-class, replayable log. This allows multiple, independent applications to consume the same event stream at different times and at their own pace.
3. Process Streams of Events in Real-Time: Kafka provides a powerful client library, Kafka Streams, and integrates with other stream processing frameworks (like Flink or Spark), allowing applications to process and analyze event streams as they arrive.
4. This unique combination of messaging, storage, and stream processing makes Kafka a powerful and versatile platform. It is designed from the ground up for high throughput (capable of handling trillions of records per day), low latency (as low as 2ms), massive horizontal scalability, and high availability through data replication. This has made it the de facto standard for real-time data pipelines, used by over 80% of Fortune 500 companies to power everything from financial fraud detection to real-time e-commerce recommendations.

2. The Core Architecture of a Kafka Cluster

The power and performance of Apache Kafka stem from a set of simple yet powerful architectural concepts. Understanding these fundamental building blocks is essential to grasping how Kafka achieves its remarkable scalability and fault tolerance.

The Log is the Heart: The Immutable, Partitioned Commit Log

The central data structure and core abstraction in Kafka is the distributed, partitioned, replicated commit log. At its heart, a Kafka topic is simply a log file. When a producer sends a message, it is appended to the end of this log. Once written, the data is immutable — it cannot be changed or deleted, only appended to.

This log-based architecture is the source of many of Kafka’s key properties:

• Durability: Data is persisted to disk, making it durable across broker restarts and failures.
• Ordering: Within a single partition of the log, messages are strictly ordered in the sequence they were appended.
• Replayability: Because messages are not deleted upon consumption, multiple consumers can read the log from any point, allowing for data to be re-processed for new use cases or to recover from application failures.
• This design choice has profound performance implications. Kafka transforms all writes into sequential appends to the end of a log file. Sequential disk I/O is orders of magnitude faster than random I/O, allowing Kafka to achieve extremely high write throughput even on commodity hardware.

Events (Records): The Atomic Unit of Data

The data that flows through Kafka is organized into discrete units called events, also referred to as records or messages. Each event is a structured piece of data with four primary components:

• Key: An optional piece of metadata, typically used to identify the business entity (e.g., a user_id, order_id). All events with the same key are guaranteed to be written to the same partition, which ensures that they will be processed in order by a consumer.
• Value: The main payload of the event, which can be a JSON object, an Avro record, or a simple string.
• Timestamp: Used for time-based operations, such as windowed aggregations in stream processing and time-based log retention policies.
• Headers: Optional user-defined key-value pairs for carrying metadata, such as tracing information.

Topics and Partitions: Scaling Data Streams

Events in Kafka are organized into topics. A topic is a logical channel or category for a stream of related events (e.g., orders, payments, user_clicks). Producers write events to specific topics, and consumers subscribe to them.

While a topic is a logical concept, its physical implementation is a set of one or more partitions. A partition is an individual, ordered, immutable log file. Each topic is split into these partitions, and the partitions themselves are distributed across the different brokers in the Kafka cluster.

Partitioning is the fundamental mechanism for horizontal scalability and parallelism. By splitting a topic’s log across multiple brokers, the read and write load can be distributed across the entire cluster. The number of partitions for a topic defines the maximum level of parallelism for consumption.

Offsets, Brokers, and Replication

• Offsets: Each event within a partition is assigned a unique, sequential integer ID called an offset. This offset uniquely identifies a record. Consumers track their progress by storing the offset of the last message they have successfully processed.
• Brokers and Clusters: A single Kafka server is known as a broker. A cluster is a collection of one or more brokers working together. Each broker stores the data for a set of partitions.
• Replication and Fault Tolerance: To prevent data loss, Kafka replicates each partition across multiple brokers based on the topic’s replication-factor (often 3 in production). For each partition, one replica is designated as the Leader, and the others as Followers. All producer writes and consumer reads go to the leader. Followers passively copy the data. If the leader fails, one of the In-Sync Replicas (ISRs) is automatically promoted to the new leader.

A write is only considered “committed” and acknowledged to the producer (when acks=all) after it has been successfully written to the leader and all replicas in the ISR set. This mechanism is the foundation of Kafka's strong durability guarantees.

3. The Actors: Producers, Consumers, and Delivery Guarantees

The Kafka cluster itself is the storage and transport layer. The applications that interact with it are known as producers and consumers.

Producers: Writing with Durability Trade-offs

A producer is any client application that publishes events to a Kafka topic. A key decision for producers is setting the acks (acknowledgments) configuration parameter, which represents a fundamental trade-off between performance and durability.

Consumers and Consumer Groups

A consumer is a client application that subscribes to one or more topics. To achieve scalable and fault-tolerant consumption, Kafka introduces the concept of the consumer group. A consumer group is a set of consumer instances that jointly consume the data from a topic. Kafka assigns each partition to exactly one consumer instance within a group, allowing for parallel processing.

A critical health metric is Consumer Lag, defined as the difference between the latest offset produced to a partition and the offset last committed by a consumer group. High lag indicates the consumer cannot keep up with the producer rate.

Achieving Exactly-Once Semantics (EOS)

While At-Least-Once is the most common guarantee (messages are not lost but may be duplicated), Kafka provides features to achieve the highly desirable Exactly-Once (EOS) semantic:

1. The Idempotent Producer: By setting enable.idempotence=true, the producer includes a unique Producer ID (PID) and sequence number with its messages. The broker tracks the highest sequence number written, discarding any message received with a lower sequence number as a duplicate retry. This prevents duplicates caused by producer retries.
2. Kafka Transactions: For applications that follow a “read-process-write” pattern (consume from Topic A, produce to Topic B), transactions allow grouping the consumption offset commit and the output produces into a single, atomic operation. Downstream consumers configured with isolation.level=read_committed will only read messages that are part of successfully committed transactions.

Log Compaction: Retaining the Latest State

Kafka’s default retention policy is time- or size-based. For state-modeling use cases (e.g., current user profile, latest sensor reading), log compaction is used. When a topic is configured for compaction, Kafka guarantees to retain at least the last known value for each message key within each partition. A background process removes older records that have the same key as a more recent record, effectively maintaining a complete snapshot of the latest state.

4. How Kafka Powers AI and Machine Learning Applications

Kafka’s characteristics — real-time event streaming, high throughput, and durable storage — make it indispensable for modern AI and Machine Learning (ML) systems, particularly those requiring real-time inference and continuous learning.

Real-Time Feature Engineering

In production ML, models rely on features (input variables). Traditionally, features are computed in batch jobs, leading to stale predictions. Kafka transforms this:

• Streaming Feature Pipelines: Raw events (user clicks, transaction data, sensor readings) are fed into Kafka topics. Kafka Streams or other stream processors consume these events to calculate real-time features (e.g., “count of clicks in the last 5 minutes,” “average transaction value in the last hour”).
• Feature Store Backing: These computed features are written back to a Kafka topic configured for log compaction, which acts as a real-time, highly available Feature Store changelog. Inference services can quickly look up the latest feature value for any given key (user_id).

Online Model Inference and Serving

Kafka provides the crucial communication layer for low-latency model serving:

1. Inference Request: A user action (e.g., loading a product page) generates an event that is sent to a dedicated request topic.
2. Model Service Consumption: A microservice hosting the trained ML model consumes this request event, retrieves the necessary real-time features from the compacted Feature Store topic, and runs the inference.
3. Real-Time Prediction: The model service produces the prediction result (e.g., a recommended product ID, a fraud score) to an output topic, which the front-end application immediately consumes to personalize the user experience.

Architecting Complex Agentic AI Scrutiny Pipelines with Conditional Routing

The flexibility and durable queuing capabilities of Kafka are perfectly suited for building complex Agentic AI systems, especially those that require a multi-stage, sequential workflow where the number of stages is dynamic and based on the document or application type.

In this architecture, each scrutiny level (Level A, B, C, D) is represented by a specialized AI microservice. Kafka topics act as the durable, highly scalable queues that pass the work from one level to the next, with the routing decision made by the processing agent itself.

Dynamic Routing Example (Visa Scrutiny)

The initial document (the event) contains a visa_type field. After an agent processes the document, it uses this field to determine the output topic, allowing for a dynamic, non-linear flow.

Pipeline Flows:

• Tourist Visa (1 Level): visa-applications-in (Level A) → final-decisions-out
• Business Visa (2 Levels): visa-applications-in (Level A) → scrutiny-queue-B (Level B) → scrutiny-queue-C (Level C) → scrutiny-queue-D (Level D) → final-decisions-out
• Student Visa (3 Levels): visa-applications-in (Level A) → scrutiny-queue-B (Level B) → scrutiny-queue-D (Level D) → final-decisions-out

The Workflow Advantage with Kafka:

1. Decoupling and Flexibility: Each agent operates independently. The decision of which queue to write to is local to the agent, based on the event data, not rigid routing configuration. This allows you to easily add a new visa type or a new scrutiny level (e.g., Level E for Medical checks) without downtime or modifying other agents.
2. Scalability: Each scrutiny-queue topic can be partitioned across the Kafka cluster. Agents can be scaled independently; if the Level B Financial Agent is the bottleneck, you only scale that specific consumer group.
3. Durability and Auditability: Because Kafka durably stores every message, the entire processing history for every document is permanently recorded. If the final decision is questioned, you can easily replay the events through the pipeline to understand exactly when and why an agent flagged a document at a specific level.
4. State Persistence (Memory): The history of conversation or the long-term memory of an agent is stored in a Kafka topic configured for log compaction. This ensures that agents can retrieve their latest context or the state of an ongoing task reliably, even if they restart.

This event-driven architecture ensures the complex scrutiny process is robust, transparent, and can scale dynamically to handle high volumes of documents with varying workflow complexity.

5. Key Architectural Questions Answered

To round out the fundamentals, here are answers to some of the most common questions regarding Kafka’s core architecture and operation:

Q: Explain the role of an ISR and why it’s important for durability.

A: ISR stands for In-Sync Replicas. For any given partition, it is the set of follower replicas that are fully caught up with the leader’s log. Its role is critical for durability. When a producer uses acks=all, the leader broker will only send an acknowledgment back to the producer after the message has been successfully replicated to all replicas in the ISR set. This ensures that if the leader fails, the message is guaranteed to exist on at least one of the up-to-date followers, which can then be promoted to the new leader without data loss.

Q: What is consumer rebalancing and what triggers it?

A: Consumer rebalancing is the process by which partitions of a topic are reassigned among the consumer instances in a consumer group. It is a core part of Kafka’s mechanism for fault tolerance and elasticity. A rebalance is triggered whenever the membership of a consumer group changes, which can happen when: 1) a new consumer instance joins the group, 2) an existing consumer instance shuts down cleanly, or 3) a consumer instance is considered dead by the group coordinator after failing to send heartbeats for a configured period (session.timeout.ms). During a rebalance, all consumers in the group temporarily stop processing messages while the partitions are redistributed.

Q: How does Kafka achieve high throughput?

A: Kafka’s high throughput is the result of several key design decisions working together:

• Sequential I/O: It uses an append-only log structure for partitions, which turns random writes into highly efficient sequential disk writes.
• Partitioning: Topics are split into partitions, allowing read and write loads to be parallelized across multiple brokers and disks in the cluster.
• Zero-Copy: Kafka uses the sendfile system call to efficiently transfer data from the disk page cache directly to the network socket without copying it into the application's memory space, minimizing CPU and memory overhead.
• Batching: Producers and consumers batch messages together to reduce network round-trips, improving overall efficiency.
• Dumb Broker/Smart Client: The broker’s logic is kept simple, offloading state management (like offsets) to the clients, allowing the broker to focus on fast I/O operations.

Apache Kafka is much more than a message queue — it is the central nervous system for any application built on the concept of real-time event processing. By understanding its foundational concepts — the immutable, partitioned log, the power of consumer groups, and the strength of its durability guarantees — you can leverage it to build scalable, fault-tolerant, and intelligent systems, including the next generation of real-time AI applications.

Let me know if you’d like to dive deeper into any of these specific areas, such as the architecture of the Schema Registry for data governance, or explore a detailed code example of a Kafka Streams application for real-time feature engineering!

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI