KafScale is a Kafka-protocol compatible streaming platform built around a simple premise:
durable logs belong in object storage, not in stateful brokers.

Traditional Kafka couples durability, replication, and compute into long-lived broker processes.
This made sense in a disk-centric world. It becomes an operational and economic liability once object storage is available.

KafScale separates concerns cleanly. Brokers are stateless and ephemeral. Durability lives in S3.
The Kafka protocol remains intact.

KafScale is open source under the Apache 2.0 license and implemented in Go.
(KafScale is not a registered trademark.)

Most Kafka deployments are not event-driven transaction engines.
They are durable pipes: producers write, consumers read, teams rely on replay when something breaks.

Despite this, traditional Kafka clusters still require:

• Stateful brokers with local disks
• Replica rebalancing and leader movement
• Overprovisioning for peak load
• Continuous operational intervention

These costs are structural, not accidental.

Object storage changes the failure model, the scaling model, and the economics of durable logs.
Once durability is externalized, broker state stops being an asset and becomes a liability.

KafScale targets this common case by:

• Keeping brokers stateless
• Writing immutable segments directly to S3
• Preserving compatibility with existing Kafka clients and tooling

KafScale is used in production environments, but there are no warranties or guarantees.

Kafka brokers were designed in a world where durable storage meant local disks and failure meant replacing machines. In that model, brokers had to own both compute and data. Replication, leader election, and rebalancing were necessary to protect durability.

Cloud object storage changes this entirely.

Object storage already provides durability, availability, and cost efficiency at a level that broker-local disks cannot match. Once log segments live in object storage, brokers no longer need to be long-lived, stateful processes. They become protocol endpoints and transient compute.

Stateful brokers introduce complexity without adding resilience:

• Local disks require replication and rebalancing.
• Leader movement causes operational churn.
• Overprovisioning is needed to survive failure and peak load.
• Recovery time is tied to broker state, not data availability.

Stateless brokers backed by immutable object storage invert this model:

• Data durability is external and stable.
• Brokers can scale, restart, or disappear without data movement.
• Failure becomes a scheduling event, not an operational incident.

KafScale is built on the assumption that broker state is a historical constraint, not a requirement. The Kafka protocol remains valuable. The broker-centric storage model does not.

In scope:

• Kafka wire protocol for core produce, fetch, and consumer group APIs
• Immutable S3-backed segment storage
• Stateless broker pods
• etcd-backed metadata, offsets, and consumer group state
• Kubernetes-native operation via the operator

Explicit non-goals:

• Exactly-once semantics and transactions
• Compacted topics
• Kafka internal replication protocols
• Embedded stream processing inside the broker

Stream processing is expected to run in external compute engines such as Apache Flink (https://flink.apache.org) or Apache Wayang (https://wayang.apache.org).

• Brokers handle Kafka protocol traffic and buffer segments in memory.
• S3 stores immutable log segments and index files (source of truth).
• etcd stores metadata, offsets, and consumer group state.

For the technical specification and data formats, see kafscale-spec.md.

A detailed architecture overview and design rationale are available here: https://www.novatechflow.com/p/kafscale.html

• Quickstart guide: examples/101_kafscale-dev-guide/README.md
• Spring Boot app demo (E20): examples/E20_spring-boot-kafscale-demo/README.md
• Flink demo (E30): examples/E30_flink-kafscale-demo/README.md
• Spark demo (E40): examples/E40_spark-kafscale-demo/README.md

• License: Apache 2.0 (LICENSE)
• Contributing: CONTRIBUTING.md
• Code of Conduct: CODE_OF_CONDUCT.md
• Security: SECURITY.md