"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

1. WarpStream
Beyond Tiered Storage: Serverless Kafka with No Local Disks

2. ● Cloud disks are expensive.
● Long retention workloads TCO can
be 80% disk cost even at low
throughput
● EBS vs instance storage doesn’t
matter
○ double/triple replication
expensive compared to S3
2
EBS (GP2) $0.1/GiB pre-replication
EBS (GP2) $0.3/GiB post-replication
S3 $0.02/GiB post-replication
Why Tiered Storage?
Retention

3. Tiered storage helps, but not enough

4. ● Stateful brokers with attached
storage make operations complex,
difficult, and inelastic
● Requires consensus, topic-partition
leaders, custom operations for
scaling in/out and doing node
replacements
● Balancing in general always a
problem
4
Why Drop the Disks?
Operations

5. ● 80%+ of TCO for high throughput
Apache Kafka clusters can be
networking fees
● $0.053 / compressed GiB
transferred in 100% ideal
conditions with fetch-from-follower
enabled
5
Why Drop the Disks?
Networking

6. Zero disks would be better

7. WarpStream’s Cloud-Native design
7
greatly reduces the cost and
complexity of Apache Kafka

8. How It Works

9. ● Entire storage engine redesigned
around minimizing PUT / GET
operations
9
● S3 PUT: $0.000005
● S3 GET: $0.0000004
Optimize for Cloud
Unit Economics

10. ● Entire storage engine redesigned
around minimizing PUT / GET
operations
● Networking is free, and storage is
cheap.
10
Optimize for Cloud
Unit Economics
● S3 PUT: $0.000005
● S3 GET: $0.0000004
● S3 Storage: $0.023/GB-mo
● S3 Cross-AZ Networking: Free

11. Step 1: Eliminate per-TopicPartition Files
11
��
��
��

12. Step 1: Eliminate per-TopicPartition Files
12

13. Step 1: Eliminate per-TopicPartition Files
13

14. Step 2: Separate Data from Metadata
14
No per-TopicPartition leaders!

15. 15
Step 3: Introduce
Data Locality for
Live Reads

16. Step 4: Introduce
Data Locality for
Historical Reads
16

17. 17
Step 4: Introduce
Data Locality for
Historical Reads

18. 18
Hard Mode: Compacted Topics

19. 19
Hard Mode: Compacted Topics
● Tiered Storage in open source Apache Kafka does not support compacted
topics
● WarpStream already does compaction internally
● … how hard could it possibly be?

20. 20
Hard Mode: Compacted Topics

21. 21
Hard Mode: Compacted Topics

22. 22
Hard Mode: Compacted Topics

23. 23
Hard Mode: Compacted Topics

24. 24
Hard Mode: Compacted Topics

25. 25
Hard Mode: Compacted Topics

26. WarpStream costs ~85% less than self-hosted Kafka for high volume workloads
26
Deployment
Model
Workload Profile Hardware Network Object Storage Total Costs
WarpStream Avg. ingress: 1 GiB/sec
Avg. egress: 3 GiB/sec
Retention: 1 day
Replication Factor = 3
3 Availability Zones
$223k/year $<2k/year $61k/year $286k/year
Self Hosted
Apache Kafka
$223k/year $1.68M/year $0 $1.9M/year

27. WarpStream Is Still Real Time
27
● P99 producer latency of ~400ms
● Producer to Consumer – End to
End Latency <1.5s

28. WarpStream Supports S3 Express One Zone
28
● P99 producer latency as low as
150ms
● Uses a majority quorum of 3
buckets to provide regional
high-availability
● Data is moved to S3 Standard
asynchronously to reduce
storage costs

29. 29