Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective? In this talk we’ll give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons. To highlight the differences, we’ll dive into a project we’ve done, transitioning from reading Kafka in a stream to reading it in batch. By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we’ve achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution. We’ll explore the tradeoffs and give you the metrics and intuition you’ll need to make such decisions yourself. We’ll cover: Costs of processing in stream compared to batch Scaling up for bursts and reprocessing Making the tradeoff between wait times and costs Recovering from outages And much more…

1. Should you read Kafka
as a stream or in
batch?

2. Stream
● Micro deliveries
● Many trips
● AND - Low waits

3. Batch
Photo by Charles J. Sharp
● Large deliveries
● Few trips
● BUT - Long waits

4. This is a story about a stream …..
gone batch

5. o Project evolution
o Metrics and Results
o Insights for your projects
Agenda

6. Who Are We ?
Opher Dubrovsky
Big Data Dev Lead
Ido Nadler
Big Data Engineer
Data Pipelines, Serverless, Spark, Analytics

7. Nielsen Marketing Cloud
Cloud native
~6,000
nodes/day
~60 TB/day
5 PetaByte
~6 million/day
Create data for:
› Running campaigns
› Business decisions

8. Our Old Stream Project
Data Lake
AWS S3
Total: 60 TB/day
Biggest topic: 27 TB/day

9. Reality
Check
Photo by Olia Nayda on

10. Total Cost
~$400,000 per Year
Daily Costs ($)

11. 300
400
500
600
700
800
900
1,000
1,100
1,200
0
:
0
0
2
:
0
0
4
:
0
0
6
:
0
0
8
:
0
0
1
0
:
0
0
1
2
:
0
0
1
4
:
0
0
1
6
:
0
0
1
8
:
0
0
2
0
:
0
0
2
2
:
0
0
0
:
0
0
2
:
0
0
4
:
0
0
6
:
0
0
8
:
0
0
1
0
:
0
0
1
2
:
0
0
1
4
:
0
0
1
6
:
0
0
1
8
:
0
0
2
0
:
0
0
2
2
:
0
0
0
:
0
0
2
:
0
0
4
:
0
0
6
:
0
0
8
:
0
0
1
0
:
0
0
1
2
:
0
0
1
4
:
0
0
1
6
:
0
0
1
8
:
0
0
2
0
:
0
0
2
2
:
0
0
GB / Hour
Problems
Cluster is
Idle
Wasted $$$

12. Data Bursts (& reprocessing)
Cluster is
underpowered
Cluster
Capacity
Long Queues !

13. Data Expiration
Processing has to be Quick

14. Slow Recovery == Ticking Timebomb
Risk to Your DATA !

15. Rearchitecting
Rearchitecting the stream to batch……

16. Spread the work
across isolated workers

17. Using Discreet Time Slots
time
4:00 5:00 6:00 7:00
3:00 8:00 9:00
Task
Task
Task

18. Task (& Cluster) Independence
Processing
Hour
4:00
5:00
6:00
8:00
3:00
7:00
9:00
1 hour 2 hours 3 hours
Some tasks are short
No more paying for
idle time !
Some tasks >1 hour

19. Mechanics
Airflow AWS S3
{ period: 05:00 - 06:00 }
{ period: 06:00 - 07:00 }
{ period: 07:00 - 08:00 }

20. Comparing to Previous System
Cluster
Capacity
Wasted
Processing
Efficiency was 30% ONLY !!
New System is 60% Cheaper !

21. Results

22. From Micro-Batch to Batch
1 hour
1 hour
1 hour 1 hour 1 hour
Missed run
Rerun
2 hours

23. Pay as You Go
Bytes per Hour
$ per Hour
$400k /year à $160k /year

24. Outage Handling
Outage start
6 hours Recovery
completed
Old System
New System
Outage start Recovery
completed
~85% Faster Recovery
40 min
5 Concurrent
isolated workers

25. Summary

26. Conclusions and Insights
o Streaming is expensive
o Fixed clusters are never perfect
o Off hours - $$$
o High loads - Slow
o Recovery is critical
o Isolation and parallelism allows you to
o Easily scale
o Save on costs
o Deal with loads

27. /opher-dubrovsky
@jumpja
/ido-nadler
@IdoNadler
You Can Reach Us At:
Opher Dubrovsky
Ido Nadler

28. Thank You