This document summarizes Daria Litvinov's presentation on using Druid, Apache Spark, and Kafka for real-time analytics. The presentation covers setting up real-time dashboards using these technologies, addressing issues like data loss on job restarts, and the solution of committing Kafka offsets manually and storing them synchronously.

1. Daria Litvinov | SW Engineer, Outbrain
November 2019
Real Time Analytics
with Druid, Apache
Spark and Kafka

2. SW Engineer with more than 15 years of
experience
Dealing with Big Data technologies
Druid, Spark, Kafka
Daria Litvinov

3. What is

4. That wonderful moment when
you discover something you
never knew existed.
We are Discovery

5. 1.8 Billion
Pages a Day**
290 Billion
Monthly Discoveries**
820 Million
Monthly Consumers*
*Comscore, 2019, **Outbrain internal data, March 2019
We are Global

6. • The Motivation
• Architecture
• The Problem
• The Way to the Solution
Agenda

7. The Motivation

8. The Motivation
Serving
(Clicks, Impressions)

9. The Motivation
Serving
(Clicks, Listing)
4 Hour
Cycle

10. Real Time Dashboard
Based on Kafka Events

11. Apache Druid
Druid is a column-oriented,
open-source, distributed data store.
Imply is a high-performance analytics solution
to store, query, and visualize streaming.

12. + =
Druid & Kafka

13. Design & Architecture
● Kafka topic format

14. ● Kafka topic format
● Kafka topic content
[Click_id, publisher_id, marketer_id,
countryCode,...]
Design & Architecture

15. input_events
Avro,
raw data
output_events
JSON,
enriched
Design & Architecture

16. Let’s Get Started!

17. Let’s Get Started!
Spark
Kafka
Spark Streaming Kafka Integration Guide
Deploy Spark Job to Production
Load Data Into Druid

18. Data in Druid

19. Spark Streaming Jobs Run Forever?
Not Really!

20. Data Loss on
Job Restart
Spark Streaming Jobs Run Forever?
Not Really!

21. input_events
Avro,
raw data
output_events
JSON,
enriched
Design & Architecture

22. It’s All About
Offsets!

23. Apache Kafka
● Distributed Streaming Platform
● Producer API
● Consumer API

24. Apache Kafka –
Consuming a Topic
● latest
● earliest
● from committed offsets
19 20 21 22 23 24 25 26 27 28
Time
29

25. Data Loss on
Job Restart
Data Loss on Job Restart

26. Step 1:
Auto Commit Offsets

27. Auto Commit Offsets
enable.auto.commit
auto.commit.interval.ms

28. Step 2:
Commit Kafka Offsets Manually

29. Micro-batches of 2 minutes
RDD RDD RDD RDD
t0 t1 t2 ti ti+1
Spark Streaming –
Micro Batches

30. Call commitAsync() in the end of each micro-batch
Commit Kafka Offsets Manually

31. Commit Kafka Offsets Manually

32. Step 3:
Graceful Shutdown

33. ● Go to the sparkUI and kill the application
● Use API to kill the application
● Graceful Shutdown
○ Create the context
spark.streaming.stopGracefullyOnShutdown=true
○ Stop the context
ssc.stop(stopSparkContext=true, stopGracefully=true)
How To Shutdown Spark Application

34. ● Go to the sparkUI and kill the application
● Use API to kill the application
● Graceful Shutdown
○ Create the context
spark.streaming.stopGracefullyOnShutdown=true
○ Stop the context
ssc.stop(stopSparkContext=true, stopGracefully=true)
How To Shutdown Spark Application

35. • Stop reading from Kafka
• Process queued events
• Stops the job’s execution
Spark Streaming - Graceful Shutdown

36. Graceful Shutdown – Check for Shutdown

37. Check for
HDFS file
Graceful Shutdown – Check for Shutdown

38. /**
Queue up offset ranges for commit to Kafka at a future
time. Threadsafe.
*/
Spark CommitAsync API

39. Step 4:
Store Kafka Offsets Synchronously

40. The Final Solution

41. Graceful
Shutdown
Store Kafka Offsets
Synchronously
The Final Solution

42. It Works!
Job Restarted
Job Restarted

43. Now We Have
Reliable Data in Druid

44. Journey