Ivan Kosianenko discusses how AppsFlyer has evolved its data pipeline from using MemSQL and Druid to handle 12 billion events daily to now using ClickHouse and Druid to handle 38 billion events daily. He then explains how building a custom data streaming solution with Clojure can become complex due to issues like distributed orchestration, monitoring, and delivery guarantees. As an alternative, Ivan recommends using Spark Structured Streaming for its built-in functionality addressing these issues like write-ahead logging and exactly-once processing. He demonstrates how AppsFlyer leverages Spark Structured Streaming to ingest streaming data from Kafka into ClickHouse while monitoring streaming queries and end-to-end latency.

1. Productionize
Spark Structured Streaming
Ivan Kosianenko

2. Who am I
Software Architect at AppsFlyer
Data Engineering
Machine Learning
Ivan Kosianenko

3. AppsFlyer in a Nutshell

4. DEMO DATA

5. Time Series Lifetime Value
1.1.2018
1.1.2017
5.1.2018 5.1.2018
...
2.1.2018
3.1.2018
4.1.2018
5.1.2018
1.1.2018
2.1.2018
3.1.2018
4.1.2018
5.1.2018
- Mutable data Immutable data
VS

6. December 2017: MemSQL + Druid
Clojure
streamers
Live 1 Day Data
MemSQL
Cluster
Dashboard
Middleware
API
Druid
Daily 4 Year
LTV Table
12 billion events daily
KAFKA

7. Now: ClickHouse + Druid
Live 1 Day Data
ClickHouse
Cluster
Dashboard
Middleware
API
Druid
Daily 5 Year
LTV Table
38 billion events daily
KAFKA
Structured Streaming

8. How to DIY Clojure streamer?
● take clj-kafka and JDBC driver
● add multi-threaded execution
● write custom business logic

9. How to DIY Clojure streamer?
● distributed orchestration
● resource allocation
● retry logic
● monitoring
● delivery guarantee
● ???

10. The Law of Leaky Abstractions

11. Kafka Connect
Apache Storm
Heron Streaming
Onyx

12. Spark Structured Streaming
source
sink
business logic

13. Custom Foreach Writer
Add row
Write batch to anywhere
Create batch

14. Be careful!
For each row

15. Spark Structured Streaming
Write-ahead logging

16. ClickHouse
+
Exactly-once end-to-end processing

17. Spark Structured Streaming
Streaming query

18. Streaming Query
● Explain
● Stop
● Await termination
● Get termination error
● Track progress

19. Last progress

20. Checkpoint dir
Track lag between Spark and Kafka
1
2
3
1. Get last offsets from Spark
2. Get last offsets from Kafka
3. Send difference to statd

21. Dashboard and alerts

22. End-to-end test monitoring
1
2
3
1. Sends test message to
Kafka
2. Wait while message will
appear in Clickhouse
3. Send latency to statsd

23. Dashboard and alerts

24. Production config
turn off dynamic allocation
or
set min/max executors

25. Deploy with custom Spark operator
Airflow
Custom operator tries to figure out:
● Is current Spark job running?
● Is it latest version with latest config?
If no -> submit Spark job to YARN with
yarn.tag = hash(jar_version+run config)
Once in a minute

27. Join us
Thank you
ivan.kosianenko@appsflyer.com
t.me/appsflyerkyiv