The document discusses the migration of batch ETL processes to streaming ETL using Apache Flink, detailing the advantages of streaming over batch processing including faster results, resource efficiency, and flexibility in deployment. It covers topics such as Flink's architecture, different window operators, and the importance of testing and replaying in streaming applications. Additionally, it highlights various global computations and practices to handle challenges in maintaining data integrity during replays.

1. Migrating batch-ETLs to
streaming Flink
by William Saar, Updab

2. About the speaker
Flink, Kafka, Cassandra, Druid, Kubernetes, Java, Scala, Rust
● Updab - Independent Consulting
● tCell.io (acquired by Rapid7) - Consulting, Remote to SF
● King
● Cinnober Financial Technology (now Nasdaq)
● BEA Systems (now Oracle) - Java Mission Control (open-source!) developer
● Essnet (now Scientific Games)
● Digital Route - Telecom

3. Evolution and Trends

4. System Architecture: Yesterday
© 2019 Updab AB

5. System Architecture: Mainstream
© 2019 Updab AB

6. System Architecture: Emerging
© 2019 Updab AB

7. System Architecture: Future?
Queries
© 2019 Updab AB

8. Data pipeline evolution
Nightly Reports Streaming Applications“Real-time” analytics
© 2019 Updab AB

9. Batch vs Streaming

10. Pipeline types
● Well-defined batch
○ Nightly reports, ML parameter computations, Data cleaning
● “Wannabe-streaming” batch
○ Charts updating every minute, Alerting
● Streaming
○ Computation for every input (every event for Flink/Kafka Streams, Spark updateStateByKey)
© 2019 Updab AB

11. Benefits of streaming
● Faster results
● Incremental computation -> Less resources -> Simpler architecture
● Flexible deployment (Flink and Kafka Streams)
○ Lyft’s Kubernetes operator for Flink https://github.com/lyft/flinkk8soperator
● Always up-to-date queryable state (Flink and Kafka Streams)
© 2019 Updab AB

12. Apache Flink
● Widespread adoption and input source compatibility
○ Used for AWS Kinesis Data Analytics
● Rigorous time models
● Flexible state storage and control of intermediate states
© 2019 Updab AB

13. Batch architecture
© 2019 Updab AB

14. Streaming Flink Architecture
© 2019 Updab AB

15. Translating Batches

16. Flink Time Windows
© 2019 Updab AB

17. Other Window Operators
● Count windows
● Session windows
● Custom windows/Process functions/Co-process functions
© 2019 Updab AB

18. Testing

19. Testing
● End-to-end tests: Standalone Flink job with source and outputs replaced
○ Docker containers with Kafka, Postgres
● Structure code to support function or stream-segment testing
○ DataStream<Output> out = process(DataStream<Input> in)
○ FlinkSpector tool https://github.com/ottogroup/flink-spector
© 2019 Updab AB

20. Global Computations

21. Join keyed and broadcast state
© 2019 Updab AB

22. Global Computations
● KeyedBroadcastProcessFunction
● Probabilistic data structures: t-Digest, HyperLogLog
● External stream topic
● External service - AsyncFunction
© 2019 Updab AB

23. Replays

24. Replays: Why?
● Corrupt data or changing data sets
● Bugs in pipeline logic
● Test different pipeline logic
© 2019 Updab AB

25. Replays
● The Good: Single checkpoint synchronizes sources with intermediate states
● Challenges: External systems outside Flink’s control
© 2019 Updab AB

26. Replays: Techniques and helpful practices
● Idempotent writes
● Regular, predictable writes may allow overwriting
● Move external state into Flink - queryable state
© 2019 Updab AB

27. Thanks!
updab.com
william@updab.com
@saarw