The need to enrich a fast, high volume data stream with slow-changing reference data is probably one of the most wide-spread requirements in stream processing applications. Apache Flink's built-in join functionalities and its flexible lower-level APIs support stream enrichment in various ways depending on the specific requirements of the use case at hand. In this webinar, I like to provide an overview of the basic methods to enrich a data stream with Apache Flink and highlight use cases, limitations, advantages and disadvantages of each.

1. © 2019 Ververica
Konstantin Knauf, @snntrable, Solutions Architect
99 WAYS TO ENRICH
STREAMING DATA WITH
APACHE FLINK

2. © 2019 Ververica2
Agenda
• Introduction
• Per-Record Reference Data Lookup
• Reference Data Pre-Loading
• Reference Data Change Stream
• Summary

3. © 2019 Ververica
Introduction

4. © 2019 Ververica4
Running Example
Sensor
Reference
Data
Apache Flink
• low update frequency
• one record per key
• high frequency
• many events per key over time
look up by sensorID
Sensor
Measurements
Enriched
Measurements

5. © 2019 Ververica
Per-Record Reference Data Lookup

6. © 2019 Ververica6
Per-Record Synchronous Lookup
Sensor
Reference
Data
Implementation in Flink:
● RichFlatMapFunction
● Database Client instantiated in open()

7. © 2019 Ververica7
Per-Record Synchronous Lookup
• very simple
• always uses up-to-date reference
data
• high latency
• high load on database
• low CPU utilization on
Taskmanagers
• low throughput
• overall not suitable for high
frequency streams
+ -

8. © 2019 Ververica8
Per-Record Asynchronous Lookup
Sensor
Reference
Data
Implementation in Flink:
● AsyncDataStream#unorderedWait
● https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/op
erators/asyncio.html

9. © 2019 Ververica9
Per-Record Asynchronous Lookup
Code

10. © 2019 Ververica10
Per-Record Asynchronous Lookup
• still pretty simple
• always uses up-to-date reference
data
• higher throughput than “Per-Record
Synchronous Lookups”
• high latency
• high load on database
• overall not suitable for high
frequency streams
+ -

11. © 2019 Ververica11
Per-Record (A)Synchronous Lookup with In-Memory Cache
Sensor
Reference
Data
C
a
c
h
e
C
a
c
h
e
Implementation in Flink:
● RichFlatMapFunction
● Simple HashMap or cache implementation of your favorite library

12. © 2019 Ververica12
Per-Record (A)Synchronous Lookup with In-Memory Cache
• lower load on database
• higher throughput if cache hit rate
is high
• low latency if cache is hit
• high tail latencies
• high load on database during warm
up phase, possibly hard to predict
• events might be enriched by stale
data
• cache size limited by available
memory
+ -

13. © 2019 Ververica
Reference Data Pre-Loading

14. © 2019 Ververica14
Pre-Loading of Reference Data
Sensor
Reference
Data
Implementation in Flink:
● RichFlatMapFunction
● HashMap populated in open()

15. © 2019 Ververica15
Pre-Loading of Reference Data
• usually very simple (depending on
the database client)
• high throughput
• low latency
• reference data size limited by
memory of single Taskmanager
• events are enriched with stale data,
no updates to reference data
+ -

16. © 2019 Ververica16
Partitioned Pre-Loading of Reference Data
Sensor
Reference
Data
Implementation in Flink:
● DataStream#partitionCustom
● RichFlatMapFunction

17. © 2019 Ververica17
Partitioned Pre-Loading of Reference Data
Code

18. © 2019 Ververica18
Partitioned Pre-Loading of Reference Data
• high throughput
• low latency
• reference data size limited by
memory of all Taskmanagers
• events are enriched with stale data,
no updates to reference data
• requires custom partitioning of
DataStream
+ -

19. © 2019 Ververica19
Periodic (Partitioned) Pre-Loading of Reference Data
Sensor
Reference
Data
Implementation in Flink:
● CoProcessFunction
● Processing or Event Time Timers for Reloading

20. © 2019 Ververica20
Periodic (Partitioned) Pre-Loading of Reference Data
• high throughput
• low latency
• staleness of reference data limited
by refresh interval
• reference data size limited by
memory of all Taskmanagers
• events are enriched with stale
reference data
• load spikes/high tail latencies during
refresh of reference data
+ -

21. © 2019 Ververica21
Per-Record Lookup with Initial Cache Pre-Loading
Sensor
Reference
Data
C
a
c
h
e
C
a
c
h
e
Implementation in Flink:
● RichFlatMapFunction
● Cache Preloading in open()

22. © 2019 Ververica22
Per-Record Lookup with Initial Cache Pre-Loading
+ -
• high throughput
• low latency
• staleness of reference data limited
by cache timeout
• reference data size limited by
memory of all Taskmanagers
• events are enriched with stale
reference data
• load spikes/high tail latencies
depending on cache miss rate

23. © 2019 Ververica23
Half-Time Assessment
• multiple solutions for enrichment via external database
• So far, always a trade-off between
– database load
– staleness of reference data
– reference data size
– latency & throughput of event stream

24. © 2019 Ververica
Reference Data Change Stream

25. © 2019 Ververica25
High-Level Architecture
Sensor
Reference
Data
Apache Flink
• low update frequency
• one record per key
• high frequency
• many events per key over time
look up by sensorID
Sensor
Measurements
Enriched
Measurements

26. © 2019 Ververica26
More Streamy High-Level Architecture
Sensor
Reference
Data
Apache Flink
• database updates are captured and written
into message queue
• local stream join instead of
external lookup
Sensor
Measurements
Enriched
Measurements
Sensor
Reference Data
Updates

27. © 2019 Ververica27
High-Level Streaming/Event-Driven Architecture
Sensor
Reference
Data
Apache Flink
• ground truth is moved to message broker
• Former sensor reference database and
Apache Flink consume the same stream of
reference data updates
Sensor
Measurements
Enriched
Measurements
Sensor
Reference Data
Updates
Sensor Management
System
• local stream join instead of
external lookup

28. © 2019 Ververica28
Simple Streaming Enrichment
KeyedCoProcessFunction
Implementation in
Flink:
● Key by sensorId
● ValueState<
SensorReference
Data>

29. © 2019 Ververica29
Simple Streaming Enrichment
Code

30. © 2019 Ververica30
Simple Streaming Enrichment
+ -
• high throughput
• low latency
• always uses up-to-date reference
data
• reference data size not limited
(RocksDB) by memory
• might necessitate change of
high-level architecture or a
conversation with DBAs
• events might be enriched by
reference data from the future

31. © 2019 Ververica31
Simple Event Time Join
CoProcessFunction
Look up
based on
event time
Implementation in
Flink:
● Key by sensorId
● MapState<Long,
SensorReference
Data>

32. © 2019 Ververica32
Simple Event Time Join
+ -
• high throughput
• low latency
• always uses latest available
reference data for each record
• reference data size not limited
(RocksDB) by memory
• might necessitate change of
high-level architecture or a
conversation with DBAs

33. © 2019 Ververica33
Temporal Table Join
KeyedCoProcessFunction
On each watermark:
joins all events up till
watermark correct
reference
Implementation in Flink:
● https://ci.apache.org/pro
jects/flink/flink-docs-rele
ase-1.8/dev/table/strea
ming/temporal_tables.ht
ml
● org.apache.flink.table.r
untime.join.TemporalR
owtimeJoin

34. © 2019 Ververica34
Temporal Table Join
+ -
• high throughput
• always uses latest available
reference data for each record &
reference data is complete
• reference data size not limited
(RocksDB) by memory
• might necessitate change of
high-level architecture or a
conversation with DBAs
• higher latency

35. © 2019 Ververica35
TL;DL
• Flink provides numerous ways to enrich streaming data with slow changing reference
data.
• The highest performance and cleanest semantics result from a stream processing
architecture and streaming enrichment methods like Temporal Table Joins.
• More Resources
– https://www.ververica.com/resources/flink-forward-san-francisco-2019/how-to-join-two-data-streams
– https://github.com/knaufk/enrichments-with-flink
• Join the community!
– Subscribe to mailing lists
– Join Flink Forward Europe 2019 in October

36. © 2019 Ververica
Questions?

37. © 2019 Ververica
www.ververica.com @VervericaDatakonstantin@ververica.com