Ghostbusters better get ready, because it's time to cross (ok, join) some streams! This talk will include easy-to-follow steps to set up and maximize a powerful, streaming data pipeline with the newest features from Apache Flink. This talk is for anyone using (or interested in) stream processing who wants to minimize their development overhead, and particularly for those who want to do so while leveraging available Open Source tools.

1. © 2020 Ververica
1

2. © 2020 Ververica
2
● Caito Scherr
Introduction

3. © 2020 Ververica
3
● Caito Scherr
● Developer Advocate
Introduction

4. © 2020 Ververica
4
● Caito Scherr
● Developer Advocate
● Ververica, GmbH
Introduction

5. © 2020 Ververica
5
● Caito Scherr
● Developer Advocate
● Ververica, GmbH
● Portland, OR, USA
Introduction

6. © 2020 Ververica
6
Introduction
Introduction

7. © 2020 Ververica
7
Introduction

8. © 2020 Ververica
8
● The Challenge
● The Tools
● The Demo
Agenda

9. © 2020 Ververica
9
● The Challenge
● The Tools
● The Demo
Agenda

10. © 2020 Ververica
10
● The Challenge
● The Tools
● The Demo
Agenda

11. © 2020 Ververica
11
● The Challenge
● The Tools
● The Demo
Agenda

12. © 2020 Ververica
12

13. © 2020 Ververica
13
● Stream processing is becoming more popular.
The Challenge

14. © 2020 Ververica
14
● Stream processing is becoming more popular.
● … It’s likely you’ll need to be familiar with it
The Challenge

15. © 2020 Ververica
15
● Stream processing is becoming more popular.
● … It’s likely you’ll need to be familiar with it
● … even if it’s still “new” to your company.
The Challenge

16. © 2020 Ververica
16
● Stream processing is becoming more popular.
● … It’s likely you’ll need to be familiar with it
● … even if it’s still “new” to your company.
● Companies usually pick streaming to solve an already
complex situation.
The Challenge

17. © 2020 Ververica
17
● Find an easier way to start streaming
● Have solid observability accessible by all impacted teams
The Challenge >> Efficient Solutions

18. © 2020 Ververica
18

19. © 2020 Ververica
19
The Tools
● Stateful, distributed
● Stream processing engine
● Unified batch & streaming
>> Flink: High Level

20. © 2020 Ververica
20
The Tools
● Stateful, distributed
● Stream processing engine
● Unified batch & streaming
● Scalable
● Highly configurable
● “Deploy anywhere”
>> Flink: High Level

21. © 2020 Ververica
21
>> Flink’s APIs
The Tools

22. © 2020 Ververica
22
>> Flink: Unified Processing
● Reuse code and logic
● Consistent semantics
● Mix historic and real-time
● Simplify operations
The Tools

23. © 2020 Ververica
23
>> Flink: Architecture
The Tools

24. © 2020 Ververica
24
>> Flink: Basic Example
The Tools

25. © 2020 Ververica
25
>> Flink: Basic Example
The Tools

26. © 2020 Ververica
26
>> Flink: Basic Example
The Tools

27. © 2020 Ververica
27
>> Flink: Basic Example
The Tools

28. © 2020 Ververica
28
>> Flink: Basic Example
The Tools

29. © 2020 Ververica
29
Flink Runtime
Stateful Computations over Data Streams
Stateful
Stream Processing
Streams, State, Time
Event-Driven
Applications
Stateful Functions
Streaming
Analytics & ML
SQL, PyFlink, Tables
Image Credit: Marta Paes
The Tools >> Why Flink SQL?

30. © 2020 Ververica
30
>> Simpler with Flink SQL
The Tools

31. © 2020 Ververica
31
● Unified, distributed
● Less implementation overhead
● Developer autonomy
The Tools >> Why Flink SQL?

32. © 2020 Ververica
32
user cnt
Mary 2
Bob 1
SELECT user_id,
COUNT(url) AS cnt
FROM clicks
GROUP BY user_id;
Take a snapshot when the
query starts
A final result is
produced
A row that was added after the query
was started is not considered
user cTime url
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
The query
terminates
>> Compare: Regular SQL Engine
Image Credit: Marta Paes
The Tools

33. © 2020 Ververica
33
user cTime url
user cnt
SELECT user_id,
COUNT(url) AS cnt
FROM clicks
GROUP BY user_id;
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
Bob 1
Liz 1
Mary 1
Mary 2
Ingest all changes as
they happen
Continuously update the
result
The result is identical to the one-time query (at this point)
>> A Streaming SQL Engine
Image Credit: Marta Paes
The Tools

34. © 2020 Ververica
34
● Standard SQL syntax and semantics (i.e. not a “SQL-flavor”)
● Unified APIs for batch and streaming
● Support for advanced time handling and operations (e.g. CDC, pattern matching)
UDF Support
Python
Java
Scala
Execution
TPC-DS Coverage
Batch
Streaming
+
Formats
Native Connectors
Apache Kafka
Elasticsearch
FileSystems
JDBC HBase
+
Kinesis
Metastore Postgres (JDBC)
Data Catalogs
Debezium
>> Flink SQL In A Nutshell
Image Credit: Marta Paes
The Tools

35. © 2020 Ververica
35

36. © 2020 Ververica
36
>> Flink SQL Pre-Requisites
1. Check Java version
2. Download Flink snapshot
3. Un-tar it
Getting Started

37. © 2020 Ververica
37
What Next? >> Flink SQL Cookbook

38. © 2020 Ververica
38

39. © 2020 Ververica
39

40. © 2020 Ververica
40

41. © 2020 Ververica
41

42. © 2020 Ververica
42

43. © 2020 Ververica
43
>> More Data!
The Demo
● Flink SQL + Datagen
● Same startup steps

44. © 2020 Ververica
44

45. © 2020 Ververica
45

46. © 2020 Ververica
46
>> Try out the full demo
What’s Next?
● Different storage systems
● Hive Metastore as an external, persistent catalog
● Batch/Stream unification of queries in action
● Different ways to join dynamic data
● Continuous SQL queries in Kafka and MySQL

47. © 2020 Ververica
47
What Next? >> Flink SQL Cookbook
Image Credit: Marta Paes

48. © 2020 Ververica
48
What Next? >> Flink SQL Cookbook

49. © 2020 Ververica
● Chad Green & the
CodepaLOUsa conference
staff!
● Marta Paes
49
Scan here for links
& resources
Thank You!

50. © 2020 Ververica
50
Find Me Here
Twitter
Caito_200_OK
Content
https://medium.com/@caito
http://caito-200-ok.com/
Email
Caito@ververica.com