Streams are sequences of data elements made available over time, like items on a conveyor belt. Functions that operate on streams are called filters and can transform one stream into another. Filters may operate on individual stream elements or use multiple elements, such as calculating a moving average. Streams are processed differently than batch data as they have potentially unlimited size.

2. ●
●
○
○
○
●
○
○
○
●

3. In computer science, a stream is a sequence of data elements made available over time. A stream can be thought of as items on a conveyor
belt being processed one at a time rather than in large batches.
Streams are processed differently from batch data – normal functions cannot operate on streams as a whole, as they have potentially
unlimited data, and formally, streams are codata (potentially unlimited), not data (which is finite). Functions that operate on a stream,
producing another stream, are known as filters, and can be connected in pipelines, analogously to function composition. Filters may operate
on one item of a stream at a time, or may base an item of output on multiple items of input, such as a moving average.
Wikipedia https://en.wikipedia.org/wiki/Stream_(computing)

4. ●
●
○
○
●
○
○

5. https://ci.apache.org/projects/flink/flink-docs-release-1.1/fig/stack.png

7. ●
●
●
https://www.youtube.com/watch?v=-LiZR1457Kw

8. 5s 10s 15s 25s
Tumbling window:
5s 10s 11s 16s
8s 13s 14s 19s
Sliding window:
stream.keyBy(0).window(TumblingEventTimeWindows.of(Time.seconds(5)))
stream.keyBy(0).window(SlidingEventTimeWindows.of(Time.seconds(5), Time.seconds(2)))

9. 5s 12s 13s 25s
Counting window:
5s 15s 20s
Session window:
stream.keyBy(0).countWindow(4)
stream.keyBy(0).window(EventTimeSessionWindows.withGap(Time.seconds(5)))

10. 5s 10s 15s 25s
2s 3s 4s 8s 11s 9s14s 11s 11s 14s 9s

11. 5s 10s 15s 25s
2s 3s 4s 8s 11s 9s14s 11s 11s 14s 9s
Late EventOut of Order Event

13. https://ci.apache.org/projects/flink/flink-docs-release-1.2/fig/tasks_slots.svg

14. https://ci.apache.org/projects/flink/flink-docs-release-1.1/concepts/fig/processes.svg

15. ●
●
●
●
https://www.youtube.com/watch?v=_qWs_rMUKPQ

16. https://ci.apache.org/projects/flink/flink-docs-release-1.1/internals/fig/stream_barriers.svg

17. https://ci.apache.org/projects/flink/flink-docs-release-1.1/internals/fig/stream_aligning.svg

18. https://ci.apache.org/projects/flink/flink-docs-release-1.1/internals/fig/checkpointing.svg
https://www.youtube.com/watch?v=9VjiIKco3bE

19. Spark
➖
➖
➖

20. ➖
Storm

21. ➖
Kafka-streams

22. Samza
Samza
➖
➖

23. Największy job ma ponad 20 operatorów, działa na 5000 vCorów w
1000 węzłowym klastrze.
Zadania przetwarzające 30 miliardów eventów dziennie, utrzymując
stan setek GB z zapewnieniem jednokrotnego przetwarzania.
30 aplikacji używanych na produkcji przez ponad rok. 10 miliardów
eventów przetwarzanych dziennie(2TB)
https://www.youtube.com/watch?v=NI1yN5FbxUM

24. Najciekawsze
funkcjonalności

25. val env = StreamExecutionEnvironment.getExecutionEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
val eventSteam = ...
tableEnv.registerDataStream("events", eventStream)
tableEnv.sql("select * from events where elem = 'AddCredit'").toDataStream[QueryableButtonClickEvent].print()
eventStream.toTable(tableEnv).where('elem === "AddCredit").toDataStream[QueryableButtonClickEvent].print()
env.execute("Sql job")
SQL
Table API
Register table
https://www.youtube.com/watch?v=vws5bv3XdD8&feature=youtu.be

26. https://flink.apache.org/img/blog/stream-sql/new-table-api.png

27. 1
2

28. val shortSessionPattern = Pattern
.begin[ClickEvent]("login").subtype(classOf[LoginClickEvent])
.followedBy("logout").within(Time.seconds(5))
CEP.pattern(eventStream, shortSessionPattern).select(events =>
s"User: ${events("login").userId} spent just
${(events("logout").timestamp - events("login").timestamp) / 1000} seconds"
).print()
https://www.youtube.com/watch?v=vws5bv3XdD8&feature=youtu.be

29. val creditCardRequest = Pattern
.begin[ClickEvent]("credit").subtype(classOf[ButtonClickEvent]).where(_.elem == Element.AddCredit)
.followedBy("card").subtype(classOf[ButtonClickEvent]).where(_.elem == Element.RequestCard)
CEP.pattern(eventStream, creditCardRequest).select(events =>
s"User: ${events("credit").userId} probably wants a credit"
).print()

30. KV Store
Query Client
https://www.youtube.com/watch?v=uuv-lnOrD0o&feature=youtu.be

31. Query Client

32. Live demo

34. Dawid Wysakowicz
@OneMoreCoder
dawidwys
wysakowicz.dawid@gmail.com

35. ●
○
○
○
○
●
○
■
■
●
○
○
○