Recommended
More Related Content
Similar to Strumienie i wiewiórka
Similar to Strumienie i wiewiórka (20)
Recently uploaded
Recently uploaded (20)
Strumienie i wiewiórka
- 3. In computer science, a stream is a sequence of data elements made available over time. A stream can be thought of as items on a conveyor belt being processed one at a time rather than in large batches. Streams are processed differently from batch data – normal functions cannot operate on streams as a whole, as they have potentially unlimited data, and formally, streams are codata (potentially unlimited), not data (which is finite). Functions that operate on a stream, producing another stream, are known as filters, and can be connected in pipelines, analogously to function composition. Filters may operate on one item of a stream at a time, or may base an item of output on multiple items of input, such as a moving average. Wikipedia https://en.wikipedia.org/wiki/Stream_(computing)
- 8. 5s 10s 15s 25s Tumbling window: 5s 10s 11s 16s 8s 13s 14s 19s Sliding window: stream.keyBy(0).window(TumblingEventTimeWindows.of(Time.seconds(5))) stream.keyBy(0).window(SlidingEventTimeWindows.of(Time.seconds(5), Time.seconds(2)))
- 9. 5s 12s 13s 25s Counting window: 5s 15s 20s Session window: stream.keyBy(0).countWindow(4) stream.keyBy(0).window(EventTimeSessionWindows.withGap(Time.seconds(5)))
- 10. 5s 10s 15s 25s 2s 3s 4s 8s 11s 9s14s 11s 11s 14s 9s
- 11. 5s 10s 15s 25s 2s 3s 4s 8s 11s 9s14s 11s 11s 14s 9s Late EventOut of Order Event
- 19. Spark ➖ ➖ ➖
- 20. ➖ Storm
- 21. ➖ Kafka-streams
- 22. Samza Samza ➖ ➖
- 23. Największy job ma ponad 20 operatorów, działa na 5000 vCorów w 1000 węzłowym klastrze. Zadania przetwarzające 30 miliardów eventów dziennie, utrzymując stan setek GB z zapewnieniem jednokrotnego przetwarzania. 30 aplikacji używanych na produkcji przez ponad rok. 10 miliardów eventów przetwarzanych dziennie(2TB) https://www.youtube.com/watch?v=NI1yN5FbxUM
- 25. val env = StreamExecutionEnvironment.getExecutionEnvironment val tableEnv = TableEnvironment.getTableEnvironment(env) val eventSteam = ... tableEnv.registerDataStream("events", eventStream) tableEnv.sql("select * from events where elem = 'AddCredit'").toDataStream[QueryableButtonClickEvent].print() eventStream.toTable(tableEnv).where('elem === "AddCredit").toDataStream[QueryableButtonClickEvent].print() env.execute("Sql job") SQL Table API Register table https://www.youtube.com/watch?v=vws5bv3XdD8&feature=youtu.be
- 27. 1 2
- 28. val shortSessionPattern = Pattern .begin[ClickEvent]("login").subtype(classOf[LoginClickEvent]) .followedBy("logout").within(Time.seconds(5)) CEP.pattern(eventStream, shortSessionPattern).select(events => s"User: ${events("login").userId} spent just ${(events("logout").timestamp - events("login").timestamp) / 1000} seconds" ).print() https://www.youtube.com/watch?v=vws5bv3XdD8&feature=youtu.be
- 29. val creditCardRequest = Pattern .begin[ClickEvent]("credit").subtype(classOf[ButtonClickEvent]).where(_.elem == Element.AddCredit) .followedBy("card").subtype(classOf[ButtonClickEvent]).where(_.elem == Element.RequestCard) CEP.pattern(eventStream, creditCardRequest).select(events => s"User: ${events("credit").userId} probably wants a credit" ).print()
- 31. Query Client
- 32. Live demo