Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FlinkCEP Library - Dawid Wysakowicz, GetInData (WHUG)

380 views

Published on

Dawid Wysakowicz - Flink Committer & Data Engineer at GetinData had a pleasure to give a presentation at Warsaw Hadoop User Grooup meetup which took place in Warsaw on July 3rd, 2017. They came forth with Flink Complex Event Processing (CEP) library.
Please take a look at the presentation with some extra slides

If you would like to know more about Apache Flink check our 2-days training provided by Dawid: Real-Time Stream Processing, http://getindata.com/real-time-stream-processing/

Published in: Technology
  • Be the first to comment

FlinkCEP Library - Dawid Wysakowicz, GetInData (WHUG)

  1. 1. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Looking for Patterns - FlinkCEP Library _D. Wysakowicz WHUG 03.07.2017
  2. 2. © Copyright. All rights reserved. Not to be reproduced without prior written consent. About me ■ Data Engineer at GetInData ■ Apache Flink Committer ■ Involved in Flink CEP library development Dawid Wysakowicz @OneMoreCoder
  3. 3. © Copyright. All rights reserved. Not to be reproduced without prior written consent. That moment when party goes wrong.
  4. 4. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Were there any signs? Any Patterns? That moment when party goes wrong.
  5. 5. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Image source: “Continuous Analytics: Stream Query Processing in Practice”, Michael J Franklin, Professor, UC Berkley, Dec 2009 and http://www.slideshare.net/JoshBaer/shortening-the-feedback-loop-big-data-spain-external
  6. 6. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
  7. 7. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Buying Home Insurance Online Property Info Offer Accept General Terms of Contract PaymentScope Change Info Change
  8. 8. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Buying Home Insurance Online Property Info Offer Accept Exit Scope Change Info Change General Terms of Contract Payment
  9. 9. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Simple Example - Matching Payments Notify whenever a payment for accepted offer arrives
  10. 10. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Create Pattern
  11. 11. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Apply Pattern
  12. 12. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Select Matches
  13. 13. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Handle Results
  14. 14. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Seamless Integration - Standard Pipeline DataStream Source e.g. DataStream Sink e.g. DataStream Transformations
  15. 15. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Seamless Integration - CEP Library DataStream Source e.g. DataStream Sink e.g. DataStream Transformations CEP DataStream Pattern DataStream Stream of matches
  16. 16. © Copyright. All rights reserved. Not to be reproduced without prior written consent. CEP Resulting Matches CEP DataStream Pattern A -> B -> C C2C1B2B1A DataStream Stream of matches A, B1, C1 A, B2, C1 A, B1, C2 A, B2, C2
  17. 17. © Copyright. All rights reserved. Not to be reproduced without prior written consent. CEP Timeouted Matches CEP DataStream Pattern A -> B -> C within 5 seconds C2(t=7)B2(t=4)B1(t=2)A(t = 1) DataStream Stream of timeouted matches W=6 ?
  18. 18. © Copyright. All rights reserved. Not to be reproduced without prior written consent. CEP Timeouted Matches CEP DataStream Pattern A -> B -> C within 5 seconds C2(t=7)B2(t=4)B1(t=2)A(t = 1) DataStream Stream of timeouted matches A A, B1 A, B2 W=6
  19. 19. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Pattern Sequence
  20. 20. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Patterns Basic building blocks
  21. 21. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Conditions Basic building blocks Condition specification
  22. 22. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Time Restrictions Basic building blocks Condition specification Time restrictions
  23. 23. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Time Restrictions Basic building blocks Condition specification Time restrictions NOTE: The time restriction is a global property. This is a period in which either whole match is generated or partial matches are timeouted.
  24. 24. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next Consuming Strategy
  25. 25. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity Consuming Strategy
  26. 26. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity ■ Skip till any Consuming Strategy
  27. 27. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity ■ Skip till any ■ Not follow Consuming Strategy
  28. 28. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity ■ Skip till any Consuming Strategy ■ Not follow
  29. 29. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity ■ Skip till any ■ Not follow ■ Not next Consuming Strategy
  30. 30. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Skip till next ■ Strict continuity ■ Skip till any Consuming Strategy ■ Not follow ■ Not next
  31. 31. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Consuming Strategy ■ Skip till next ■ Strict continuity ■ Skip till any ■ Not follow ■ Not next
  32. 32. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Looping Example How many offer changes before accepting? ... ?
  33. 33. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Pattern Specification Quantifier application Basic building blocks Condition specification Time restrictions
  34. 34. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Quantifiers ■ Singleton pattern ■ Complex pattern ● Times (count) ● Looping pattern
  35. 35. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Quantifiers ■ Singleton pattern ■ Complex pattern ● Times (count) ● Looping pattern ■ Optional pattern ● singleton -> optional ● times -> no or exactly n times ● oneOrMore -> zeroOrMore
  36. 36. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Complex Patterns - Consuming Strategies ?
  37. 37. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ? Complex Patterns - Consuming Strategies
  38. 38. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ? Complex Patterns - Consuming Strategies
  39. 39. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ? Complex Patterns - Consuming Strategies
  40. 40. © Copyright. All rights reserved. Not to be reproduced without prior written consent. ■ Consuming strategy - before first element ■ Inner consuming strategy - between events matched in the Pattern Complex Patterns - Consuming Strategies
  41. 41. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Iterative Condition Trigger alert when the same property changed. PL Flood PL
  42. 42. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Iterative Condition
  43. 43. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Iterative Condition Trigger alert when the same property changed. PL Flood PL
  44. 44. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
  45. 45. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Complex Example Alert when a user constantly decreases value of the property -12.000€ -14.000€ -24.000€ -100.000€
  46. 46. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Complex Example - Implementation Alert when a user changed value by more than 200% of previous average change
  47. 47. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Summary ■ Shortest possible feedback loop ■ Seamless integration with Flink ecosystem ■ Complex patterns ● Dynamic length ● Conditions based on calculations
  48. 48. © Copyright. All rights reserved. Not to be reproduced without prior written consent. How does it works underneath?
  49. 49. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventStream Order handling 1 5 4 2 PatternOperator ElementsQueue 1 2 4 5 NFA
  50. 50. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventStream Order handling 1 5 4 82 W=5 PatternOperator ElementsQueue 1 2 4 5 W=5 NFA
  51. 51. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventStream Order handling 1 5 4 82 W=5 PatternOperator ElementsQueue 1 2 4 5 W=5 NFA MatchesStream A B
  52. 52. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventStream Order handling - late elements 1 5 4 8 32 W=5 PatternOperator ElementsQueue 1 2 4 5 W=5 NFA MatchesStream A B
  53. 53. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventStream Order handling - late elements 1 5 4 8 32 W=5 PatternOperator ElementsQueue 1 2 4 5 W=5 NFA MatchesStream A B
  54. 54. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Nondeterministic Finite Automaton ■ Graph where: ● Vertices = States ● Edges = Transitions ■ IGNORE - event not consumed ■ TAKE - event consumed ■ PROCEED - event analyzed in the target state
  55. 55. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Example of NFA Pattern .begin(“A”) .followedByAny(“B”).optional() .next(“C”) .followedBy(“D”)
  56. 56. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE Pattern .begin(“A”) .followedByAny(“B”).optional() .next(“C”) .followedBy(“D”)
  57. 57. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE
  58. 58. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A
  59. 59. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A
  60. 60. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream A B Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE
  61. 61. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream A B Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE
  62. 62. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A B
  63. 63. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B?
  64. 64. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A B C
  65. 65. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A B? C D TAKE TAKE TAKE PROCEED IGNORE IGNORE IGNORE TAKE A B C
  66. 66. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B? D B? D A C
  67. 67. © Copyright. All rights reserved. Not to be reproduced without prior written consent. EventsStream Example of NFA A A B? A B A C B? D B? D A C D Match: A, C, D B? Match: A, B, C, D A
  68. 68. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  69. 69. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  70. 70. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  71. 71. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  72. 72. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  73. 73. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  74. 74. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  75. 75. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  76. 76. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  77. 77. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  78. 78. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  79. 79. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  80. 80. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  81. 81. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Shared Buffer - how it works MiddleStart End S1 S2 M1 M2 M3 E1 E2 1 2 1.0 1.0.0 1.0.0.0 2.0 1.0.0.0.0 1.0.0.0.1 2.0.0 Events order S1 M1 M2 S2 M3 E1 E2
  82. 82. © Copyright. All rights reserved. Not to be reproduced without prior written consent. Summary ■ Shortest possible feedback loop ■ Seamless integration with Flink ecosystem ■ Complex patterns ● Dynamic length ● Conditions based on calculations

×