Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Flink: The state of FlinkCEP

541 views

Published on

Pattern matching over event streams is increasingly being employed in many areas including financial services and click stream analysis. Flink, as a true stream processing engine, emerges as a natural candidate for these usecases. In this talk, we will present FlinkCEP, a library for Complex Event Processing (CEP) based on Flink. At the conceptual level, we will see the different patterns the library can support, we will present the main building blocks we implemented to support them, and we will discuss possible future additions that will further enhance the coverage of the library. At the practical level, we will show how the integration of FlinkCEP with Flink allows the former to take advantage of Flink's rich ecosystem (e.g. connectors) and its stream processing capabilities, such as support for event-time processing, exactly-once state semantics, fault-tolerance, savepoints and high throughput.

Published in: Data & Analytics
  • Be the first to comment

Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Flink: The state of FlinkCEP

  1. 1. 1 Kostas Kloudas @kkloudas Flink Forward Berlin SEPTEMBER 12, 2017 Complex Event Processing with Flink The state of FlinkCEP
  2. 2. 2 Original creators of Apache Flink® Providers of dA Platform 2, including open source Apache Flink + dA Application Manager
  3. 3. What is CEP? 3
  4. 4. CEP: Complex Event Processing  Detecting event patterns  Over continuous streams of events  Often arriving out-of-order 4
  5. 5. CEP: Complex Event Processing 5 Input
  6. 6. CEP: Complex Event Processing 6 Pattern Input
  7. 7. CEP: Complex Event Processing 7 Pattern Output Input
  8. 8. CEP: use-cases  IoT  Infrastructure Monitoring and Alarms  Intrusion detection  Inventory Management  Click Stream Analysis  Trend detection in financial sector  ...yours? 8
  9. 9. What is Stream Processing? 9
  10. 10. Stream Processing 10 Computation Computations on never-ending “streams” of events
  11. 11. Distributed Stream Processing 11 Computation Computation spread across many machines Computation Computation
  12. 12. Stateful Stream Processing 12 Computation State Result depends on history of stream
  13. 13. 13 Stream Processors are a natural fit for CEP
  14. 14. FlinkCEP 14 Pattern Output FlinkCEP Input
  15. 15. What does FlinkCEP offer? 15
  16. 16. Pattern Definition 16 Pattern
  17. 17. Pattern Definition  Composed of Individual Patterns • P1(shape == rectangle) • P2(shape == triangle) 17 Pattern P2 P1
  18. 18. Pattern Definition  Composed of Individual Patterns • P1(shape == rectangle) • P2(shape == triangle)  Combined by Contiguity Conditions • ...later 18 Pattern P2 P1
  19. 19. FlinkCEP Individual Patterns  Unique Name  Quantifiers : how many times ? • Looping oneOrMore(), times(from, to), greedy() • Optional optional()  Condition : which elements to accept ? • Simple e.g shape == rectangle • Iterative e.g rectangle.surface < triangle.surface • Stop until(cond.) 19 Pattern P2 P1
  20. 20. FlinkCEP Complex Patterns  Combine Individual Patterns  Contiguity Conditions • how to select relevant events given an input mixing relevant and irrelevant events  Time Constraints (event/processing time) • within(time) e.g. all events have to come within 24h 20 Pattern P2 P1
  21. 21. FlinkCEP Contiguity Conditions 21 Pattern Input
  22. 22. FlinkCEP Contiguity Conditions 22 Pattern OutputInput Strict Contiguity • matching events strictly follow each other
  23. 23. FlinkCEP Contiguity Conditions 23 Pattern OutputInput
  24. 24. FlinkCEP Contiguity Conditions 24 Pattern Relaxed Contiguity • non-matching events to simply be ignored Input Output
  25. 25. FlinkCEP Contiguity Conditions 25 Pattern Input Output
  26. 26. FlinkCEP Contiguity Conditions 26 Pattern Input Output
  27. 27. FlinkCEP Contiguity Conditions 27 Pattern Input Output Non-Deterministic Relaxed Contiguity • allows non-deterministic actions on relevant events
  28. 28. FlinkCEP Contiguity Conditions 28 Pattern NOT patterns: • for strict and relaxed contiguity • for cases where an event should invalidate a match Input
  29. 29. FlinkCEP Grouping Patterns 29 Pattern P2 P1  Define Individual Patterns  Combine them into Complex Patterns  Can we combine ...Complex Patterns ?
  30. 30. FlinkCEP Grouping Patterns 30 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
  31. 31. FlinkCEP Grouping Patterns 31 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
  32. 32. FlinkCEP Grouping Patterns 32 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
  33. 33. FlinkCEP Grouping Patterns 33 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
  34. 34. FlinkCEP Summary 34  Quantifiers oneOrMore(), times(), optional()  Conditions Simple, Iterative, Stop  Time Constraints Event and Processing time  Contiguity Constraints Strict, relaxed, non-deterministic relaxed, NOT  Grouping Patterns
  35. 35.  Flink already supports SQL: • match_recognize clause in SQL:2016 • ongoing effort with a lot of interest from the community 35 FlinkCEP Integration with SQL
  36. 36. Example 36
  37. 37.  Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h 37 Running Example: retailer A B M1 M2 M3 M4 M5
  38. 38.  Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h 38 Observation A Individual Patterns Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  39. 39. 39 Observation B Quantifiers  Start/End: single event  Middle: multiple events • .oneOrMore() Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  40. 40. 40 Observation C Conditions  Start -> Simple • properties of the event  Middle/End -> Iterative • Depend on previous events Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  41. 41. 41  Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h Observation D Time Constraints Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
  42. 42. 42  We opt for relaxed continuity Observation E Contiguity
  43. 43. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Individual Patterns
  44. 44. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Quantifiers
  45. 45. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Conditions
  46. 46. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Time Constraint
  47. 47. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
  48. 48. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
  49. 49. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
  50. 50. Documentation  FlinkCEP documentation: FlinkCEP 1.3: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html FlinkCEP 1.4: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html 50
  51. 51. 5 Thank you! @kkloudas @ApacheFlink @dataArtisans
  52. 52. We are hiring! data-artisans.com/careers

×