Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- Márton Balassi Streaming ML with F... by Flink Forward 1526 views
- Kostas Kloudas - Complex Event Proc... by dataArtisans 2350 views
- Machine Learning with Apache Flink ... by Till Rohrmann 8547 views
- Data Intensive Applications with Ap... by Simone Robutti 306 views
- Flink Forward SF 2017: Eron Wright ... by Flink Forward 1374 views
- Flink Forward SF 2017: Ted Dunning ... by Flink Forward 556 views

590 views

Published on

Published in:
Data & Analytics

No Downloads

Total views

590

On SlideShare

0

From Embeds

0

Number of Embeds

0

Shares

0

Downloads

0

Comments

0

Likes

5

No embeds

No notes for slide

My name is Kostas Kloudas and I am here to talk to you about FlinkCEP, a library for complex event processing built atop Apache Flink.

This was the main motivation behind the first implementation of FlinkCEP, more than a year ago, and

this talk focuses on what the current capabilities of FlinkCEP, (slide)

Given that complex patterns are composed of individual patterns, we start by describing them first, before showing how to combine them together.

Individual Patterns must have a unique name and for each one of them we can define a condition based on which it accepts relevant events.

This condition can depend on properties of the event itself, in which case it is a SIMPLE condition, or on properties or statistics over a subset of previously accepted events, in which case it is an ITERATIVE Condition.

In addition to the condition, a pattern can also have quantifiers. By default, when an individual pattern appears in a complex pattern, FlinkCEP expects the described type of event to appear exactly once, in order to have a match. This is a singleton pattern. In our case, we expect exactly one rectangle, followed by exactly one triangle. FlinkCEP also supports quantifiers. In 1.3, these are oneOrMore() for usecases where a specific type of event is expected “at-least once”, times() when we want it to appear a specified amount of times, and optional() if the event is optional.

The above are the possibilities offered when defining individual Patterns. These patterns can be combined into complex patterns (slide)

To understand contiguity, let’s take our pattern as shown on the left-hand side, and our previous input... (slide)

This is enough for many usecases, but in some cases you may need to COMPOSE multiple complex patterns into one or apply a quantifier on a full complex pattern.

This is where grouping patterns come into play! In this example, ...

And not only that...

This shows how grouping patterns allow you to COMPOSE complex patterns together.

In terms of code, essentially wherever before you could put individual patterns, now you can put pattern sequences, and the expression is still valid (except for greedy()).

Flink already supports SQL, and the 2016 SQL spec contains the MATCH_RECOGNIZE clause for pattern matching. The semantics of the clause are pretty similar to

our semantics, so the community is actively working to expose FlinkCEP through Flink’s SQL API.

- 1. 1 Kostas Kloudas @kkloudas Flink Forward Berlin SEPTEMBER 12, 2017 Complex Event Processing with Flink The state of FlinkCEP
- 2. 2 Original creators of Apache Flink® Providers of dA Platform 2, including open source Apache Flink + dA Application Manager
- 3. What is CEP? 3
- 4. CEP: Complex Event Processing Detecting event patterns Over continuous streams of events Often arriving out-of-order 4
- 5. CEP: Complex Event Processing 5 Input
- 6. CEP: Complex Event Processing 6 Pattern Input
- 7. CEP: Complex Event Processing 7 Pattern Output Input
- 8. CEP: use-cases IoT Infrastructure Monitoring and Alarms Intrusion detection Inventory Management Click Stream Analysis Trend detection in financial sector ...yours? 8
- 9. What is Stream Processing? 9
- 10. Stream Processing 10 Computation Computations on never-ending “streams” of events
- 11. Distributed Stream Processing 11 Computation Computation spread across many machines Computation Computation
- 12. Stateful Stream Processing 12 Computation State Result depends on history of stream
- 13. 13 Stream Processors are a natural fit for CEP
- 14. FlinkCEP 14 Pattern Output FlinkCEP Input
- 15. What does FlinkCEP offer? 15
- 16. Pattern Definition 16 Pattern
- 17. Pattern Definition Composed of Individual Patterns • P1(shape == rectangle) • P2(shape == triangle) 17 Pattern P2 P1
- 18. Pattern Definition Composed of Individual Patterns • P1(shape == rectangle) • P2(shape == triangle) Combined by Contiguity Conditions • ...later 18 Pattern P2 P1
- 19. FlinkCEP Individual Patterns Unique Name Quantifiers : how many times ? • Looping oneOrMore(), times(from, to), greedy() • Optional optional() Condition : which elements to accept ? • Simple e.g shape == rectangle • Iterative e.g rectangle.surface < triangle.surface • Stop until(cond.) 19 Pattern P2 P1
- 20. FlinkCEP Complex Patterns Combine Individual Patterns Contiguity Conditions • how to select relevant events given an input mixing relevant and irrelevant events Time Constraints (event/processing time) • within(time) e.g. all events have to come within 24h 20 Pattern P2 P1
- 21. FlinkCEP Contiguity Conditions 21 Pattern Input
- 22. FlinkCEP Contiguity Conditions 22 Pattern OutputInput Strict Contiguity • matching events strictly follow each other
- 23. FlinkCEP Contiguity Conditions 23 Pattern OutputInput
- 24. FlinkCEP Contiguity Conditions 24 Pattern Relaxed Contiguity • non-matching events to simply be ignored Input Output
- 25. FlinkCEP Contiguity Conditions 25 Pattern Input Output
- 26. FlinkCEP Contiguity Conditions 26 Pattern Input Output
- 27. FlinkCEP Contiguity Conditions 27 Pattern Input Output Non-Deterministic Relaxed Contiguity • allows non-deterministic actions on relevant events
- 28. FlinkCEP Contiguity Conditions 28 Pattern NOT patterns: • for strict and relaxed contiguity • for cases where an event should invalidate a match Input
- 29. FlinkCEP Grouping Patterns 29 Pattern P2 P1 Define Individual Patterns Combine them into Complex Patterns Can we combine ...Complex Patterns ?
- 30. FlinkCEP Grouping Patterns 30 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
- 31. FlinkCEP Grouping Patterns 31 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
- 32. FlinkCEP Grouping Patterns 32 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
- 33. FlinkCEP Grouping Patterns 33 Grouping Patterns are for CEP what parenthesis are for mathematical expressions.
- 34. FlinkCEP Summary 34 Quantifiers oneOrMore(), times(), optional() Conditions Simple, Iterative, Stop Time Constraints Event and Processing time Contiguity Constraints Strict, relaxed, non-deterministic relaxed, NOT Grouping Patterns
- 35. Flink already supports SQL: • match_recognize clause in SQL:2016 • ongoing effort with a lot of interest from the community 35 FlinkCEP Integration with SQL
- 36. Example 36
- 37. Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h 37 Running Example: retailer A B M1 M2 M3 M4 M5
- 38. Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h 38 Observation A Individual Patterns Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
- 39. 39 Observation B Quantifiers Start/End: single event Middle: multiple events • .oneOrMore() Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
- 40. 40 Observation C Conditions Start -> Simple • properties of the event Middle/End -> Iterative • Depend on previous events Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
- 41. 41 Trace all shipments which: • start at location A • have at least 5 stops • end at location B • within the last 24h Observation D Time Constraints Start End Mid ev.from == A ev[i].from == ev[i-1].to ev.to == B && size(“mid”) >= 5
- 42. 42 We opt for relaxed continuity Observation E Contiguity
- 43. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Individual Patterns
- 44. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Quantifiers
- 45. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Conditions
- 46. Pattern<Event, ?> pattern = Pattern .<Event>begin("start") .where(mySimpleCondition) .followedBy ("middle") .where(myIterativeCondition1) .oneOrMore() .followedBy ("end”) .where(myIterativeCondition2) .within(Time.hours(24)) Start Middle End Running Example Time Constraint
- 47. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
- 48. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
- 49. Running Example Pattern Integration Pattern<Event, ?> pattern = ... PatternStream<Event> patternStream = CEP.pattern(input, pattern); DataStream<Alert> result = patternStream.select ( new PatternSelectFunction<Event, Alert>() { @Override public Alert select(Map<String, List<Event>> pattern) { return parseMatch(pattern); } );
- 50. Documentation FlinkCEP documentation: FlinkCEP 1.3: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html FlinkCEP 1.4: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html 50
- 51. 5 Thank you! @kkloudas @ApacheFlink @dataArtisans
- 52. We are hiring! data-artisans.com/careers

No public clipboards found for this slide

Be the first to comment