The document discusses Complex Event Processing (CEP) with Apache Flink, outlining its capabilities in detecting event patterns over continuous streams of events. It presents various use cases for CEP, such as IoT and click stream analysis, and describes how to define and combine complex patterns using quantifiers, conditions, and contiguity constraints. Additionally, it highlights Flink's integration with SQL for flexible event tracing and provides practical examples.

1. 1
Kostas Kloudas
@kkloudas
Flink Forward Berlin
SEPTEMBER 12, 2017
Complex Event Processing with
Flink The state of FlinkCEP

2. 2
Original creators of Apache
Flink®
Providers of
dA Platform 2, including
open source Apache Flink +
dA Application Manager

3. What is CEP?
3

4. CEP: Complex Event Processing
 Detecting event patterns
 Over continuous streams of events
 Often arriving out-of-order
4

5. CEP: Complex Event Processing
5
Input

6. CEP: Complex Event Processing
6
Pattern
Input

7. CEP: Complex Event Processing
7
Pattern
Output
Input

8. CEP: use-cases
 IoT
 Infrastructure Monitoring and Alarms
 Intrusion detection
 Inventory Management
 Click Stream Analysis
 Trend detection in financial sector
 ...yours?
8

9. What is Stream Processing?
9

10. Stream Processing
10
Computation
Computations on
never-ending
“streams” of events

11. Distributed Stream Processing
11
Computation
Computation
spread across
many machines
Computation Computation

12. Stateful Stream Processing
12
Computation
State
Result depends
on history of
stream

13. 13
Stream Processors are a natural fit
for CEP

14. FlinkCEP
14
Pattern
Output
FlinkCEP
Input

15. What does FlinkCEP offer?
15

16. Pattern Definition
16
Pattern

17. Pattern Definition
 Composed of Individual Patterns
• P1(shape == rectangle)
• P2(shape == triangle)
17
Pattern
P2
P1

18. Pattern Definition
 Composed of Individual Patterns
• P1(shape == rectangle)
• P2(shape == triangle)
 Combined by Contiguity Conditions
• ...later
18
Pattern
P2
P1

19. FlinkCEP Individual Patterns
 Unique Name
 Quantifiers : how many times ?
• Looping oneOrMore(), times(from, to), greedy()
• Optional optional()
 Condition : which elements to accept ?
• Simple e.g shape == rectangle
• Iterative e.g rectangle.surface < triangle.surface
• Stop until(cond.)
19
Pattern
P2
P1

20. FlinkCEP Complex Patterns
 Combine Individual Patterns
 Contiguity Conditions
• how to select relevant events given an input mixing
relevant and irrelevant events
 Time Constraints (event/processing time)
• within(time) e.g. all events have to come within 24h
20
Pattern
P2
P1

21. FlinkCEP Contiguity Conditions
21
Pattern
Input

22. FlinkCEP Contiguity Conditions
22
Pattern
OutputInput
Strict Contiguity
• matching events strictly follow each other

23. FlinkCEP Contiguity Conditions
23
Pattern
OutputInput

24. FlinkCEP Contiguity Conditions
24
Pattern
Relaxed Contiguity
• non-matching events to simply be ignored
Input Output

25. FlinkCEP Contiguity Conditions
25
Pattern
Input Output

26. FlinkCEP Contiguity Conditions
26
Pattern
Input Output

27. FlinkCEP Contiguity Conditions
27
Pattern
Input Output
Non-Deterministic Relaxed Contiguity
• allows non-deterministic actions on relevant events

28. FlinkCEP Contiguity Conditions
28
Pattern
NOT patterns:
• for strict and relaxed contiguity
• for cases where an event should invalidate a match
Input

29. FlinkCEP Grouping Patterns
29
Pattern
P2
P1
 Define Individual Patterns
 Combine them into Complex Patterns
 Can we combine ...Complex Patterns ?

30. FlinkCEP Grouping Patterns
30
Grouping Patterns are for CEP what
parenthesis are for mathematical expressions.

31. FlinkCEP Grouping Patterns
31
Grouping Patterns are for CEP what
parenthesis are for mathematical expressions.

32. FlinkCEP Grouping Patterns
32
Grouping Patterns are for CEP what
parenthesis are for mathematical expressions.

33. FlinkCEP Grouping Patterns
33
Grouping Patterns are for CEP what
parenthesis are for mathematical expressions.

34. FlinkCEP Summary
34
 Quantifiers oneOrMore(), times(), optional()
 Conditions Simple, Iterative, Stop
 Time Constraints Event and Processing time
 Contiguity Constraints
Strict, relaxed, non-deterministic relaxed, NOT
 Grouping Patterns

35.  Flink already supports SQL:
• match_recognize clause in SQL:2016
• ongoing effort with a lot of interest from the
community
35
FlinkCEP Integration with SQL

36. Example
36

37.  Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
37
Running Example: retailer
A
B
M1
M2
M3
M4
M5

38.  Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
38
Observation A Individual Patterns
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

39. 39
Observation B Quantifiers
 Start/End: single event
 Middle: multiple events
• .oneOrMore()
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

40. 40
Observation C Conditions
 Start -> Simple
• properties of the event
 Middle/End -> Iterative
• Depend on previous events
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

41. 41
 Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
Observation D Time Constraints
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

42. 42
 We opt for relaxed continuity
Observation E Contiguity

43. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Individual Patterns

44. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Quantifiers

45. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Conditions

46. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Time Constraint

47. Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select (
new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) {
return parseMatch(pattern);
}
);

48. Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select (
new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) {
return parseMatch(pattern);
}
);

49. Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select (
new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) {
return parseMatch(pattern);
}
);

50. Documentation
 FlinkCEP documentation:
FlinkCEP 1.3: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
FlinkCEP 1.4: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
50

51. 5
Thank you!
@kkloudas
@ApacheFlink
@dataArtisans

52. We are hiring!
data-artisans.com/careers