Overview of Flink CEP library based on insurance selling use-case. Presentation for Warsaw Hadoop User Group meetup on 03.07.2017

1. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Looking for Patterns -
FlinkCEP Library
_D. Wysakowicz
WHUG
03.07.2017

2. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
About me
■ Data Engineer at GetInData
■ Apache Flink Committer
■ Involved in Flink CEP library development
Dawid Wysakowicz
@OneMoreCoder

3. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
That moment
when party
goes wrong.

4. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Were there
any signs?
Any Patterns?
That moment
when party
goes wrong.

5. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Image source: “Continuous Analytics: Stream Query Processing in Practice”, Michael J Franklin, Professor, UC Berkley, Dec 2009 and
http://www.slideshare.net/JoshBaer/shortening-the-feedback-loop-big-data-spain-external

6. © Copyright. All rights reserved. Not to be reproduced without prior written consent.

7. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Buying Home Insurance Online
Property
Info
Offer
Accept
General Terms
of Contract
PaymentScope
Change
Info
Change

8. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Buying Home Insurance Online
Property
Info
Offer
Accept
Exit
Scope
Change
Info
Change
General Terms
of Contract
Payment

9. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Simple Example - Matching Payments
Notify whenever a payment for accepted offer arrives

10. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Create Pattern
/* create pattern sequence*/
Pattern<Event, ?> pattern = Pattern<Event>
.begin("offer accepted")
.subtype(OfferAccepted.class)
.followedBy("payment received")
.subtype(PaymentReceived.class)
.within(Time.hours(48));

11. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Apply Pattern
/* create pattern sequence*/
Pattern<Event, ?> pattern = Pattern<Event>
.begin("offer accepted")
.subtype(OfferAccepted.class)
.followedBy("payment received")
.subtype(PaymentReceived.class)
.within(Time.hours(48));
/* convert match sequence into alerts */
DataStream<Alert> alerts = CEP.pattern(stream, pattern)

12. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Select Matches
/* create pattern sequence*/
Pattern<Event, ?> pattern = Pattern<Event>
.begin("offer accepted")
.subtype(OfferAccepted.class)
.followedBy("payment received")
.subtype(PaymentReceived.class)
.within(Time.hours(48));
/* convert match sequence into alerts */
DataStream<Alert> alerts = CEP.pattern(stream, pattern)
.select(new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> match) throws Exception {
return createAlert(match);
}
});

13. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Handle Results
/* create pattern sequence*/
Pattern<Event, ?> pattern = Pattern<Event>
.begin("offer accepted")
.subtype(OfferAccepted.class)
.followedBy("payment received")
.subtype(PaymentReceived.class)
.within(Time.hours(48));
/* convert match sequence into alerts */
DataStream<Alert> alerts = CEP.pattern(stream, pattern)
.select(new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> match) throws Exception {
return createAlert(match);
}
});
/* write out results */
alerts.addSink(...);

14. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Seamless Integration - Standard Pipeline
DataStream
Source e.g.
DataStream
Sink e.g.
DataStream
Transformations

15. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Seamless Integration - CEP Library
DataStream
Source e.g.
DataStream
Sink e.g.
DataStream
Transformations
CEP
DataStream Pattern
DataStream
Stream of matches

16. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
CEP Resulting Matches
CEP
DataStream
Pattern
A -> B -> C
C2C1B2B1A
DataStream
Stream of matches
A, B1, C1 A, B2, C1 A, B1, C2 A, B2, C2

17. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
CEP Timeouted Matches
CEP
DataStream
Pattern
A -> B -> C within 5 seconds
C2(t=7)B2(t=4)B1(t=2)A(t = 1)
DataStream
Stream of timeouted matches
W=6
?

18. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
CEP Timeouted Matches
CEP
DataStream
Pattern
A -> B -> C within 5 seconds
C2(t=7)B2(t=4)B1(t=2)A(t = 1)
DataStream
Stream of timeouted matches
A A, B1 A, B2
W=6

19. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Pattern Sequence
Pattern
.begin("start")
.subtype(Event.class)
.where(...)
.or(...)
.followedBy("next")
.subtype(Event.class)
.where(...)
.where(...)

20. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Patterns
Pattern
.begin("start")
.subtype(Event.class)
.where(...)
.or(...)
.followedBy("next")
.subtype(Event.class)
.where(...)
.where(...)
Basic building blocks

21. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Conditions
Pattern
.begin("start")
.subtype(Event.class)
.where(...)
.or(...)
.followedBy("next")
.subtype(Event.class)
.where(...)
.where(...)
Basic building blocks
Condition specification

22. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Time Restrictions
Pattern
.begin("start")
.subtype(Event.class)
.where(...)
.or(...)
.followedBy("next")
.subtype(Event.class)
.where(...)
.where(...)
.within(Time.seconds(3))
Basic building blocks
Condition specification
Time restrictions

23. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Time Restrictions
Pattern
.begin("start")
.subtype(Event.class)
.where(...)
.or(...)
.followedBy("next")
.subtype(Event.class)
.where(...)
.where(...)
.within(Time.seconds(3))
Basic building blocks
Condition specification
Time restrictions
NOTE: The time restriction is a
global property. This is a period in
which either whole match is
generated or partial matches are
timeouted.

24. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
Consuming Strategy

25. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
■ Strict continuity
Consuming Strategy

26. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
■ Strict continuity
■ Skip till any
Consuming Strategy

27. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
■ Strict continuity
■ Skip till any
■ Not follow
Consuming Strategy

28. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
■ Strict continuity
■ Skip till any
Consuming Strategy
■ Not follow

29. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
■ Strict continuity
■ Skip till any
■ Not follow
■ Not next
Consuming Strategy

30. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Skip till next
■ Strict continuity
■ Skip till any
Consuming Strategy
■ Not follow
■ Not next

31. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Consuming Strategy
■ Skip till next
■ Strict continuity
■ Skip till any
■ Not follow
■ Not next

32. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Looping Example
How many offer changes before accepting?
...
?

33. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Pattern Specification
Pattern
.begin("start")
.subtype(Event.class)
.where(...)
.or(...)
.times(3)
.followedBy("next")
.subtype(Event.class)
.where(...)
.where(...)
.oneOrMore().optional()
.within(Time.seconds(3))
Quantifier application
Basic building blocks
Condition specification
Time restrictions

34. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Quantifiers
■ Singleton pattern
■ Complex pattern
● Times (count)
● Looping pattern

35. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Quantifiers
■ Singleton pattern
■ Complex pattern
● Times (count)
● Looping pattern
■ Optional pattern
● singleton -> optional
● times -> no or exactly n times
● oneOrMore -> zeroOrMore

36. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Complex Patterns - Consuming Strategies
?
Pattern
.begin("offer").subtype(Offer.class)

37. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
?
Pattern
.begin("offer").subtype(Offer.class)
.followedByAny("changes").subtype(OfferChanged.class)
Complex Patterns - Consuming Strategies

38. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
?
Pattern
.begin("offer").subtype(Offer.class)
.followedByAny("changes").subtype(OfferChanged.class)
.oneOrMore()
.consecutive()
Complex Patterns - Consuming Strategies

39. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
?
Pattern
.begin("offer").subtype(Offer.class)
.followedByAny("changes").subtype(OfferChanged.class)
.oneOrMore()
.consecutive()
.followedBy(“accepted”).subtype(OfferAccepted.class)
Complex Patterns - Consuming Strategies

40. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
■ Consuming strategy - before first element
■ Inner consuming strategy - between events matched in the
Pattern
Complex Patterns - Consuming Strategies

41. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Iterative Condition
Trigger alert when the same property changed.
PL Flood PL

42. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Iterative Condition
public abstract class IterativeCondition<T> implements Function, Serializable {
public abstract boolean filter(T value, Context<T> ctx) throws Exception;
/**
* @return An {@link Iterable} over the already accepted elements
* for a given pattern. Elements are iterated in the order they were
* inserted in the pattern.
*
* @param name The name of the pattern.
*/
public interface Context<T> extends Serializable {
Iterable<T> getEventsForPattern(String name);
}
}

43. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Iterative Condition
Trigger alert when the same property changed.
Pattern
.begin("first change").subtype(OfferChanged.class)
.followedBy("second change").subtype(OfferChanged.class)
.where(new IterativeCondition<OfferChanged>() {
@Override
public boolean filter(OfferChanged value, Context<OfferChanged> ctx) {
return ctx.getEventsForPattern("first change").iterator().next()
.property().equals(value.property());
}
});
PL Flood PL

44. © Copyright. All rights reserved. Not to be reproduced without prior written consent.

45. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Complex Example
Alert when a user constantly decreases value of the
property
-12.000€
-14.000€
-24.000€
-100.000€

46. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Complex Example - Implementation
Alert when a user changed value by more than 200% of
previous average change
Pattern
.begin("change").subtype(ValueChanged.class)
.oneOrMore()
.followedBy("alerting change").subtype(ValueChanged.class)
.where(new IterativeCondition<OfferChanged>() {
@Override
public boolean filter(OfferChanged value, Context<OfferChanged> ctx) {
double averageChange = countAverage(ctx.getEventsForPattern("change"));
return value.change > 2 * averageChange;
}
});

47. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
FlinkCEP & Asian Telco
■ +15 subscriber-oriented triggers to implement
● e.g. when a subscriber registers in a roaming and then
spends X USD in a roaming

48. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Summary
■ Shortest possible feedback loop
■ Seamless integration with Flink ecosystem
■ Complex patterns
● Dynamic length
● Conditions based on calculations

49. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
How does it works underneath?

50. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventStream
Order handling
1 5 4 2
PatternOperator
ElementsQueue
1 2 4 5
NFA

51. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventStream
Order handling
1 5 4 82
W=5
PatternOperator
ElementsQueue
1 2 4 5
W=5
NFA

52. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventStream
Order handling
1 5 4 82
W=5
PatternOperator
ElementsQueue
1 2 4 5
W=5
NFA
MatchesStream
A B

53. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventStream
Order handling - late elements
1 5 4 8 32
W=5
PatternOperator
ElementsQueue
1 2 4 5
W=5
NFA
MatchesStream
A B

54. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventStream
Order handling - late elements
1 5 4 8 32
W=5
PatternOperator
ElementsQueue
1 2 4 5
W=5
NFA
MatchesStream
A B

55. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Nondeterministic Finite Automaton
■ Graph where:
● Vertices = States
● Edges = Transitions
■ IGNORE - event not consumed
■ TAKE - event consumed
■ PROCEED - event analyzed in the target state

56. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Example of NFA
Pattern
.begin(“A”)
.followedByAny(“B”).optional()
.next(“C”)
.followedBy(“D”)

57. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE
Pattern
.begin(“A”)
.followedByAny(“B”).optional()
.next(“C”)
.followedBy(“D”)

58. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE

59. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE
A

60. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A
A
B?
A

61. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
A B
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE

62. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
A B
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE

63. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE
A B

64. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A
A
B?
A
B
A
C
B?

65. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE
A B C

66. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A B? C D
TAKE TAKE
TAKE
PROCEED
IGNORE IGNORE IGNORE
TAKE
A B C

67. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A
A
B?
A
B
A
C
B?
D
B?
D
A
C

68. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
EventsStream
Example of NFA
A
A
B?
A
B
A
C
B?
D
B?
D
A
C D
Match: A, C, D
B?
Match: A, B, C, D
A

69. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

70. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

71. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

72. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

73. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

74. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

75. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

76. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

77. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

78. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

79. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

80. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

81. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

82. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Shared Buffer - how it works
MiddleStart End
S1
S2
M1
M2
M3
E1
E2
1
2
1.0
1.0.0
1.0.0.0
2.0
1.0.0.0.0
1.0.0.0.1
2.0.0
Events order
S1 M1 M2 S2 M3 E1 E2

83. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Summary
■ Shortest possible feedback loop
■ Seamless integration with Flink ecosystem
■ Complex patterns
● Dynamic length
● Conditions based on calculations