For a long time, complex event processing (CEP) and stream analytics have been treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest, enrich, and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput usually due to distributed scale-out architectures. Apache Flink was one of the first open source stream processors that was able to address the full spectrum of stream processing applications, ranging from applications with low latency requirements to applications that process millions of events per second. On top of this powerful processing engine, the Flink community built APIs for complex event processing and streaming analytics, namely the CEP library and support for streaming SQL. Since recently, the Flink community is integrating both APIs by extending Flink SQL to support the MATCH RECOGNIZE clause for row pattern matching that was introduced with the SQL:2016 standard. In my talk, I will discuss the new MATCH RECOGNIZE feature and present use cases that benefit from pattern matching support in streaming SQL, such as process monitoring or anomaly detection. I will demonstrate the feature with a few example queries and take a brief look under hood to discuss a few details of the implementation.

1. Dawid Wysakowicz, SOFTWARE ENGINEER
Detecting Patterns in
Event Streams with Flink
SQL

2. © 2018 data Artisans2
ABOUT DATA ARTISANS
Original Creators of
Apache Flink®
Enterprise Ready
Real Time Stream Processing

3. © 2018 data Artisans3
FREE TRIAL DOWNLOAD
data-artisans.com/download

4. © 2018 data Artisans4
Why people like SQL?
• Well known interface: "lingua franca of data processing"
• No programming required - easier to learn
• Declarative way to express your query, WHAT not HOW
• Out-of-the box optimization
• Reusability
‒ built-in functions
‒ user defined functions

5. © 2018 data Artisans5
SQL on Streams
● Unified semantics for stream and batch input
○ Same query, same input, same result
Stream is converted
into dynamic table
Dynamic table is
converted into stream
Continuous query on
dynamic table produces
result d table

6. © 2018 data Artisans6
The New York Taxi Rides Data Set
• The New York City & Limousine Commission provides a public data set about
past taxi rides in New York City
• We can derive a streaming table from the data
• Table: TaxiRides
rideId: BIGINT // ID of the taxi ride
taxiId: BIGINT // ID of the taxi cab
driverId: BIGINT // ID of the drive
isStart: BOOLEAN // flag for pick-up (true) or drop-off
(false) event
lon: DOUBLE // longitude of pick-up or
drop-off location
lat: DOUBLE // latitude of pick-up or drop-
off location
rowTime: TIMESTAMP // time of pick-up or drop-off event

7. © 2018 data Artisans7
Number of rides per 30 minutes per area
Time
Number of rides

8. © 2018 data Artisans8
SQL Example
SELECT
toCellId(lat, lon) as cellId,
COUNT(distinct rideId) as rideCount,
TUMBLE_ROWTIME(rowTime, INTERVAL '30' minute) AS rowTime,
cast (TUMBLE_START(rowTime, INTERVAL '30' minute) as TIMESTAMP) AS startTime,
cast(TUMBLE_END(rowTime, INTERVAL '30' minute) as TIMESTAMP) AS endTime
FROM
TaxiRides
GROUP BY
toCellId(lat, lon),
TUMBLE(rowTime, INTERVAL '30' minute)

9. © 2018 data Artisans9
SQL Support in Flink 1.6
• SELECT FROM WHERE
• GROUP BY / HAVING
‒ Non-windowed, TUMBLE, HOP, SESSION windows
• JOIN
‒ Windowed INNER, LEFT / RIGHT / FULL OUTER JOIN
‒ Non-windowed INNER / LEFT / RIGHT / FULL OUTER JOIN
• Scalar, aggregation, table-valued UDFs
• Many built-in functions
• [streaming only] OVER / WINDOW
‒ UNBOUNDED / BOUNDED PRECEDING

10. © 2018 data Artisans10
SQL Support in Flink 1.6
• SELECT FROM WHERE
• GROUP BY / HAVING
‒ Non-windowed, TUMBLE, HOP, SESSION windows
• JOIN
‒ Windowed INNER, LEFT / RIGHT / FULL OUTER JOIN
‒ Non-windowed INNER / LEFT / RIGHT / FULL OUTER JOIN
• Scalar, aggregation, table-valued UDFs
• Many built-in functions
• [streaming only] OVER / WINDOW
‒ UNBOUNDED / BOUNDED PRECEDING
12:20-13:00 Kesselhaus
Timo Walther, data Artisans
Flink SQL in Action

11. © 2018 data Artisans11
• Find rides with mid-stops
What is hard with SQL?

12. © 2018 data Artisans12
Mid stops
Pattern.<Row>begin("S”).where((row) -> {
return row.isStart == true;
}).next("E").where( (row) -> {
return row.isStart == true;
});
CEP.pattern(input.keyBy(“driverId”), pattern)
.flatSelect(
new PatternFlatSelectFunction<Row, Row>() {
@Override
public void flatSelect(
Map<String, List<Row>> pattern,
Collector<Row> out) throws Exception {
out.collect((
pattern.get(“S”).get(0).getRideId
));
}
);

13. © 2018 data Artisans13
MATCH_RECOGNIZE
SQL:2016 extension

14. © 2018 data Artisans14
Common use-cases
• stock market analysis
• customer behaviour
• tracking money laundering
• service quality
• network intrusion detection

15. © 2018 data Artisans15
Syntax
tableReference:
tablePrimary
[ matchRecognize ]
[ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]
select:
SELECT [ STREAM ] [ ALL | DISTINCT ]
{ * | projectItem [, projectItem ]* }
FROM tableExpression
[matchRecognize]
[ WHERE booleanExpression ]
[ GROUP BY { groupItem [, groupItem ]* } ]
….

16. © 2018 data Artisans16
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rowTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S E)
DEFINE
S AS S.isStart = true,
E AS E.isStart = true
)
Rides with mid-stops
partition the data by
given field = keyBy

17. © 2018 data Artisans17
Rides with mid-stops
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rowTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S E)
DEFINE
S AS S.isStart = true,
E AS E.isStart = true
)
● specify order
● primary order = Event or
Processing time
1 3 2 5 4

18. © 2018 data Artisans18
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rowTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S E)
DEFINE
S AS S.isStart = true,
E AS E.isStart = true
)
Rides with mid-stops
construct pattern
S E

19. © 2018 data Artisans19
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rowTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S E)
DEFINE
S AS S.isStart = true,
E AS E.isStart = true
)
Rides with mid-stops
extract measures from
matched sequence
S E

20. © 2018 data Artisans20
Multi-Stop

21. © 2018 data Artisans21
Rides with more than one mid-stop
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rowTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S M{2,} E)
DEFINE
S AS S.isStart = true,
M AS M.rideId <> S.rideId,
E AS E.isStart = false
AND E.rideId = S.rideId
)
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rowTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S E)
DEFINE
S AS S.isStart = true,
E AS E.isStart = true
)

22. © 2018 data Artisans22
Rush (peak) hours - V shape
Time
Number of rides

23. © 2018 data Artisans23
Statistics per Area
CREATE VIEW RidesInArea AS
SELECT
toCellId(lat, lon) as cellId,
COUNT(distinct rideId) as rideCount,
TUMBLE_ROWTIME(rowTime, INTERVAL '30' minute) AS rowTime,
cast (TUMBLE_START(rowTime, INTERVAL '30' minute) as TIMESTAMP) AS startTime,
cast(TUMBLE_END(rowTime, INTERVAL '30' minute) as TIMESTAMP) AS endTime
FROM
TaxiRides
GROUP BY
toCellId(lat, lon),
TUMBLE(rowTime, INTERVAL '30' minute)

24. © 2018 data Artisans24
Rush (peak) hours
SELECT * FROM RidesInArea
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
FIRST(UP.rideCount) AS preCnt,
LAST(UP.rideCount) AS rushCnt, FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd,
LAST(DOWN.rideCount) AS postCnt
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE UP AS UP.rideCount > PREV(UP.rideCount) or PREV(UP.rideCount) IS
NULL,
DOWN AS DOWN.rideCount < PREV(DOWN.rideCount) AND
DOWN.rideCount < LAST(UP.rideCount),
E AS E.rideCount > LAST(DOWN.rideCount)
Time
Number of rides
Use previous table/view

25. © 2018 data Artisans25
Rush (peak) hours
SELECT * FROM RidesInArea
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
FIRST(UP.rideCount) AS preCnt,
LAST(UP.rideCount) AS rushCnt, FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd,
LAST(DOWN.rideCount) AS postCnt
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE UP AS UP.rideCount > PREV(UP.rideCount) or PREV(UP.rideCount) IS
NULL,
DOWN AS DOWN.rideCount < PREV(DOWN.rideCount) AND
DOWN.rideCount < LAST(UP.rideCount),
E AS E.rideCount > LAST(DOWN.rideCount)
Time
Number of rides
apply match to result of the
inner query
Use previous table/view

26. © 2018 data Artisans26
Rush (peak) hours
SELECT * FROM RidesInArea
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
FIRST(UP.rideCount) AS preCnt,
LAST(UP.rideCount) AS rushCnt, FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd,
LAST(DOWN.rideCount) AS postCnt
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE UP AS UP.rideCount > PREV(UP.rideCount) or PREV(UP.rideCount) IS
NULL,
DOWN AS DOWN.rideCount < PREV(DOWN.rideCount) AND
DOWN.rideCount < LAST(UP.rideCount),
E AS E.rideCount > LAST(DOWN.rideCount)
Time
Number of rides
access elements of
looping pattern

27. © 2018 data Artisans27
Rush (peak) hours
SELECT * FROM RidesInArea
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
FIRST(UP.rideCount) AS preCnt,
LAST(UP.rideCount) AS rushCnt, FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd,
LAST(DOWN.rideCount) AS postCnt
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE UP AS UP.rideCount > PREV(UP.rideCount) or PREV(UP.rideCount) IS
NULL,
DOWN AS DOWN.rideCount < PREV(DOWN.rideCount) AND
DOWN.rideCount < LAST(UP.rideCount),
E AS E.rideCount > LAST(DOWN.rideCount)
Time
Number of rides
access elements of
looping pattern

28. © 2018 data Artisans28
Feature set of MATCH_RECOGNIZE
(Experimental)
• Quantifiers support:
‒ + (one or more), * (zero or more), {x,y} (times)
‒ greedy(default), ?(reluctant)
• with some restrictions (not working for last pattern)
• After Match Skip
‒ skip_to_first/last, skip_past_last, skip_to_next
• Not supported:
‒ alter(|), permute, exclude ‘{- -}’
‒ aggregates within MATCH_RECOGNIZE

29. © 2018 data Artisans29
Summary
● opens for new use cases, that assume order of events
● enables reuse of SQL goods
● still in experimental phase
○ https://github.com/dawidwys/flink/SQL_MATCH_RECOGNIZE

30. THANK YOU!
@dwysakowicz
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers

31. © 2018 data Artisans31
Under the hood
Dissecting MATCH_RECOGNIZE

32. © 2018 data Artisans32
Pattern.<Row>begin("UP”, SKIP_PAST_LAST).timesOrMore(4).where((row, ctx) ->
{
Long prevRideCount = ctx.get(“UP”).tail().get(“rideCount”);
return prevRideCount == null || row.get(“rideCount”) >
prevRideCount;
}).next("DOWN").timesOrMore(2).where( (row, ctx) -> {
...
}).next("E").where( (row, ctx) -> {
...
});
CEP.pattern(input.keyBy(“cellId”), pattern).flatSelect(
new PatternFlatSelectFunction<Row, Row>() {
@Override
public void flatSelect(Map<String, List<Row>> pattern,
Collector<Row> out) throws Exception {
Row last = pattern.get(“UP”).tail();
Row first = pattern.get(“UP”).headl();
out.collect(Row.of(...));
}
);
Under the hood
SELECT * FROM TaxiStatistics
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
LAST(UP.rideCount) AS rushCnt,
FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE
UP AS UP.rideCount > PREV(UP.rideCount)
or PREV(UP.rideCount) IS NULL,
DOWN AS DOWN.rideCount <
LAST(UP.rideCount),
E AS E.rideCount >
LAST(DOWN.rideCount)
)

33. © 2018 data Artisans33
Under the hood
Pattern.<Row>begin("UP”, SKIP_PAST_LAST).timesOrMore(4).where((row, ctx) ->
{
Long prevRideCount = ctx.get(“UP”).tail().get(“rideCount”);
return prevRideCount == null || row.get(“rideCount”) >
prevRideCount;
}).next("DOWN").timesOrMore(2).where( (row, ctx) -> {
...
}).next(“E").where( (row, ctx) -> {
...
});
CEP.pattern(input.keyBy(“cellId”), pattern).flatSelect(
new PatternFlatSelectFunction<Row, Row>() {
@Override
public void flatSelect(Map<String, List<Row>> pattern,
Collector<Row> out) throws Exception {
Row last = pattern.get(“UP”).tail();
Row first = pattern.get(“UP”).headl();
out.collect(Row.of(...));
}
);
SELECT * FROM TaxiStatistics
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
LAST(UP.rideCount) AS rushCnt,
FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE
UP AS UP.rideCount > PREV(UP.rideCount)
or PREV(UP.rideCount) IS NULL,
DOWN AS DOWN.rideCount <
LAST(UP.rideCount),
E AS E.rideCount >
LAST(DOWN.rideCount)
)

34. © 2018 data Artisans34
Pattern.<Row>begin("UP”, SKIP_PAST_LAST).timesOrMore(4).where((row, ctx) ->
{
Long prevRideCount = ctx.get(“UP”).tail().get(“rideCount”);
return prevRideCount == null || row.get(“rideCount”) >
prevRideCount;
}).next("DOWN").timesOrMore(2).where( (row, ctx) -> {
...
}).next("E").where( (row, ctx) -> {
...
});
CEP.pattern(input.keyBy(“cellId”), pattern).flatSelect(
new PatternFlatSelectFunction<Row, Row>() {
@Override
public void flatSelect(Map<String, List<Row>> pattern,
Collector<Row> out) throws Exception {
Row last = pattern.get(“UP”).tail();
Row first = pattern.get(“UP”).headl();
out.collect(Row.of(...));
}
);
SELECT * FROM TaxiStatistics
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
LAST(UP.rideCount) AS rushCnt,
FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE
UP AS UP.rideCount > PREV(UP.rideCount)
or PREV(UP.rideCount) IS NULL,
DOWN AS DOWN.rideCount <
LAST(UP.rideCount),
E AS E.rideCount >
LAST(DOWN.rideCount)
)
Under the hood

35. © 2018 data Artisans35
Under the hood
Pattern.<Row>begin("UP”, SKIP_PAST_LAST).timesOrMore(4).where((row, ctx) ->
{
Long prevRideCount = ctx.get(“UP”).tail().get(“rideCount”);
return prevRideCount == null || row.get(“rideCount”) >
prevRideCount;
}).next("DOWN").timesOrMore(2).where( (row, ctx) -> {
...
}).next("E").where( (row, ctx) -> {
...
});
CEP.pattern(input.keyBy(“cellId”), pattern).flatSelect(
new PatternFlatSelectFunction<Row, Row>() {
@Override
public void flatSelect(Map<String, List<Row>> pattern,
Collector<Row> out) throws Exception {
Row last = pattern.get(“UP”).tail();
Row first = pattern.get(“UP”).headl();
out.collect(Row.of(...));
}
);
SELECT * FROM TaxiStatistics
MATCH_RECOGNIZE(
PARTITION BY cellId
MEASURES
LAST(UP.rideCount) AS rushCnt,
FIRST(UP.startTime) as rushStart,
LAST(UP.endTime) as rushEnd
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE
UP AS UP.rideCount > PREV(UP.rideCount)
or PREV(UP.rideCount) IS NULL,
DOWN AS DOWN.rideCount <
LAST(UP.rideCount),
E AS E.rideCount >
LAST(DOWN.rideCount)
)

36. THANK YOU!
@dwysakowicz
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers