Complex event processing (CEP) is about identifying business opportunities and threats in real time by detecting patterns in data and taking appropriate automated action. Example business use cases for CEP include location-based marketing, smart inventories, targeted ads, Wi-Fi offloading, fraud detection, churn prediction, fleet management, predictive maintenance, security incident event management, and many more. While Spark Streaming provides a distributed resilient framework for ingesting events in real time, effort is still needed to build CEP applications. This is because CEP use cases require correlation of events, which in turn requires us to treat every incoming event as a discrete occurrence in time. Spark Streaming treats the entire batch of events as single occurrence. Many CEP use cases also require alerts to be fired even when there is no incoming event. An example of such use case is to fire an alert when an order-shipped event is NOT received within the SLA times following an order-received event. At Oracle we have adopted a few neat techniques like running continuous query engines as long running tasks, using empty batches as triggers, etc. to bring complex event processing to Spark Streaming. Join us to learn more on CEP for Spark, the fastest growing data processing platform in the world. Speakers Prabhu Thukkaram, Senior Director, Product Development, Oracle Hoyong Park, Architect, Oracle

1. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 1
Oracle Stream Analytics
Complex Event Processing for Apache Spark Streaming
Complex Event Processing for Spark Streaming
Prabhu Thukkaram, Senior Director, Oracle Product Development
Hoyong Park, Architect, Oracle Product Development

2. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 2
Complex Event Processing with Continuous Query Processor
Continuous
Query Processor
Pre-registered Queries
E3@T3, E2@T2, E1@T1
ResultsInput Events
R3@T3, R2@T2, R1@T1
Heartbeats

3. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Why Continuous Query Processor ?
Oracle Confidential – Internal/Restricted/Highly Restricted 3
• Complex event processing requires events to be processed one at a time
– Each event must be processed as identified by its individual timestamp
– Real world events originate at different times and must be processed as such
– CEP applications seek correlation and patterns across events in time within or across
batches and irrespective of batch boundaries
• Window length can span fractional batches
• Micro-batching with Spark Streaming
– All events in the batch are identified by same time (RDD Time)
– No progression of time between events in the same batch
– No progression of time when RDD partitions are empty
• Critical for missing event scenarios. E.g. alert when order status “Received” is not followed by
“Shipped” for order Id 10001 within 1 hour

4. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Pattern Detection in CQP
Oracle Confidential – Internal/Restricted/Highly Restricted 4
Checks if temperature readings from a power sensor are wobbling
during a certain time interval.
The CQP code checks for a W-Pattern in temperature readings during a
10 minute interval and selects support levels as output
SELECT LAST(A.value), LAST(C.value) FROM TEMP_STREAM
MATCH_RECOGNIZE (
PARTITION BY DEVICE_ID
PATTERN (A+ B+ C+ D+) DURATION OF 10 MINUTES
DEFINE
A AS (value < PREV(value))
B AS (value > PREV(value))
C AS (value < PREV(value))
D AS (value > PREV(value))
)
A
B
C D
10 Minutes

5. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Distributed Complex Event Processing
Oracle Confidential – Internal/Restricted/Highly Restricted 5
• Continuous Query Processor
– Event by event processing
• Each event is assigned a unique timestamp
• Apache Spark
– Distributed computing with scale out and fault tolerance
Spark Streaming + Continuous Query Processor
=
Distributed, Scalable, and Fault Tolerant CEP Platform

6. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
How CQP complements Spark Streaming ?
Oracle Confidential – Internal/Restricted/Highly Restricted 6
• Continuous, event-by-event, and stateful processing
• Flexible temporal windows (Time & Rows)
• Automatic progression of time
– Heartbeat propagation to advance time
• Pattern detection without batch boundaries
– Built-in finite state automaton
• Declarative SQL like language

7. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
How Spark complements CQP ?
• Distributed computing framework for CQP
• OOTB sources for data Ingestion
• Horizontal scale-out
• Fault tolerance and high availability
• Spark can detect failures, restart required resources, and automatically replay parts
of the stream that experienced failure
Oracle Confidential – Internal

8. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
High-Level Architecture
Data Stream Oracle’s CQP Engine
Finite State Automaton for
Pattern Detection across
Discrete Events
HDFS
Journaled CQP engine State serialized to HDFS
after computing each partition
CQP Engine State restored on
Executor Restart and Recompute of a
partition
Geo Sensing Cartridge for
Spatial Analytics
RETE Rules for Conditional
Logic
Complex Pattern Detection, Temporal Queries, Spatial
Queries, Contextual Queries, and Conditional Logic are all
executed in Oracle’s CEP Engine
Distributed Cache for
Ultra-fast Context Lookup

9. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
CQP Spark Application
Oracle Confidential – Internal
Cluster
Spark Standalone or YARN or Mesos
Spark Driver
Spark Executor
Spark Executor
Spark Executor
• Transformation delegated to CQP
• CQP application parsed in Spark’s Driver process
• Driver builds a DAG (job definition) from CQP application
• Driver runs job for each micro-batch
CQP
CQP
CQP

10. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Spark Integration - High Level
• Extensions
– CQLDStream extends DStream overriding compute()
– CQLRDD extends RDD overriding compute()
• Initialization/DAG creation phase
– Create Spark Context, Streaming Context, and start CQP on each Executor
– Setup DAG of CQLDStream
• Physical plan execution phase
– Assign unique timestamp to each event in micro-batch based on RDD time
– Register query with CQP on RDD compute if not already registered
– Send micro-batch to CQP and accept returned results
– Checkpoint CQP state if HA enabled
Oracle Confidential – Internal

11. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Spark Integration - High Availability
• Executor Failure
– Restart CQP on newly restarted Executor
– On RDD compute, re-register query and restore query state from snapshots
– Continue processing stream
Oracle Confidential – Internal

12. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal
Demo

13. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal
Q & A

14. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
CQP scalability in Spark
Oracle Confidential – Internal/Restricted/Highly Restricted 14
0
20000
40000
60000
80000
w1(c3) w2(c5) w3(c7) w4(c9)
Processing Time (seconds) for 40 Million
Records
0
5000
10000
15000
20000
w1(c3) w2(c5) w3(c7) w4(c9)
Avg. Processing Time Per Batch
(milliseconds)
0
20
40
60
80
100
120
w1(c3) w2(c5) w3(c7) w4(c9)
Number of Batches
Processed over 10
Minutes
Legend
Wn = n number of workers or
executors
W2(c5) means 2 executors
and a total of 5 cores across
both executors