Presentation from reactconf 2014 in San Francisco.
Covers Event Stream Processing, some of the theory behind it and some implementation details in the context of local and distributed. Also covers some Big Data technologies
1. {MAKING SENSE OF THE FIRE-HOSE IN REAL-TIME}
{EVENT PROCESSING}
Dr Andy Piper
Push Technology
Reappt, a Push Technology product offers the enterprise grade Diffusion technology as a
service.
2. Time @cobbscomedyclub
The past, the present and the future walked
Copyright Push Technology 2014
into a bar
It was tense
3. Will the real Andy Piper please stand up?
– CTO at Push Technology
– Ex-BEA/Oracle
– Spring contributor (Spring DM) and Author
– Standards contributor – OMG, JCP etc
– PhD, Cambridge, Distributed Systems
✗ ✗
Copyright Push Technology 2014
4. Agenda in 140 characters
What is it - What not? Why? History. Measure
infinity. Windows. Queries. Going fast –
reliably, distributed, distributed and fast and big
Copyright Push Technology 2014
5. What is Event Stream Processing?
Copyright Push Technology 2014
6. What is Event Stream Processing?
• It’s not stream processing
– Typically focused on local parallelism
– I have opinions but they get me in trouble
Copyright Push Technology 2014
7. What is Event Stream Processing?
• Not event passing
– Event exchange not processing, e.g. JMS
– Stateless
Copyright Push Technology 2014
8. What is Event Stream Processing?
• Not event mediation (brokering)
– Filtering, routing, and enrichment, e.g. ESB
– Stateless
Copyright Push Technology 2014
9. What is Event Stream Processing?
“Event Stream Processing deals with the
task of processing streams of event data
with the goal of identifying the meaningful
pattern within those streams” – Wikipedia
Copyright Push Technology 2014
10. What is Event Stream Processing?
• ESP is about querying data
streams
– Looking for something
– Haystack won’t stay still!
– Answers depend on multiple events
– Extremely stateful
Copyright Push Technology 2014
Where the interesting questions
are!
11. Meta-analogy
“Producing thrust with a scramjet has been
“Event stream processing is like looking for
compared to lighting a match in a hurricane
a needle in a haystack in a hurricane”
and keeping it burning” - NASA
Copyright Push Technology 2014
12. It’s like an inverted database
• Data is ‘static’
• Queries are ‘dynamic’
Query Event
Copyright Push Technology 2014
• Data is ‘dynamic’
• Queries are ‘static’
RDBMS CEP
Data Query
13. Why bother?
• Too much data
• Time is integral to the questions
• Data is moving too fast
• Databases assume static datasets
Copyright Push Technology 2014
?
14. History – Two schools of thought
• Database and make it time driven
• Logic approach with time constraints
Copyright Push Technology 2014
15. Stream Processing History
• Tapestry – ’92
– Early inverted database (not Apache!)
• Materialized views – ‘95
– [A. Gupta and I. S. Mumick. “Maintenance of materialized views: Problems,
techniques, and applications.” 1995]
• David Luckham coined term CEP – “The Power Events”. 2001
– Logic-based CEP
– Company acquired by Avaya
• Michael Franklin
– Dataflow processing in PostgreSQL
– [“TelegraphCQ: continuous dataflow processing.” 2003]
• Aurora – ‘03
– [Cherniack et al – “Scalable distributed stream processing.” 2003]
• STREAM – ‘03
– [Arasu et al – “STREAM: The Stanford Stream Data Manager.” 2003]
• Borealis – ‘05
– [Abadi et al – “The design of the Borealis stream processing engine.” 2005]
Copyright Push Technology 2014
16. Some definitions
• Tuple – a multi-set of elements ( e1, e2, … en )
– A single tuple is a monad!
• Event or Data Stream 푺풏 - any ordered pair 풔, Δ 풏
– 푠 is an unbounded sequence of tuples and
– Δ is an unbounded sequence of positive real time
intervals
– 푠 and Δ are of equal length
• Event stream processing transforms event streams
into new event streams through queries
• Outputs and inputs continuous
– Operators are continuous queries
Copyright Push Technology 2014
17. How do you measure infinity?
Copyright Push Technology 2014
How do you measure an event
stream if it’s unbounded?
18. Measuring infinity
• Don’t do it
– But just event passing – where is the fun in
that?!
• Synopses – store summary information
– Continuous average = running total + items
• Windows – define working set
– Continuous average over last N items
Copyright Push Technology 2014
21. What to do with a working set?
• Windows define the scope of interest
• Run queries against working set as it
changes
– Continuous Queries
Copyright Push Technology 2014
22. When should you run queries?
• Run queries when output is not
idempotent
• When is that?
– Contents of the window changes – maybe?
– Time advances – possibly?
– Depends on window and query
Linking cause and effect in an efficient manner
lies at the heart of CEP and is why the answer
Copyright Push Technology 2014
is not simply programming
23. How can we define queries on windows?
• Describe queries on windows using a
SQL-like syntax
SELECT AVG(price) FROM stream [ROWS N]
• [Arasu et al. – “The CQL Continuous Query
Language: Semantic Foundations and Query
Execution” 2003]
Copyright Push Technology 2014
24. Querying windows
• Sliding
SELECT * FROM s [ROWS 4 SLIDE 4]
• Partitioned
SELECT a, b FROM s
[PARTITION BY b ROWS 3]
• Time-based
SELECT * FROM s [RANGE 30 SECONDS]
Copyright Push Technology 2014
25. How do you make it fast?
• Generally in-memory the only way
• Operate as a gigantic state machine and
optimize like crazy
– Go reactive!
– Talk to Martin
Copyright Push Technology 2014
26. Why must it be fast?
• Not reactive streams!
• Flow control causes causal paradox
• Stream processing must keep up
Copyright Push Technology 2014
27. How do you make it resilient?
• Making stateful systems resilient has
challenges
• State generally changing extremely quickly
Copyright Push Technology 2014
28. Resiliency approaches
• Save all the things and replay
– But infinite data?!
– Sometimes possible because
append-only
• Save all the state
– Assumes there is less of it
– State is changing rapidly
– Too rapid to be effective
Copyright Push Technology 2014
29. Resiliency approaches
• Elsewhere checkpoint and record changes
– Maybe we can record state and things
– Many commercial systems do
• No recording - identical parallel systems
– Synchronization an issue
– Catch-up an issue
Copyright Push Technology 2014
30. How do you scale stream processing?
• Follow the crowd
• Distribute processing
• Multiple input sources
– If independent
– Flume
– Kafka
Copyright Push Technology 2014
31. How do you distribute stream processing?
• DAG of event streams
– Inputs and outputs are event streams
– Nodes are operators or groups of operators
– Nodes can be distributed
Copyright Push Technology 2014
32. Apache Storm
• Toolkit for creating distributed event flows
• Bolts (operators) and spouts (sources)
• Composed using a Clojure DSL
• Storm runs topologies
– Map-Reduce jobs finish – batch
– Topologies process forever – continuous
Copyright Push Technology 2014
34. How do you reliably distribute?
• State is now distributed
– Synchronization all but impossible
– Deterministic if relative order is preserved
• Depends on operators and their effect
• [L. Lamport - “Time, Clocks, and the Ordering of
Events in a Distributed System.” 1978]
– In theory a replay of things through the
network will recover the state
– Alternative of storing the state for all the
operators is harder
Copyright Push Technology 2014
35. How do you reliably distribute?
• Different classes of recovery
– [Hwang et al. – “High-Availability Algorithms
for Distributed Stream Processing”. 2005]
• Precise recovery – failure effects hidden perfectly
• Rollback recovery – no data loss, but outputs
may be duplicated
• Gap recovery – data lost during recovery
• Reliable distribution overlaps distribution
– Upstream backup, reactive streams?
Copyright Push Technology 2014
37. Stream Processing with Data
• Time dimension to data problems
• Data dimension to stream problems
• JOIN streams to tables
• Easy when small
• Large datasets harder
– Cache join data in memory?
– Push query into datastore?
Copyright Push Technology 2014
38. Stream Processing with Big Data
• Time dimension to Big Data problems
– Velocity (vvv) implies stream processing
• Large dataset problem domain
• But now the data is distributed!
Copyright Push Technology 2014
41. Fast Data Architecture
• Similar to recoverable architectures:
– Snapshot (queries) + incremental updates
– Current state = known state + changes
– Requires static queries - cached results
• Spark does this quite well
Copyright Push Technology 2014
42. Fast data technology
• Storm – topology deployment
• Spark – logic queries on RDDs
• Spark streaming
– repeating snapshots / micro-batch
– Fast data-ish
• Flume – fast ingest of log data
• Kafka
– pub-sub messaging as distributed commit log
• Hadoop streaming
– create M-R jobs using executable scripts
• Hive
• Cloudera Impala
– MPP SQL query engine on top of Hadoop
Copyright Push Technology 2014
44. Future Stream Processing
• Ease-of-use
– CQL or Graphical - both have drawbacks
– Queries get really complicated really quickly
• Ease-of-use + distribution
– Real systems challenge
• Fast data architectures
• Real-time machine learning
– Spark ML Library
– Hadoop Mahout
• Interactive streaming queries – declarative and
caching
– Hive and Spark
Copyright Push Technology 2014
Architect for WebLogic Server Core
Architect and then Engineering Director for Oracle Event Processing
Effectively infinite when varying over all time
Theory assumes unbounded streams
The data is moving
Slow moving data can be stored and queried (somewhat inefficiently)
Although conceptually similar to RDBMS query against a small data set, in practice very different
Many optimizations based on how the data is changing are possible
Thus the output of the query is another event stream
Streams and operators on streams are thus composable
Query plans
Can be written in a regular programming language, but often declarative
If you can’t you have to throw things overboard
And performance is a key aspect of event stream processing
E.g. commit log in a database
event streams flow over communication links
event streams flow over communication links
Requires that you use time associated with the tuple, not the current time
Some operators make this hard
Requires that you use time associated with the tuple, not the current time
Some operators make this hard
Determinism requirements?
Independence of tuples within streams?
Queries?
For instance calculate all the late running Tube (BART) trains in London (SF)
Want to join the stream of time-based location data with a static timetable
Effectively working set state
Can be held in memory
Data locality very important
Lambda architecture
As you get results from M-R, because of immutable ops are associative and commutative – easy to add to without losing semantics
Break same question into 2 parts for deep data and recent data if associate and commutative and then merge results
Can combine the two!
Data for static querying and recovery is the same