Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Walk through Streaming Technologies: EPL

241 views

Published on

Walk through Streaming Technologies: EPL
Riccardo Tommasini - DEIB Politecnico di Milano

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Walk through Streaming Technologies: EPL

  1. 1. BigData Semantic Approach to Big Data and Event Processing Walk through Streaming Technologies: EPL Riccardo Tommasini PhD Student Politecnico di Milano riccardo.tommasini@polimi.it
  2. 2. BigData Who I am 2 Master Degree in C.S. @ Politecnico Of Milano M.D. Thesis on Stream Reasoning I’ll start my Phd in November 2k15
  3. 3. BigData Agenda 3 ✓ •  IntroducAon •  Event Processing Languages Overview •  EPL Running Example: – Event CreaAon – DSMS Features – CEP Features •  Java Demo (if we have Ame) •  QuesAons
  4. 4. BigData Intro: Esper Architecture Source Link
  5. 5. BigData Esper Features: Efficient Event Processing • Expressive Event Query Language : • Continuous queries • filtering, aggregation, joins of streams • Comprehensive pattern detection • Pull and Push • High performances, low latency
  6. 6. BigData Esper Features: Extensible Middleware •  RunAme statement management •  API or configuraAon driven •  Plug-in SDK for funcAons, aggregaAons, views and paSern detecAon extensions •  Adapters: CSV, JMS in/out, API, DB, Socket, HTTP •  Data distribuAon service: •  data push management •  service layer
  7. 7. BigData Esper Features: Rich Web-Based User Interface • Eventlet technology: • interactive continuous displays • CEP Engine management • EPL Statements design • Drill-down and script browsing integration
  8. 8. BigData Esper Features: HA enabled (EsperHA) • Per-statement configuration • Transient combinable with fully resilient behaviour • Hot standby API • Hot backup • Highly optimised data storage technology • Engine state RDBMS storage option
  9. 9. BigData9 Development •  EPL Statements have SQL-like syntax •  java POJO or .Net object for real-time customised actionable intelligence Deployment •  Standalone deployment •  Embedded (into existing middleware systems) Development and Deployment
  10. 10. BigData Agenda 10 ✓ •  IntroducAon ✓ •  Event Processing Languages Overview •  EPL Running Example: – Event CreaAon – DSMS Features – CEP Features •  Java Demo •  QuesAons
  11. 11. BigData1 1 EPL Statements are registered into Esper and continuously executed as the live data stream are pushed through Event Streams/Complex Event Processing: EPL Event Processing Language (EPL) Continuous Query Answering Complex Pattern Matching (causality relationship)
  12. 12. BigData1 2 •  Push Based •  Data tuple from sensors, trace files etc. •  Push Based •  Continuously executed against incoming events •  Receive data tuples from queries •  Push data tuple to other queries •  Receive processed tuples EPL Query Subscriber Source Listener EPL ConAnuous Query Processing Model
  13. 13. BigData1 3 •  Sources act as input •  EPL Queries Integrate sources •  Listener propagates query results •  Subscribers act as output EPL Query Model can be represented as a graph Graph nodes are functional components, manually connect by the data stream. EPL ConAnuous Query Processing Model
  14. 14. BigData Agenda 14 ✓ •  IntroducAon ✓ •  Event Processing Languages Overview✓ •  EPL Running Example: – Event CreaAon – DSMS Features – CEP Features •  Java Demo •  QuesAons
  15. 15. BigData1 5 We starts with the demo time https://github.com/riccardotommasini/esper-tutorial http://esper-epl-tryout.appspot.com/epltryout/mainform.html ONLINE VERSION Java Code Data for the running example DEMO
  16. 16. BigData16 Count the number of fires detected using a set of smoke and temperature sensors in the last 10 minutes Request Events CondiAon •  Smoke Event: String sensor, boolean state •  Temperature Event: String sensor, double temperature •  Fire Event: String sensor, boolean smoke, double temperature Fire: The same sensor detects smoke followed by temperature > 50°C Running Example
  17. 17. BigData1 7 Running Example Graph
  18. 18. BigData18 EPL create schema clause create schema schema_name [as] (property_name property_type [,property_name property_type [,…]) [inherits inherited_event_type [,inherited_event_type][,...]] EPL: Event DeclaraAon
  19. 19. BigData19 create schema SmokeSensorEvent( sensor string, smoke boolean ); create schema TemperatureSensorEvent( sensor string, temperature double ); create schema FireEvent( sensor string, smoke boolean, temperature double ); EPL: Running Example: Events Types
  20. 20. BigData SmokeSensorEvent={sensor='S1', smoke=false} TemperatureSensorEvent={sensor='S1', temperature=30} t=t.plus(1 seconds) SmokeSensorEvent={sensor='S1', smoke=true} TemperatureSensorEvent={sensor='S1', temperature=40} t=t.plus(1 seconds) SmokeSensorEvent={sensor='S1', smoke=true} TemperatureSensorEvent={sensor='S1', temperature=55} t=t.plus(1 seconds) SmokeSensorEvent={sensor='S1', smoke=true} TemperatureSensorEvent={sensor='S1', temperature=55} t=t.plus(1 seconds) SmokeSensorEvent={sensor='S1', smoke=false} TemperatureSensorEvent={sensor='S1', temperature=55} 20Stream & Complex Event Processing - Introduction Advancing time manually is only required by the online interface, normally time pass by on its own :-) EPL: Running Example: Events Types
  21. 21. BigData Agenda 21 ✓ •  IntroducAon ✓ •  Event Processing Languages Overview✓ •  EPL Running Example:✓ – Event CreaAon✓ – DSMS Features – CEP Features •  Java Demo •  QuesAons
  22. 22. BigData22 [insert into insert_into_def] select select_list from stream_def [as name] [, stream_def [as name]] [,...] [where search_conditions] [group by grouping_expression_list] [having grouping_search_conditions] [output output_specification] [order by order_by_expression_list] [limit num_rows] •  Similar to SQL (select, where) •  views are used inset of tables EPL: Query RegistraAon
  23. 23. BigData23 @Name(‘Q0’) select * from TemperatureSensorEvent where temperature > 50; @Name(‘Q1’) select avg(temperature) from TemperatureSensorEvent; Stream Lookup Aggregate EPL Queries: Examples
  24. 24. BigData24 • Similar to SQL tables • define data available for querying and filtering • sorting, aggregation, grouping operation are possible • usually represented as windows over the streams of events EPL: Views
  25. 25. BigData25 Type Syntax Description Logical Sliding win:time(time period) Sliding time window extending the specified time interval into the past Logical Tumbling win:time_batch(time period) Tumbling window that batches events and releases them every specified time interval, with flow control options Physical Sliding win:length(size) Sliding length window extending the specified number of events into the past Physical Tumbling win:length_batch(size) Tumbling window that batches events and releases them when a given minimum number is has been collected EPL: Statements and Windows
  26. 26. BigData26 EPL: Logical Sliding Window
  27. 27. BigData At: 2015-01-07 14:00:03.000 Insert Q2-output={avg(temperature)=45.0} At: 2015-01-07 14:00:04.000 Insert Q2-output={avg(temperature)=50.0} At: 2015-01-07 14:00:04.000 Insert Q2-output={avg(temperature)=51.25} 27 Query Q2 Trace @Name(‘Q2’) select avg(temperature) from TemperatureSensorEvent.win:time(4 sec); Running Example: Logical Sliding Window At: 2015-01-07 14:00:00.000 Insert Q2-output={avg(temperature)=30.0} At: 2015-01-07 14:00:01.000 Insert Q2-output={avg(temperature)=35.0} At: 2015-01-07 14:00:02.000 Insert Q2-output={avg(temperature)=41.66...} In Esper, logical sliding windows report as soon as a new event arrives and an old one expires
  28. 28. BigData28 EPL: Logical Tumbling Window
  29. 29. BigData29 Query Q3 Trace @Name(‘Q3’) select avg(temperature) from TemperatureSensorEvent.win:time_batch(4 sec); In Esper, logical tumbling windows report only when the window closes Running Example: Logical Tumbling Window At: 2015-01-07 14:00:04.000 Insert Q3-output={avg(temperature)=45.0}
  30. 30. BigData30 EPL: Physical Sliding Window
  31. 31. BigData31 select avg(temperature) from TemperatureSensorEvent.win:length(5); Running Example: Physical Sliding Window Query Q4 Trace At: 2015-01-07 14:00:00.000 Insert Q4-output={avg(temperature)=30.0} At: 2015-01-07 14:00:01.000 Insert Q4-output={avg(temperature)=35.0} At: 2015-01-07 14:00:02.000 Insert Q4-output={avg(temperature)=41.6...} At: 2015-01-07 14:00:03.000 Insert Q4-output={avg(temperature)=45.0} At: 2015-01-07 14:00:04.000 Insert Q4-output={avg(temperature)=47.0} In Esper, physical sliding windows report as soon as a new event arrives
  32. 32. BigData32 Query Q5 Trace @Name(‘Q5’) select avg(temperature) from TemperatureSensorEvent.win:length_batch(5); In Esper, logical tumbling windows report only when the window closes EPL: Physical Tumbling Window At: 2015-01-07 14:00:04.000 Insert Q5- output={avg(temperature)=47.0}
  33. 33. BigData3 3 [output [[all|first|last|snapshot|] every output_rate [seconds | events ]] •  The output clause is optional •  It can control the event output rate •  It can suppress output events EPL: Controlling ReporAng
  34. 34. BigData34 Query Q6: Logical Windows Output Clause Trace @Name(’Q6') select avg(temperature) from TemperatureSensorEvent.win:time(4 sec) output snapshot every 2 sec; Running Example: Controlling ReporAng At: 2015-01-07 14:00:02.000 Insert Q6-output={avg(temperature)=35.0} At: 2015-01-07 14:00:04.000 Insert Q6-output={avg(temperature)=45.0}
  35. 35. BigData35 Query Q7: Physical Windows Output Clause Trace @Name('Q7') select avg(temperature) from TemperatureSensorEvent.win:length(4) output snapshot every 2 events; Running Example: Controlling ReporAng At: 2015-01-07 14:00:01.000 Insert Q7-output={avg(temperature)=35.0} At: 2015-01-07 14:00:03.000 Insert Q7-output={avg(temperature)=45.0}
  36. 36. BigData36 •  Looking for a pattern of events over the incoming data streams •  pattern can be temporal as well as physical •  it’s implemented via finite states machine EPL: Event PaSern Matching
  37. 37. BigData Agenda 37 ✓ •  IntroducAon ✓ •  Event Processing Languages Overview✓ •  EPL Running Example:✓✓ – Event CreaAon✓ – DSMS Features✓ – CEP Features •  Java Demo •  QuesAons
  38. 38. BigData38 EPL: PaSern Operators •  Logical operators: and, or, not •  Temporal operators that operate on event order: -> (followed-by) •  Operators that control paSern finder creaAon and terminaAon: every, every-dis9nct, [num] and un9l •  sub-expression lifeAme guards: 9mer:within, 9mer:withinmax and while-expression
  39. 39. BigData39 @Name('Q8') select a.sensor from pattern [every (a = SmokeSensorEvent(smoke=true) -> TemperatureSensorEvent(temperature > 50, sensor=a.sensor) where timer:within(2 sec))]; Running Example: within PaSern Operator Query Trace At: 2015-01-07 14:00:02.000 Insert Q8-output={a.sensor='S1'} At: 2015-01-07 14:00:03.000 Insert Q8-output={a.sensor='S1'}
  40. 40. BigData •  Default: every operator the paSern sub- expression stops when the paSern sub-expression evaluates to true or false •  Every Clause: every Ame a paSern sub-expression within an every operator turns true the engine starts a new acAve sub-expression looking for more event(s) or Aming condiAons that match the paSern sub-expression. 40Stream & Complex Event Processing - Introduction EPL: “every” PaSern Operator Precedence MaSers!
  41. 41. BigData Simple Example: every (A -> B) •  PaSern – every ( A -> B ) •  Events – A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4 •  Results – Detect an A event followed by a B event. At the Ame when B occurs the paSern matches, then the paSern matcher restarts and looks for the next A event. 1.  Matches on B1 for combinaAon {A1 , B1} 2.  Matches on B3 for combinaAon {A2 , B3} 3.  Matches on B4 for combinaAon {A4 , B4} 41Stream & Complex Event Processing - Introduction
  42. 42. BigData •  PaSern – every A -> B •  Events – A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4 •  Results – The paSern fires for every A event followed by a B event. 1.  Matches on B1 for combinaAon {A1 , B1} 2.  Matches on B3 for combinaAon {A2 , B3} and {A3 , B3} 3.  Matches on B4 for combinaAon {A4 , B4} 42Stream & Complex Event Processing - Introduction Simple Example: every A -> B
  43. 43. BigData •  PaSern – A -> every B •  Events – A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4 •  Results – The paSern fires for an A event followed by every B event. 1.  Matches on B1 for combinaAon {A1 , B1} 2.  Matches on B2 for combinaAon {A1 , B2} 3.  Matches on B3 for combinaAon {A1 , B3} 4.  Matches on B4 for combinaAon {A1 , B4} 43Stream & Complex Event Processing - Introduction Simple Example: A -> every B
  44. 44. BigData •  PaSern –  every A -> every B •  Events –  A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4 •  Results –  The paSern fires for every A event followed by every B event. 1.  Matches on B1 for combinaAon {A1 , B1} 2.  Matches on B2 for combinaAon {A1 , B2} 3.  Matches on B3 for combinaAon {A1 , B3}, {A2 , B3} and {A3 , B3} 4.  Matches on B4 for combinaAon {A1 , B4}, {A2 , B4}, {A3 , B4} and {A4 , B4} 44Stream & Complex Event Processing - Introduction Simple Example: every A -> every B
  45. 45. BigData EPL: Guards PaSern Operators Claim: the data window onto a paSern stream does not serve to limit paSern sub-expression lifeAme. •  The 9mer:within construct and the and not constructs can be used to end acAve sub- expressions. 45Stream & Complex Event Processing - Introduction
  46. 46. BigData •  Events –  A1 A2 B1 •  PaSern –  every A -> B •  Results –  {A1 , B1} and {A2 , B1} •  Events –  A1 A2 B1 •  PaSern –  every A -> (B and not A) •  Results –  {A2 , B1} –  The and not operators cause the sub- expression looking for {A1, B?} to end when A2 arrives. 46Stream & Complex Event Processing - Introduction Guards Example 1
  47. 47. BigData •  Events –  A1 received at to+ 1 sec –  A2 received at to+ 3 sec –  B1 received at to+ 4 sec •  PaSern –  every A -> B •  Results –  {A1 , B1} and {A2 , B1} •  Events –  A1 received at to+ 1 sec –  A2 received at to+ 2 sec –  B1 received at to+ 3 sec •  PaSern –  every A -> (B where Amer:within(2 sec) ) •  Results –  {A2 , B1} –  The where 9mer:within operators cause the sub-expression looking for {A1, B?} to end aker 2 seconds. 47Stream & Complex Event Processing - Introduction Guards Example 2
  48. 48. BigData Forwards events to other streams for further downstream processing: Acts as a listener 48Stream & Complex Event Processing - Introduction EPL: “Insert Into” Clause
  49. 49. BigData49Stream & Complex Event Processing - Introduction Running Example: “Insert Into” Clause @Name(‘Final’) insert into FireComplexEvent select a.sensor as sensor, a.smoke as smoke, b.temperature as temperature from pattern [every ( a = SmokeSensorEvent(smoke=true) -> b = TemperatureSensorEvent(temperature>5, sensor=a.sensor) where timer:within(2 sec) ) ] ; Final Query Trace At: 2015-01-07 14:00:01.000 Insert FireComplexEvent={sensor='S1', smoke=true, temperature=40.0} At: 2015-01-07 14:00:02.000 Insert FireComplexEvent={sensor='S1', smoke=true, temperature=55.0} At: 2015-01-07 14:00:03.000 Insert FireComplexEvent={sensor='S1', smoke=true, temperature=55.0}
  50. 50. BigData50 Downstream Query Trace Running Example Downstream Query select count(*) from FireComplexEvent.win:time(10 sec) At: 2015-01-07 14:00:01.000 Insert Downstream-output={count(*)=1} At: 2015-01-07 14:00:02.000 Insert Downstream-output={count(*)=2} At: 2015-01-07 14:00:03.000 Insert Downstream-output={count(*)=3}
  51. 51. BigData Agenda 51 ✓ •  IntroducAon ✓ •  Event Processing Languages Overview✓ •  EPL Running Example:✓✓✓ – Event CreaAon✓ – DSMS Features✓ – CEP Features✓ •  Java Demo •  QuesAons
  52. 52. BigData52 Equivalent methods alternative to EPL: • Java POJO • Maps • XML One More Thing: Event DeclaraAon AlternaAves
  53. 53. BigData53 package us.wsu.knoesis.tutorial.events; public class FireEvent { private String sensor; private boolean smoke; }•  Getter and Setter naming convention are relevant to access parameter form EPL •  the construct must be defined as well •  toString is used to represent display the event, consider redefinition Event DeclaraAon Java POJO
  54. 54. BigData Agenda 54 ✓ •  IntroducAon ✓ •  Event Processing Languages Overview✓ •  EPL Running Example:✓✓✓ – Event CreaAon✓ – DSMS Features✓ – CEP Features✓ •  Java Demo •  QuesAons
  55. 55. BigData5 5 Thank You! End!
  56. 56. BigData5 6 RiccardoTommasini+ @rictomm tomma156 riccardo@knoesis.orgRiccardo Tommasini riccardotommasini Contact
  57. 57. BigData57 Resources
  58. 58. BigData58 Streamreasoning.org StreamReasoning@GitHub RDF Stream Processors PhD CEP Course @Polimi Stream Reasoning Tutorial Resources
  59. 59. BigData Ques9ons? 59 ???

×