Complex Event Processing Across Web Applications
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Complex Event Processing Across Web Applications

on

  • 4,347 views

 

Statistics

Views

Total Views
4,347
Views on SlideShare
4,323
Embed Views
24

Actions

Likes
7
Downloads
215
Comments
3

2 Embeds 24

http://www.slideshare.net 22
http://www.linkedin.com 2

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Actually the terms in slide 13 like 'situation refinement' and 'predicted situations' are from my CEP work, Mike and Opher; so the comment that all of slide 13 are from Tail are not accurate.... The slide is a combination of Tali's simple diagram, but the concepts like situation refinement, etc in the context of CEP started with my work, as I recall. If this is not correct, please update, as this was a long time ago. I recall the earlier IBM side not using terms like 'situation refinement' etc; which I adapted from multisensor data fusion concepts.
    Are you sure you want to
    Your message goes here
    Processing…
  • Opher,

    My apologies to Tali Yatzkar. I first viewed this diagram in a presentation by Tim Bass. I will make sure to note Tali's work in the future.

    Thank you for pointing this out. I absolutely want to make sure that I cite the work of others as I make use of reference materials. My continued learning on this subject has been due to prior work by great thinkers.

    --Mike
    Are you sure you want to
    Your message goes here
    Processing…
  • Slide 13 with the 'lego stones' was actually done by Tali Yatzkar who used to work in my team in IBM Haifa Research Lab. If she would got royalties for the many people that has 'borrowed' this slide (without mentioning the source), she would have been very rich today. -).
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

Complex Event Processing Across Web Applications Presentation Transcript

  • 1. Complex Event Processing Across Web Applications Mike Clymer - Founder, CoBaLU Glue Conference 2009 http://www.clynetic.com
  • 2. Agenda Problem Space and Solutions CEP and Web Application Use Cases Complex Event Processing (CEP) Overview Approaching CEP Design for the Web Final Thoughts Q&A
  • 3. Problems Web as a platform is growing radically Consumer Web is increasingly about real-time information sharing Information overload for the consumer Too complex - data loses meaning Efficiency vs effectiveness in data consumption Your time is valuable
  • 4. Current Solutions Feed subscription ser vices / tools Too many data streams, lose focus Constant monitoring Aggregators Narrows the focus some Still a fire-hose of data Too noisy
  • 5. Current Solutions Grouping, tagging, and filtering (for previous solutions) Still requires too much management and setup Semantic applications * Are becoming more available Relatively new to the consumer conscience Top-Down approach has similar goals to CEP * http://www.readwriteweb.com/archives/10_semantic_apps_to_watch.php
  • 6. Goals for Improvement Require the user to monitor less data Guide the user with intelligently filtered data for context and relevance Discovery and analysis Create situational knowledge that is actionable from many data streams The Implicit Web Paying attention to what is important - context
  • 7. Agenda Problem Space and Solutions CEP and Web Application Use Cases Complex Event Processing (CEP) Overview Approaching CEP Design for the Web Final Thoughts Q&A
  • 8. CEP and Web Apps The various tools, ser vices, and content consumed create a seemingly unrelated net work of data nodes Interested in computing events related to “real life”, or web usage Could build event models from data related to the social graph
  • 9. Web App CEP Use Cases Yelp recommendations for vendors and services based on FB status, Amazon purchases, and Google calendar Media getting news leads faster (Twitter) * Tracking global health trends http://www.google.org/flutrends/ Use location to warn of epidemic hotspots * http://www.readwriteweb.com/archives/can_media_take_tips_from_t witter_techmemes_experie.php
  • 10. Web App CEP Use Cases Professional net work user lists employer as one recently acquired Notifies user to update contacts / endorsements Grabs relevant job listings from additional sources Customer Development application tracking message boards, blogs and Twitter about customer problems, needs, wants, likes, complaints, reviews
  • 11. Web App CEP Use Cases Conference management app Connects attendees, speakers, and domain experts based on event calendars, session attendance, Twitter trends, and user profiles Uses location to increase net working opportunities from previous correlation Helps conference organizers manage conference hotspots based on location of attendees, Twitter trends, service usage
  • 12. Agenda Problem Space and Solutions CEP and Web Application Use Cases Complex Event Processing (CEP) Overview Approaching CEP Design for the Web Final Thoughts Q&A
  • 13. CEP / EP Visually Detected & Real-Time Situation Predicted Events Refinement Situations Patterns from Historical Data * Adapted by Mike Clymer from (Tim Bass): http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795
  • 14. CEP / EP Concepts The main focus of CEP is to build a net work technology that creates knowledge that is both actionable and situational CEP is made for processing large amounts of real-time data with dynamic analysis and low-latency Power comes from processing real-time and historical data in an event cloud
  • 15. CEP / EP Concepts CEP uses raw events as input and processes the event data to detect actionable situations. CEP then triggers the right action, at the right time Typical CEP progression event >> correlation >> assessment >> decision >> action * Adapted by Mike Clymer from (Tim Bass): http://www.thecepblog.com/2007 /05/14/what-is-complex-event-processing-part-1/
  • 16. CEP / EP Concepts Many different definitions for CEP CEP: Architecture vs Algorithm Architectures for the event transport Can be built as EDA, ESP Needed to supply CEP with the input events Algorithms for event processing and pattern detection (situation recognition) Define an Event Processing Language (patterns)
  • 17. CEP / EP Concepts Event is the atomic element for analysis Event is some activity that happened Events are context dependent Event patterns related to timing and causal relationships bet ween events
  • 18. CEP / EP Concepts Events are represented by abstractions Raw data should be modeled as events Events are organized into hierarchies representing abstractions of real-life hierarchies Event hierarchies provide the ability to apply computation and analysis to related sets of events
  • 19. CEP / EP Concepts Goal of CEP w.r.t. ESP: Identify the meaningful events within those event streams Employ techniques such as: Detection of complex patterns of many events Event correlation and abstraction Building event hierarchies, and relationships bet ween events such as causality, membership, and timing * Adapted by Mike Clymer from (Wikipedia): http://en.wikipedia.org/wiki/Event_Stream_Processing
  • 20. Common CEP / EP Uses Algorithmic trading - Finance, Energy Fraud detection - Web Commerce, Credit Track and trace - Patients, Packages Sensor net works - RFID, GPS MMOGs - Player monitoring Net work security - Intrusion detection * Adapted by Mike Clymer from (Tim Bass): http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795
  • 21. Agenda Problem Space and Solutions Complex Event Processing (CEP) Overview CEP and Web Application Use Cases Approaching CEP Design for the Web Final Thoughts Q&A
  • 22. Architectural Approach Normalize with Build and Track Analyze Events Compare, Correlate, Common Event Data Event State Context / Scenario Analyze Scenarios Event Sources CEP Consumers User Interface Web Data Web Level One Level Two Level Three Collection API Event Agent Pre-Process Event Situation Predictive Tracking Detection Analysis Data Collection Messaging Backbone GUI Agent CEP Engine DB Management Level Four Data Profiles Process Historical Collection and Pub / Sub Refinement Data Agent Patterns Take Action from Detected Situations * Adapted by Mike Clymer from (Tim Bass): http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795
  • 23. CEP Design Thoughts Build a messaging backbone to direct communication bet ween collecting events, processing events, and analyzing situations (ner vous system) Message queues Jabber like application ser vers for pub / sub Define sets of event patterns, or a pattern language (EPLs) Possibly use DSL - Spec for specific problem
  • 24. CEP Design Thoughts Normalize incoming data streams Define and adhere to standard formats Identify common event data and event pattern sets Try to leverage data standards Improves data transport, eases event comparison Utilize data snapshots for historical and pattern analysis
  • 25. CEP Design Thoughts Make use of domain specific agentry Agents to collect from heterogeneous data sources (sensors) Agents to handle basic event processing and normalization (pre-processing) Agents to apply rules and perform analysis (Bayesian nets, neural nets, semantic rules, etc) Possibly use agents to drive actions from detected situations
  • 26. CEP Design Thoughts Design for low-latency, high throughput Situation and context detection are often associated with a time element Data collection agentry will often produce high- volumes of data based on the diversity of data nodes Messaging backbone should be scalable for concurrency
  • 27. Agenda Problem Space and Common Solutions Complex Event Processing (CEP) Overview Applying CEP to the Consumer Web Approaching CEP Design for the Web Final Thoughts Q&A
  • 28. Final Thoughts Design CEP applications with passive user experience in mind Value is derived from helping user figure out what is important UX is applicable to thirdparty ser vice consumers Standards, protocols, and reference architectures can be “good enough” Definitely needed for common info exchange Practical application > Optimal theory
  • 29. Final Thoughts Design with goal adding true data utility Data pass-through is not enough Context is the key to data enrichment Tim O'Reilly Quote: quot;Create more value than you capture.quot; Different context, but relevant to our discussion
  • 30. Agenda Problem Space and Common Solutions Complex Event Processing (CEP) Overview Applying CEP to the Consumer Web Approaching CEP Design for the Web Final Thoughts Q&A
  • 31. Further Exploration http://complexevents.com/ http://www.thecepblog.com/2007 /05/14/what- is-complex-event-processing-part-1/ http://www.slideshare.net/TimBassCEP/slideshows http://en.wikipedia.org/wiki/ Complex_event_processing http://apama.typepad.com/my_weblog/ http://epthinking.blogspot.com/
  • 32. Further Exploration http://www.amazon.com/Distributed-Event- Based-Systems-Gero-M%C3%BChl/dp/ 3540326510/ref=sr_1_3? ie=UTF8&s=books&qid=1241925043&sr=8-3 http://www.amazon.com/XMPP-Definitive-Real- Time-Applications-Technologies/dp/059652126X/ ref=sr_1_1? ie=UTF8&s=books&qid=1241973202&sr=8-1 http://www.amazon.com/Power-Events- Introduction-Processing-Distributed/dp/ 0201727897 /ref=sr_1_1? ie=UTF8&s=books&qid=1241973250&sr=8-1