Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Complex Event Processing
Across Web Applications
   Mike Clymer - Founder, CoBaLU

       Glue Conference 2009

     http:...
Agenda
Problem Space and Solutions
CEP and Web Application Use Cases
Complex Event Processing (CEP)
Overview
Approaching C...
Problems
Web as a platform is growing radically
Consumer Web is increasingly about
real-time information sharing
Informati...
Current Solutions
Feed subscription ser vices / tools
 Too many data streams, lose focus
 Constant monitoring

Aggregators...
Current Solutions
Grouping, tagging, and filtering (for
previous solutions)
 Still requires too much management and setup

...
Goals for Improvement
 Require the user to monitor less data
 Guide the user with intelligently filtered
 data for context ...
Agenda
Problem Space and Solutions
CEP and Web Application Use Cases
Complex Event Processing (CEP)
Overview
Approaching C...
CEP and Web Apps
The various tools, ser vices, and content
consumed create a seemingly unrelated
net work of data nodes
In...
Web App CEP Use Cases
 Yelp recommendations for vendors and
 services based on FB status, Amazon
 purchases, and Google ca...
Web App CEP Use Cases
 Professional net work user lists
 employer as one recently acquired
  Notifies user to update contac...
Web App CEP Use Cases
 Conference management app
  Connects attendees, speakers, and domain
  experts based on event calen...
Agenda
Problem Space and Solutions
CEP and Web Application Use Cases
Complex Event Processing (CEP)
Overview
Approaching C...
CEP / EP Visually
                                                                                                        ...
CEP / EP Concepts
The main focus of CEP is to build a
net work technology that creates
knowledge that is both actionable a...
CEP / EP Concepts
CEP uses raw events as input and
processes the event data to detect
actionable situations. CEP then trig...
CEP / EP Concepts
Many different definitions for CEP
CEP: Architecture vs Algorithm
 Architectures for the event transport
...
CEP / EP Concepts

Event is the atomic element for analysis
Event is some activity that happened
Events are context depend...
CEP / EP Concepts
Events are represented by abstractions
 Raw data should be modeled as events

Events are organized into ...
CEP / EP Concepts
Goal of CEP w.r.t. ESP:
  Identify the meaningful events within those
  event streams
  Employ technique...
Common CEP / EP Uses
      Algorithmic trading - Finance, Energy
      Fraud detection - Web Commerce, Credit
      Track ...
Agenda
Problem Space and Solutions
Complex Event Processing (CEP)
Overview
CEP and Web Application Use Cases
Approaching C...
Architectural Approach
                           Normalize with              Build and Track            Analyze Events   ...
CEP Design Thoughts
Build a messaging backbone to direct
communication bet ween collecting
events, processing events, and
...
CEP Design Thoughts
Normalize incoming data streams
 Define and adhere to standard formats
 Identify common event data and ...
CEP Design Thoughts
Make use of domain specific agentry
 Agents to collect from heterogeneous data
 sources (sensors)
 Agen...
CEP Design Thoughts
Design for low-latency, high throughput
 Situation and context detection are often
 associated with a ...
Agenda
Problem Space and Common Solutions
Complex Event Processing (CEP)
Overview
Applying CEP to the Consumer Web
Approac...
Final Thoughts
Design CEP applications with passive
user experience in mind
 Value is derived from helping user figure out ...
Final Thoughts
Design with goal adding true data
utility
 Data pass-through is not enough
 Context is the key to data enri...
Agenda
Problem Space and Common Solutions
Complex Event Processing (CEP)
Overview
Applying CEP to the Consumer Web
Approac...
Further Exploration
http://complexevents.com/
http://www.thecepblog.com/2007   /05/14/what-
is-complex-event-processing-pa...
Further Exploration
http://www.amazon.com/Distributed-Event-
Based-Systems-Gero-M%C3%BChl/dp/
3540326510/ref=sr_1_3?
ie=UT...
Upcoming SlideShare
Loading in …5
×

Complex Event Processing Across Web Applications

4,310 views

Published on

Published in: Technology
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Actually the terms in slide 13 like 'situation refinement' and 'predicted situations' are from my CEP work, Mike and Opher; so the comment that all of slide 13 are from Tail are not accurate.... The slide is a combination of Tali's simple diagram, but the concepts like situation refinement, etc in the context of CEP started with my work, as I recall. If this is not correct, please update, as this was a long time ago. I recall the earlier IBM side not using terms like 'situation refinement' etc; which I adapted from multisensor data fusion concepts.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Opher,

    My apologies to Tali Yatzkar. I first viewed this diagram in a presentation by Tim Bass. I will make sure to note Tali's work in the future.

    Thank you for pointing this out. I absolutely want to make sure that I cite the work of others as I make use of reference materials. My continued learning on this subject has been due to prior work by great thinkers.

    --Mike
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Complex Event Processing Across Web Applications

  1. 1. Complex Event Processing Across Web Applications Mike Clymer - Founder, CoBaLU Glue Conference 2009 http://www.clynetic.com
  2. 2. Agenda Problem Space and Solutions CEP and Web Application Use Cases Complex Event Processing (CEP) Overview Approaching CEP Design for the Web Final Thoughts Q&A
  3. 3. Problems Web as a platform is growing radically Consumer Web is increasingly about real-time information sharing Information overload for the consumer Too complex - data loses meaning Efficiency vs effectiveness in data consumption Your time is valuable
  4. 4. Current Solutions Feed subscription ser vices / tools Too many data streams, lose focus Constant monitoring Aggregators Narrows the focus some Still a fire-hose of data Too noisy
  5. 5. Current Solutions Grouping, tagging, and filtering (for previous solutions) Still requires too much management and setup Semantic applications * Are becoming more available Relatively new to the consumer conscience Top-Down approach has similar goals to CEP * http://www.readwriteweb.com/archives/10_semantic_apps_to_watch.php
  6. 6. Goals for Improvement Require the user to monitor less data Guide the user with intelligently filtered data for context and relevance Discovery and analysis Create situational knowledge that is actionable from many data streams The Implicit Web Paying attention to what is important - context
  7. 7. Agenda Problem Space and Solutions CEP and Web Application Use Cases Complex Event Processing (CEP) Overview Approaching CEP Design for the Web Final Thoughts Q&A
  8. 8. CEP and Web Apps The various tools, ser vices, and content consumed create a seemingly unrelated net work of data nodes Interested in computing events related to “real life”, or web usage Could build event models from data related to the social graph
  9. 9. Web App CEP Use Cases Yelp recommendations for vendors and services based on FB status, Amazon purchases, and Google calendar Media getting news leads faster (Twitter) * Tracking global health trends http://www.google.org/flutrends/ Use location to warn of epidemic hotspots * http://www.readwriteweb.com/archives/can_media_take_tips_from_t witter_techmemes_experie.php
  10. 10. Web App CEP Use Cases Professional net work user lists employer as one recently acquired Notifies user to update contacts / endorsements Grabs relevant job listings from additional sources Customer Development application tracking message boards, blogs and Twitter about customer problems, needs, wants, likes, complaints, reviews
  11. 11. Web App CEP Use Cases Conference management app Connects attendees, speakers, and domain experts based on event calendars, session attendance, Twitter trends, and user profiles Uses location to increase net working opportunities from previous correlation Helps conference organizers manage conference hotspots based on location of attendees, Twitter trends, service usage
  12. 12. Agenda Problem Space and Solutions CEP and Web Application Use Cases Complex Event Processing (CEP) Overview Approaching CEP Design for the Web Final Thoughts Q&A
  13. 13. CEP / EP Visually Detected & Real-Time Situation Predicted Events Refinement Situations Patterns from Historical Data * Adapted by Mike Clymer from (Tim Bass): http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795
  14. 14. CEP / EP Concepts The main focus of CEP is to build a net work technology that creates knowledge that is both actionable and situational CEP is made for processing large amounts of real-time data with dynamic analysis and low-latency Power comes from processing real-time and historical data in an event cloud
  15. 15. CEP / EP Concepts CEP uses raw events as input and processes the event data to detect actionable situations. CEP then triggers the right action, at the right time Typical CEP progression event >> correlation >> assessment >> decision >> action * Adapted by Mike Clymer from (Tim Bass): http://www.thecepblog.com/2007 /05/14/what-is-complex-event-processing-part-1/
  16. 16. CEP / EP Concepts Many different definitions for CEP CEP: Architecture vs Algorithm Architectures for the event transport Can be built as EDA, ESP Needed to supply CEP with the input events Algorithms for event processing and pattern detection (situation recognition) Define an Event Processing Language (patterns)
  17. 17. CEP / EP Concepts Event is the atomic element for analysis Event is some activity that happened Events are context dependent Event patterns related to timing and causal relationships bet ween events
  18. 18. CEP / EP Concepts Events are represented by abstractions Raw data should be modeled as events Events are organized into hierarchies representing abstractions of real-life hierarchies Event hierarchies provide the ability to apply computation and analysis to related sets of events
  19. 19. CEP / EP Concepts Goal of CEP w.r.t. ESP: Identify the meaningful events within those event streams Employ techniques such as: Detection of complex patterns of many events Event correlation and abstraction Building event hierarchies, and relationships bet ween events such as causality, membership, and timing * Adapted by Mike Clymer from (Wikipedia): http://en.wikipedia.org/wiki/Event_Stream_Processing
  20. 20. Common CEP / EP Uses Algorithmic trading - Finance, Energy Fraud detection - Web Commerce, Credit Track and trace - Patients, Packages Sensor net works - RFID, GPS MMOGs - Player monitoring Net work security - Intrusion detection * Adapted by Mike Clymer from (Tim Bass): http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795
  21. 21. Agenda Problem Space and Solutions Complex Event Processing (CEP) Overview CEP and Web Application Use Cases Approaching CEP Design for the Web Final Thoughts Q&A
  22. 22. Architectural Approach Normalize with Build and Track Analyze Events Compare, Correlate, Common Event Data Event State Context / Scenario Analyze Scenarios Event Sources CEP Consumers User Interface Web Data Web Level One Level Two Level Three Collection API Event Agent Pre-Process Event Situation Predictive Tracking Detection Analysis Data Collection Messaging Backbone GUI Agent CEP Engine DB Management Level Four Data Profiles Process Historical Collection and Pub / Sub Refinement Data Agent Patterns Take Action from Detected Situations * Adapted by Mike Clymer from (Tim Bass): http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795
  23. 23. CEP Design Thoughts Build a messaging backbone to direct communication bet ween collecting events, processing events, and analyzing situations (ner vous system) Message queues Jabber like application ser vers for pub / sub Define sets of event patterns, or a pattern language (EPLs) Possibly use DSL - Spec for specific problem
  24. 24. CEP Design Thoughts Normalize incoming data streams Define and adhere to standard formats Identify common event data and event pattern sets Try to leverage data standards Improves data transport, eases event comparison Utilize data snapshots for historical and pattern analysis
  25. 25. CEP Design Thoughts Make use of domain specific agentry Agents to collect from heterogeneous data sources (sensors) Agents to handle basic event processing and normalization (pre-processing) Agents to apply rules and perform analysis (Bayesian nets, neural nets, semantic rules, etc) Possibly use agents to drive actions from detected situations
  26. 26. CEP Design Thoughts Design for low-latency, high throughput Situation and context detection are often associated with a time element Data collection agentry will often produce high- volumes of data based on the diversity of data nodes Messaging backbone should be scalable for concurrency
  27. 27. Agenda Problem Space and Common Solutions Complex Event Processing (CEP) Overview Applying CEP to the Consumer Web Approaching CEP Design for the Web Final Thoughts Q&A
  28. 28. Final Thoughts Design CEP applications with passive user experience in mind Value is derived from helping user figure out what is important UX is applicable to thirdparty ser vice consumers Standards, protocols, and reference architectures can be “good enough” Definitely needed for common info exchange Practical application > Optimal theory
  29. 29. Final Thoughts Design with goal adding true data utility Data pass-through is not enough Context is the key to data enrichment Tim O'Reilly Quote: quot;Create more value than you capture.quot; Different context, but relevant to our discussion
  30. 30. Agenda Problem Space and Common Solutions Complex Event Processing (CEP) Overview Applying CEP to the Consumer Web Approaching CEP Design for the Web Final Thoughts Q&A
  31. 31. Further Exploration http://complexevents.com/ http://www.thecepblog.com/2007 /05/14/what- is-complex-event-processing-part-1/ http://www.slideshare.net/TimBassCEP/slideshows http://en.wikipedia.org/wiki/ Complex_event_processing http://apama.typepad.com/my_weblog/ http://epthinking.blogspot.com/
  32. 32. Further Exploration http://www.amazon.com/Distributed-Event- Based-Systems-Gero-M%C3%BChl/dp/ 3540326510/ref=sr_1_3? ie=UTF8&s=books&qid=1241925043&sr=8-3 http://www.amazon.com/XMPP-Definitive-Real- Time-Applications-Technologies/dp/059652126X/ ref=sr_1_1? ie=UTF8&s=books&qid=1241973202&sr=8-1 http://www.amazon.com/Power-Events- Introduction-Processing-Distributed/dp/ 0201727897 /ref=sr_1_1? ie=UTF8&s=books&qid=1241973250&sr=8-1

×