CEP Overview v1 2 for public use
Upcoming SlideShare
Loading in...5
×
 

CEP Overview v1 2 for public use

on

  • 1,263 views

Overview of CEP

Overview of CEP

Statistics

Views

Total Views
1,263
Views on SlideShare
1,178
Embed Views
85

Actions

Likes
0
Downloads
73
Comments
0

1 Embed 85

http://ctrlaltcep.com 85

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

CEP Overview v1 2 for public use CEP Overview v1 2 for public use Presentation Transcript

  • CEP 101Complex Event Processing Paul Vincent TIBCO Software © Copyright 2000-2012 TIBCO Software Inc.
  • PRESENTER Paul Vincent, CTO Business Rules and CEP ◦ Part of TIBCO QL product team ◦ Contributor to OMG DMN Decision Model and Notation standard ◦ Contributor to http://www.thetibcoblog.com category on CEP TIBCO Software Inc.: ◦ Software tools+solutions for “event enabled enterprises” ◦ History: pub-sub middleware (eg RV), standard event processing (eg BW) ◦ Recent innovations, news —Hardware Logging and JMS servers —Statistical server for R and S+ analytic languages —Event processing combined with decisions and BPMN ◦ Headquartered in Palo Alto, California ◦ Over 3,000 customers and offices in 40 countries © Copyright 2000-2012 TIBCO Software Inc.
  • TOPICS Intro to Complex Event Processing IT perspective and Technologies Examples CEP Market, Tools Available © Copyright 2000-2012 TIBCO Software Inc.
  • 2-3 DECADE’S CHANGES IN BUSINESS LATENCY Process Cycle-Time Improvements Perform Trading Analytics 30 minutes 5 seconds Airline Operations 20 minutes 30 seconds Handle Call Center Inquiries 8 hours 10 seconds Track Financial/Risk Position 1 day 5 minutes Supply Chain Updates 1 day 15 minutes Mail… express… fax… e-mail Document Transfer 3 days 20 seconds Phone/Service Activation 3 days 1 hour Refresh Data Warehouse 1 month 1 hour Settle Trade/Transaction 5 days 1 day Build-to-Order Product 6 weeks 1 day Source: Gartner © Copyright 2000-2012 TIBCO Software Inc.
  • REAL-WORLD EVENTS Customer New Customer Checks Order Logon “Close Account” Fed Base Rate Web Page Increase Contract Mobile Call Production Submitted New from CT Item @11.13 Arrives at Liability Contract Store Added Returned Rental thru EDI Rental Car Car Returned Crashed © Copyright 2000-2012 TIBCO Software Inc.
  • WHERE IS THE INFORMATION? Customer New Customer Checks Order Logon Close Account Fed Base Rate Web Page Increase Contract Mobile Call Production Submitted New from CT Item @11.13 Arrives at Liability Contract Store Added Returned Rental thru EDI Rental Car Car Returned Crashed © Copyright 2000-2012 TIBCO Software Inc.
  • WHERE IS THE INFORMATION? 7 Change in Fraud Customer Product Sales Risk! CrossSell Trend Risk of Opportunity Customer Defection Contract Cell phone Validated Employee Compliance fraud alert Over hours Limit Approached Contract Customer Valid Rental now rated Contract Gold Complete © Copyright 2000-2012 TIBCO Software Inc.
  • COMPLEX BUSINESS PROBLEMS Fraud / Theft Relevant event of interest ◦ Thousands-to-millions of high-value small-size product items or transactions Product location ◦ How do you identify known patterns of “suspicious” change behavior? Logistics / Scheduling ◦ Raw material, production & delivery scheduling and resources are complex and prone to change Resource, ◦ How do we reallocate resources to handle business and requirement change production changes? Activity Monitoring ◦ Complex production and supply process with multiple Process data change actors ◦ How to measure and action Key Performance Indicators? © Copyright 2000-2012 TIBCO Software Inc.
  • TYPES OF “EVENT” Change of state in some entity ◦ Customer call Incident ◦ Bank debit ◦ Aircraft movement Observation of some entity ◦ CRM record of a customer call Observation ◦ ATM report of debit transaction success ◦ Radar plot update of an aircraft IT Message ◦ Queued point-to-point message IT Message ◦ Publish / subscribe message © Copyright 2000-2012 TIBCO Software Inc.
  • ASSOCIATED EVENTS Positive Events ◦ Product item X arrives at Production station S from Store T ◦ Production worker Y arrives at Production station S ◦ Production contract for item Z by time T is posted Negative Events ◦ Product item X has been in transit to Store T for >15 minutes ◦ Subcomponent Y hasn’t arrived at the Production station by the ETA ◦ Delivery of contract Z has not taken place Sets of Events ◦ 5+ items of Product item type Y failed to arrive at destination ◦ Supplier Y was 5 mins late for 1 delivery, but made it early to the next ◦ Return rate on component Z exceeds SLA % © Copyright 2000-2012 TIBCO Software Inc.
  • SIGNIFICANT FEATURES OF THESE EVENTS Time Sensitivity ◦ A thief may leave the building at the same time as stolen product t ◦ A product should take 40 minutes to travel a given production line segment Distributed Event Sources ◦ A series of produced items fails at various QA stages, and their common attribute was a storage location ◦ Multiple suppliers for a subcomponent are reporting delivery delays ERP Supplier © Copyright 2000-2012 TIBCO Software Inc.
  • THE EVENT CLOUD Can we infer important business events by correlating events automatically + earlier, regardless of source / type? Personnel events RFID events Supply chain events Timeout events Delivery events 12 © Copyright 2000-2012 TIBCO Software Inc.
  • POWER OF EVENTS “The events we have access to are not always tailored to the problems we are trying to solve. Therefore, we need a technology that enables us to progress in stages. 1: The first stage is recognizing relevant patterns of events in the sources of events we do have access to and can monitor. 2: The second stage is aggregating information in those events to build up information that is needed to solve our problems.” © Copyright 2000-2012 TIBCO Software Inc.
  • EPTS ON DERIVED AND COMPOSITE EVENTS Derived event (also synthesized event): an event that is generated as a result of applying a method or process to one or more other events. ◦ Example: —A “Company B is a good buy” event may be derived from “Company B price has dropped where Company A price has risen within 5 minutes” ◦ Note: The absence of an event, say in a given time interval, can lead to a derived event for the missing event. Composite event: a derived, complex event that is created by combining base events using a specific set of event constructors ◦ Operators include disjunction, conjunction, sequence, etc. ◦ A composite event always includes the base (member) events from which it is derived. ◦ Note: A derived event may not be a composite event if its method of derivation lies outside a specified set of allowed constructors. Note: The term aggregate event is sometimes used for some forms of composite or derived event. © Copyright 2000-2012 TIBCO Software Inc.
  • EPTS ON COMPLEX EVENTS Complex event: an event that is an abstraction of other events ◦ Abstracted events are the members of the complex event ◦ Examples: —the 1929 stock market crash (an abstraction denoting many thousands of member events), —the 2004 Indonesian Tsunami (an abstraction of many natural events) , —a completed stock purchase (an abstraction of the events in a transaction to purchase the stock). —a successful on-line shopping cart checkout (an abstraction of shopping cart events on an on- line website). ◦ Note: a complex event denotes or signifies the set of its member events. This may or may not include references to individual member events. A composite event or a derived event is a complex event. The converses are not necessarily true. © Copyright 2000-2012 TIBCO Software Inc.
  • EPTS ON EVENT PROCESSING Event processing: computing that performs operations on events, including reading, creating, transforming and deleting events. ◦ Note: The overloaded meaning event object processing is intended in this context. Complex-event processing (CEP): Computing that performs operations on complex events, including reading, creating, transforming or abstracting them. ◦ Note: CEP results in creating complex events even if some or all of the source events are simple events © Copyright 2000-2012 TIBCO Software Inc.
  • ALTERNATIVE TERMS Event Cloud Personnel events RFID events Supply chain events Timeout events Delivery events Event Streams CEP (technology) applies pattern detection & filtering to the event clouds & streams and their histories Multiple modelling / execution paradigms are available for pattern detection Related terms: real-time analytics, continuous intelligence, event processing, … CEP © Copyright 2000-2012 TIBCO Software Inc.
  • WHAT DOES CEP SOLVE? … the “ Latency Problem“ warnings precede threats CEP allows fast Business Event responses to detected issues Potential Business Resulting Complex Event Measured Value Root Cause / Correlation Corrective Decision Made Action Taken Action time © Copyright 2000-2012 TIBCO Software Inc.
  • TOPICS Intro to Complex Event Processing IT perspective and Technologies Examples CEP Market, Tools Available © Copyright 2000-2012 TIBCO Software Inc.
  • USE CASE PATTERNS “Situational Awareness” “Sense and Respond” “Track and Trace” © Copyright 2000-2012 TIBCO Software Inc.
  • CEP’S ROLE IN I.T. © Copyright 2000-2012 TIBCO Software Inc.
  • COMPLEX EVENT PROCESSING Processing type: Decision Processing Event Processing -- TIBCO Reference Architecture by Alan Lundberg 2005 © Copyright 2000-2012 TIBCO Software Inc.
  • EVENT-DECISION ARCHITECTURE EVENT SOURCES EVENT PRE- Event-Decision Architecture PROCESSING EXTERNAL LEVEL ONE LEVEL TWO LEVEL THREE DISTRIBUTED Event Tracking & Situation Detection Impact Assessment LOCAL Refinement & Refinement & Predictive Analysis EVENT SERVICES User . Interface . EVENT . & Systems . PROFILES . . . EVENT + DATA STORE DATA BASES Historical Profiles & . LEVEL FOUR Data Patterns . OTHER DATA Process Refinement & Adaptation -- Adapted from JDL Data Fusion Model by Tim Bass Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001 © Copyright 2000-2012 TIBCO Software Inc.
  • EPTS FUNCTIONAL REFERENCE ARCHITECTURE -- EPTS Reference Archjitecture by EPTS RA Working Group editors Alex Alves, Adrian Paschke, Catherine Moxey, Paul Vincent Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, ACM, 2012 © Copyright 2000-2012 TIBCO Software Inc.
  • STANDARD OR SIMPLE EVENT PROCESSING Fixed to specific Synchronous event types Events new order fulfilment BPM: business SOA: process IT services Data Persistence (service) © Copyright 2000-2012 TIBCO Software Inc.
  • MULTIPLE SIMPLE EVENT PROCESSING new order fulfilmentnew supplier purchasingchange order rectification delivery billing Multiples of Multiples of Processes Services Multiples of Database Views © Copyright 2000-2012 TIBCO Software Inc.
  • MULTIPLE SIMPLE EVENT PROCESSING (THEORY) new order fulfilmentnew supplier purchasingchange order rectification delivery billing Multiples of Multiples of Processes Services Multiples of Database Views © Copyright 2000-2012 TIBCO Software Inc.
  • COMPLEX EVENT PROCESSING Asynchronous Flexible to any Events Continuous event types identification of event patterns business events source events CEP: complex event processing State store BPM: business SOA: IT process services Data Persistence (service) © Copyright 2000-2012 TIBCO Software Inc.
  • REQUIREMENTS FOR CEP TECHNOLOGY Access and Monitor Event Bus or Source the “Event Cloud” Low-latency reliable ◦ Instrumentation (for state changes) message delivery ◦ JMS, RV, MQ, TCP/IP, etc… ◦ Timers to detect lack of events Event and Data Model Match Patterns for information modeling (+ Apply Business Logic) History and State Temporal Model ◦ Detect events for determining ◦ Detect event patterns time-dependent info ◦ Maintain State and Facts over time Pattern Detection Model ◦ Update Detection algorithms as events change for recognizing patterns, defining actions © Copyright 2000-2012 TIBCO Software Inc.
  • CEP TECHNOLOGIES Types of CEP Processing Rule-Driven CEP Event Sources Rule Engine Rulebase Neural Net CEP Neural Net Network Event Bus Event Stream Query CEP Event Query Engine Stream Queries Consumers Orchestrated Event Stream CEP Flow Engine Flow Model © Copyright 2000-2012 TIBCO Software Inc.
  • CEP BEHAVIOR: STATE-ORIENTED New New Condition / Order New TimeEvent Order Order State 1 State 2 State Model © Copyright 2000-2012 TIBCO Software Inc.
  • STATE MODEL CEP AGENT FEATURES 1. Visual modeling metaphor ◦ State diagram / flow diagram is simple to follow 2. State / flow transitions can be time-related ◦ Can model missing events through time-outs etc © Copyright 2000-2012 TIBCO Software Inc.
  • CEP BEHAVIOR: RULE-ORIENTED New (Lots of New Order New Events) Order Order Complex Reaction Patterns Rule © Copyright 2000-2012 TIBCO Software Inc.
  • INFERENCE RULE CEP AGENT FEATURES 1. High performance pattern matching ◦ Rete (or other) algorithm determines what rules are executable based on underlying data changes 2. Declarative + Inferencing ◦ Rules defined in terms of classes: can be relevant for any # instances ◦ Rules’ actions can cause other rules to fire automatically 3. In-memory ◦ Limited only by JVM / process memory © Copyright 2000-2012 TIBCO Software Inc.
  • CEP BEHAVIOR: QUERY-ORIENTED New New (Lots of Order New Events) Order Order Complex Results Event Query Query © Copyright 2000-2012 TIBCO Software Inc.
  • QUERY CEP AGENT FEATURES 1. Common query language ◦ Usually SQL-based – widely used language ◦ May be in-memory, in-file or both ◦ Can include query optimizers 2. Continuous ◦ Extensions usually support time windows for the query to operate over © Copyright 2000-2012 TIBCO Software Inc.
  • CEP MEMORY ARCHITECTURES Usually in-memory for simple apps like trading Cache / DB memory stores used where failover / DR required, or scalable distributed processing Other options too (e.g. transactional shared memory systems) © Copyright 2000-2012 TIBCO Software Inc.
  • CEP TOOLS EVOLVING – THE “EVENT SERVER” Event events Transport & Distribution Pattern Detection Decisions Reactions Event & State store © Copyright 2000-2012 TIBCO Software Inc.
  • TOPICS Intro to Complex Event Processing IT perspective and Technologies Examples CEP Market, Tools Available © Copyright 2000-2012 TIBCO Software Inc.
  • EVENT-DRIVEN MARKETING USING CEP #Campaign and Offers pa 500 Campaigns + 4 M Offers #Target Customers >4M Transaction Volume >240 Msgs/Sec Revenue Growth Campaign Performance 1. Campaign 1: 235% Over Campaign Revenue Target 2. Campaign 2: 160% Over Campaign Revenue Target 3. Campaign 3 270% Over on Additional Spending Target 4. Campaign 4 250% Over Campaign Revenue Target Time to Market per Marketing Concept to Execution: Campaign Before: 3 – 6 Months After: 7 – 8 Days © Copyright 2000-2012 TIBCO Software Inc.
  • SHIPPING TRACK AND TRACE USING CEP Shipment container track and trace: ◦ Monitoring shipment milestones and invoking appropriate processes 54 million active milestones for shipments in-situ at any one time Implemented 3 times to compare technology approachesApproach Cost Cost vs JavaJava programming 64 1xBPM for events 400 6xEvent-driven rules 28 overall 0.5xEvent-driven rules 12 best case 0.2x © Copyright 2000-2012 TIBCO Software Inc.
  • MANUFACTURING QUALITY CONTROL USING CEP Test results data vs Error lifecycle Example: after a restart 100% testing? production station is Manual, 75% reliable for first 4 uses 100% reliable for the next 11 expensive tests days 80% reliable thereafter Manufacturing tasks per day 24,000 Number of process steps per unit Several 100s Solution results 10-15% avg. reduction in manual test costs “Significant reduction” in overall costs © Copyright 2000-2012 TIBCO Software Inc.
  • TOPICS Intro to Complex Event Processing IT perspective and Technologies Examples CEP Market, Tools Available © Copyright 2000-2012 TIBCO Software Inc.
  • FORRESTER NOTE ON BR PLATFORMS, JULY 2011 © Copyright 2000-2012 TIBCO Software Inc.
  • GARTNER PREDICTIONS, OCT 2011 © Copyright 2000-2012 TIBCO Software Inc.
  • CEP MARKET PLAYERS Different technologies for different CEP subtypes © Copyright 2000-2012 TIBCO Software Inc.
  • © Copyright 2000-2012 TIBCO Software Inc.