Optimizing Your SOA with Event Processing Tim Bass Principal Global Architect, Director Emerging Technologies Group TIBCO Software Inc.

TUCON Session Information Can your business sense, respond, analyze, interpret and respond to event-driven business situations in real time? In this session we explore how to get the maximum value from your SOA by understanding event-based architectures, complex event processing models, and how TIBCO BusinessEvents™ fits into your SOA and event-driven architecture (EDA) strategies.

Can your business sense, respond, analyze, interpret and respond to event-driven business situations in real time?

In this session we explore how to get the maximum value from your SOA by understanding event-based architectures, complex event processing models, and how TIBCO BusinessEvents™ fits into your SOA and event-driven architecture (EDA) strategies.

Our Agenda Events, Event Processing & EDA EDA, SOA, CEP & TIBCO BusinessEvents™ Wrap Up Question & Answers

Events, Event Processing & EDA

EDA, SOA, CEP & TIBCO BusinessEvents™

Wrap Up

Question & Answers

EDA Business Value Proposition Sense and respond to business events – both threats and opportunities to your enterprise – quickly.

Sense and respond to business events – both threats and opportunities to your enterprise – quickly.

What is an Event? An Event is a significant change in state. State 1 State 2 Event Can make your flight connection Cannot make your flight Connection

An Event is a significant change in state.

What is a Significant State Change? Source: Ranadiv é , V., The Power to Predict , 2006. Sense reality Compare reality with expectation Adapt model and response to new reality

Sense reality

Compare reality with expectation

Adapt model and response to new reality

Summary of Types of Events Normal event Arrival of your airline baggage on time Login attempt from known fraudulent IP address Anticipated “abnormal” event Penalty likely because of delayed or lost baggage We don’t know when baggage will be delayed, but we need to be prepared for that eventuality Unanticipated event Baggage conveyor system “frozen” due to unusually severe ice storm Airport workers union strike at major hub

Normal event

Arrival of your airline baggage on time

Login attempt from known fraudulent IP address

Anticipated “abnormal” event

Penalty likely because of delayed or lost baggage

We don’t know when baggage will be delayed, but we need to be prepared for that eventuality

Unanticipated event

Baggage conveyor system “frozen” due to unusually severe ice storm

Airport workers union strike at major hub

Detecting Situations from Events Anticipated event (pattern matching/detection) Specify pattern of the anticipated event and the appropriate response Unanticipated event (anomaly detection) Specify patterns of normality; event is deviation from pattern when reality doesn’t fit “normality” then alert business user.

Anticipated event (pattern matching/detection)

Specify pattern of the anticipated event and the appropriate response

Unanticipated event (anomaly detection)

Specify patterns of normality; event is deviation from pattern

when reality doesn’t fit “normality” then alert business user.

Event Processing Characteristics Asynchronous Timing : The timing of events are not controlled by the enterprise. Noise: External event data is noisy. Complex Event Processing: The significant state-change for the enterprise is detected by fusing data from multiple sources.

Asynchronous Timing : The timing of events are not controlled by the enterprise.

Noise: External event data is noisy.

Complex Event Processing: The significant state-change for the enterprise is detected by fusing data from multiple sources.

Managing Uncertainty Asynchronous Timing : Integrate request-response SOA with asynchronous EDA Noise: Manage uncertainty about errors; both false positives and false negatives. Multisensor Event Fusion: Extreme decoupling . Houston Denver Edmonton London Sydney NY, NY Trader Dashboards Risk Manager Houston Corporate VP, Risk Risk Management Dashboards Scheduler Dashboards

Asynchronous Timing :

Integrate request-response SOA with asynchronous EDA

Noise:

Manage uncertainty about errors; both false positives and false negatives.

Multisensor Event Fusion:

Extreme decoupling .

Is Your Enterprise Event-Driven? Does your enterprise monitor its external environment? Does your enterprise monitor its competitors? Government agencies? Do people in your enterprise correlate information from multiple sources? e.g., correlate power outage at a supplier’s factory with delivery deadlines for critical customers.

Does your enterprise monitor its external environment?

Does your enterprise monitor its competitors? Government agencies?

Do people in your enterprise correlate information from multiple sources? e.g., correlate power outage at a supplier’s factory with delivery deadlines for critical customers.

Is Your Enterprise Event-Driven? Are you expected to respond to events? An unexpected storm has just occurred in a region that will severely effect air traffic customers. Which scenario represents your business? The CEO does not expect the VP of Operations to report anything unless the CEO asks. The CEO expects VP of Operations to inform the CEO.

Are you expected to respond to events?

An unexpected storm has just occurred in a region that will severely effect air traffic customers.

Which scenario represents your business?

The CEO does not expect the VP of Operations to report anything unless the CEO asks.

The CEO expects VP of Operations to inform the CEO.

Key Take-Aways on Events Event Processing Characteristics: Sense and Respond : Respond quickly when reality deviates from expectation or plans. Asynchrony : The timing of events are not controlled by the enterprise. Global situational awareness: Can be achieved only by correlating multiple sources of data from outside the enterprise with enterprise data. Errors : Events are “noisy.”

Event Processing Characteristics:

Sense and Respond : Respond quickly when reality deviates from expectation or plans.

Asynchrony : The timing of events are not controlled by the enterprise.

Global situational awareness: Can be achieved only by correlating multiple sources of data from outside the enterprise with enterprise data.

Errors : Events are “noisy.”

What is an Event Driven Architecture? EDA is an architectural style that manages and executes rules of the form: WHEN reality deviates from expectations THEN update expectations and initiate response.

EDA is an architectural style that manages and executes rules of the form:

WHEN reality deviates from expectations

THEN update expectations and initiate response.

EDA Characteristics Aggregate events across multiple sources; compare reality with expectations Analyze Detect events across extended environment in real-time Sense Update expectations; Invoke distributed services in real-time Respond

Event Processing Reference Architecture 24 EVENT PRE-PROCESSING EVENT SOURCES EXTERNAL . . . LEVEL ONE EVENT TRACKING Visualization, BAM, User Interaction Event-Processing Architecture DB MANAGEMENT Historical Data Profiles & Patterns DISTRIBUTED LOCAL EVENT SERVICES . . EVENT PROFILES . . DATA BASES . . OTHER DATA LEVEL TWO SITUATION DETECTION LEVEL THREE PREDICTIVE ANALYSIS LEVEL FOUR ADAPTIVE BPM

Situational Awareness via Event Processing Multi-level inference in a distributed event-decision architectures User Interface (Dashboards, BAM, Visualization, Portals) Human visualization, monitoring, interaction and situation management Level 4 – Process Refinement (Adaptive BPM) Decide on control feedback, for example resource allocation, sensor and state management, parametric and algorithm adjustment Level 3 – Impact Assessment (Predictive Analytics) Impact assessment, i.e. assess intent on the basis of situation development, recognition and prediction Level 2 – Situation Refinement (Situational Detection) Identify situations based on sets of complex events, state estimation, etc. Level 1 – Event Refinement (Event Tracking) Identify events & make initial decisions based on association and correlation Level 0 – Event Preprocessing Cleansing of event-stream to produce semantically understandable data Level of Inference Low Med High

Multi-level inference in a distributed event-decision architectures

User Interface (Dashboards, BAM, Visualization, Portals)

Human visualization, monitoring, interaction and situation management

Level 4 – Process Refinement (Adaptive BPM)

Decide on control feedback, for example resource allocation, sensor and state management, parametric and algorithm adjustment

Level 3 – Impact Assessment (Predictive Analytics)

Impact assessment, i.e. assess intent on the basis of situation development, recognition and prediction

Level 2 – Situation Refinement (Situational Detection)

Identify situations based on sets of complex events, state estimation, etc.

Level 1 – Event Refinement (Event Tracking)

Identify events & make initial decisions based on association and correlation

Level 0 – Event Preprocessing

Cleansing of event-stream to produce semantically understandable data

Event Processing Characteristics Event Processing Agents, Sensors and ESB “System” can “learn” expectations from positive and negative examples Users can specify expectations using: SQL-like queries Fuzzy matches Statistical operators Regular expressions and rules CEP

Event Processing Agents, Sensors and ESB

“System” can “learn” expectations from positive and negative examples

Users can specify expectations using:

SQL-like queries

Fuzzy matches

Statistical operators

Regular expressions and rules

CEP

Our Agenda Events, Event Processing & EDA EDA, SOA, CEP & TIBCO BusinessEvents™ Wrap Up Question & Answers

Events, Event Processing & EDA

EDA, SOA, CEP & TIBCO BusinessEvents™

Wrap Up

Question & Answers

EDA and SOA SOA: Components are collaborators Accounting “client” calls a “sales pipeline expectation” method on a sales “service” which returns with a report SOA: Time of interaction determined by client SOA: Service protocols and schemas are well defined. Often transactional semantics. SOA: Units obtain global situational awareness by invoking multiple services.

SOA: Components are collaborators

Accounting “client” calls a “sales pipeline expectation” method on a sales “service” which returns with a report

SOA: Time of interaction determined by client

SOA: Service protocols and schemas are well defined. Often transactional semantics.

SOA: Units obtain global situational awareness by invoking multiple services.

EDA and SOA SOA: Request-response and transactions are often the appropriate pattern within the enterprise. SOA: Encapsulating existing capabilities (e.g., mainframe) with WS as required. EDA: Monitor and respond to events within and outside the enterprise by processing asynchronous events.

SOA: Request-response and transactions are often the appropriate pattern within the enterprise.

SOA: Encapsulating existing capabilities (e.g., mainframe) with WS as required.

EDA: Monitor and respond to events within and outside the enterprise by processing asynchronous events.

EDA Visualized Asynchronous, Strongly Decoupled, Not Orchestrated A A A A A A A A A A Message Message Message Message Message Message Message Message

“Traditional SOA” + EDA Visualized Synchronous & Asynchronous, Loosely Coupled & Strongly Decoupled, Managed, Orchestrated, Not Orchestrated, Consumer-Driven, Producer-Driven Not Orchestrated Messaging (EDA) Note: Request/Reply Implied in “ Orchestrated Messaging” Orchestrated Messaging (“Traditional” SOA) Message A A A A A A A A A A Message Message Message Message Message Message Message Message

Table of Characteristics SOA and EDA Asynchronous Event Triggers Synchronous Service Invocation Application Interaction Flow Control Faster Sense/Respond Service Component Reuse Primary Technical Goal Reduced Costs and Increased Visibility Reduce Costs and Time-to-Market Primary Business Goal One-to-One, One-to-Many, Many-to-Many One-to-One Process Communication Models Publish/Subscribe Orchestration Process Management Producer Consumer Process Trigger No Scheduler Scheduler Required Process Coordination Strongly Decoupled Loosely Coupled Application Interaction EDA SOA Architectural Characteristic

Table of Characteristics “New SOA” and EDA Asynchronous Event Triggers Synchronous & Asynchronous Service Invocation Application Interaction Flow Control Faster Sense/Respond Distributed Computing Primary Technical Goal Reduced Costs and Increased Visibility Reduced Costs, Time-to-Market, and Increased Visibility Primary Business Goal One-to-One, One-to-Many, Many-to-Many One-to-One, One-to-Many, Many-to-Many Process Communication Models Publish/Subscribe Orchestration & Pub/Sub Process Management Producer Consumer or Producer Process Trigger No Scheduler Scheduled & Unscheduled Process Coordination Strongly Decoupled Decoupled, Loosely Coupled Application Interaction EDA “ New SOA” Architectural Characteristic

FYI: Event Processing (EP) and CEP Visualized A A RE A A A A A A A Messages Messages Messages Messages Messages Message Messages Messages Many-to-One Asynchronous Events Processing with Rules Engine (RE)

SOA + EDA + CEP Visualized (with Rules Engine) A A A A A A A RE A A Message Message Message Message Message Messages Messages Message Synchronous & Asynchronous, Loosely Coupled & Decoupled, Managed, Orchestrated, Not Orchestrated, Consumer-Driven, Producer-Driven Orchestrated Messaging (“Traditional” SOA) Not Orchestrated Messaging (EDA) Messages Note: Request/Reply Implied in “ Orchestrated Messaging” Messages

Complex Event Processing " Events in several forms, from simple events to complex events, will become very widely used in business applications during 2004 through 2008 " --- Gartner July 2003 Situation Detection

Bloor Report on Event Processing Automated Operational Decisions Automated Predictive Decisions Human Predictive Decisions Human Operational Decisions Decision Latency Event Complexity Process Complexity Pattern Matching and Inferencing Anti-Money Laundering Credit-Card Fraud Exchange Compliance Database Monitoring Algorithmic Trading Trade Desk Monitoring Customer Interaction Order Routing RFID Tariff Look-Up Rail Networks Search & Rescue Baggage Handling Liquidity Management

A Business Optimization Perspective What Classes of Rule-Based Problems Do Businesses Need to Solve? Rule-Based Pattern Recognition Anomaly Detection Track and Trace Monitoring (BAM) Dynamic Resource Allocation Adaptive Resource Allocation Constraint Satisfaction (CSP) Dynamic CSP Adaptive Marketing Dynamic CRM Fault Management Impact Assessment Fraud Detection Intrusion Detection Fault Detection Rule-Based Access Control Exception Management Compliance Work Flow Risk Management Fault Analysis Impact Assessment Example Event Processing Applications Detection Prediction Scheduling

Pattern Recognition

Anomaly Detection

Track and Trace

Monitoring (BAM)

Dynamic Resource Allocation

Adaptive Resource Allocation

Constraint Satisfaction (CSP)

Dynamic CSP

Adaptive Marketing

Dynamic CRM

Fault Management

Impact Assessment

Fraud Detection

Intrusion Detection

Fault Detection

Rule-Based Access Control

Exception Management

Compliance Work Flow

Risk Management

Fault Analysis

Impact Assessment

TIBCO BusinessEvents™ Solutions Overview BusinessEvents™ Solutions Space Data: Events & Databases -Real-Time & Historical Data Models: Statistical Financial Optimization Comms: Pub/Sub Messaging Queues Topics UIs Knowledge: Facts & Rules

TIBCO BusinessEvents™ Overview High performance, low latency business rules engine. Top down business process modeling. Real-time event processing. Cross-application and cross-process integration. Analytical and predictive models . Modeling Tools, Statefulness, Business Rules and Process Integration UML Conceptual UML State Business Rules Business Users Event Analyzer

High performance, low latency business rules engine.

Top down business process modeling.

Real-time event processing.

Cross-application and cross-process integration.

Analytical and predictive models .

TIBCO BusinessEvents™ Overview Collection, Normalization Metric of Managed Objects, Normalized Non-Contextual Events Metadata Repository Semantic Model Events Rules Design Environment State Model Event Management, Correlation, Aggregation, Inference and Analysis Correlated, Analyzed, Contextual Dialogue Events Rules, Knowledge, Patterns, Models Visualization, Reporting, Alert Management Application Interface Feeds Visualization: Detection Metrics Agents Synthetic Warehouse Visualization: Process View Dialogue Manager Inference Engine FDS/IDS Logfiles Edge Devs Sensors

BusinessEvents™ Components Enterprise Metadata (Concepts, Properties, State Models, XML Schemas, Business Rules) BusinessEvents Workbench (Designtime) BusinessEvents Engine (Runtime) Business User Interface Business User Language Decision Tables Runtime Viewer Management Server

Runtime – BusinessEvents™ Engine Engine Inference Engine forward chaining – optimized Rete based rule inferencing history of objects – calculation of real-time time-series persistence – virtual memory, 100% failsafe performance – 10s of 1000s of rules per second Models ontology - objects, events, inheritance, relationships, properties state model – objects life cycle, event patterns, time, alerts, reports kpi model - real-time calculation, thresholds / alerts Monitor and Management Channels Embedded DB

forward chaining – optimized Rete based rule inferencing

history of objects – calculation of real-time time-series

persistence – virtual memory, 100% failsafe

performance – 10s of 1000s of rules per second

ontology - objects, events, inheritance, relationships, properties

state model – objects life cycle, event patterns, time, alerts, reports

kpi model - real-time calculation, thresholds / alerts

Standard TIBCO Design, Deploy, Run Model

CEP Use Case Illustrated

CEP Use Case Illustrated

Wrap Up Your enterprise is already event-driven SOA, EDA and CEP are both critical and complimentary technologies So, an event processing strategy is important to help you achieve maximum value our of your SOA design and implementation TIBCO BusinessEvents™ helps you sense, analyze, interpret and respond to event-driven business situations in real time.

Your enterprise is already event-driven

SOA, EDA and CEP are both critical and complimentary technologies

So, an event processing strategy is important to help you achieve maximum value our of your SOA design and implementation

TIBCO BusinessEvents™ helps you sense, analyze, interpret and respond to event-driven business situations in real time.

Q & A ?

?

Thank You! Tim Bass, CISSP Principal Global Architect, Director TIBCO Software Inc.

Appendix: SOA (1.0) Defined** SOA is a application architecture in which all functions, or services, are defined using a description language and have invokable interfaces that are called to perform business processes. Each interaction is independent of each and every other interaction and the interconnect protocols of the communicating devices (i.e., the infrastructure components that determine the communication system do not affect the interfaces). Because interfaces are platform-independent, a client from any device using any operating system in any language can use the service. http://www.webopedia.com/TERM/S/Service_Oriented_Architecture.html **Please Note: Gartner Principal Analysts now say that SOA 2.0 ~ “distributed computing services w/interfaces”

SOA is a application architecture in which all functions, or services, are defined using a description language and have invokable interfaces that are called to perform business processes.

Each interaction is independent of each and every other interaction and the interconnect protocols of the communicating devices (i.e., the infrastructure components that determine the communication system do not affect the interfaces).

Because interfaces are platform-independent, a client from any device using any operating system in any language can use the service.

http://www.webopedia.com/TERM/S/Service_Oriented_Architecture.html

Appendix: EDA Defined EDA is an enterprise software infrastructure model in which events trigger the real-time exchange of messages between independent software applications. EDA relies on an event-processing agent that detects events across an enterprise and, using a push approach, notifies all of the other software applications that need to be notified of the change in data, all at the same time. For example: the e-commerce Web site of an enterprise receives an order for a product, completing a business event. An event agent detects this transaction and simultaneously notifies all other applications in the enterprise that need to know about the order, which can include such aspects as an inventory database, accounts receivable software, customer service applications, marketing and advertising monitors, and shipping software. http://www.webopedia.com/TERM/E/EDA.html

EDA is an enterprise software infrastructure model in which events trigger the real-time exchange of messages between independent software applications.

EDA relies on an event-processing agent that detects events across an enterprise and, using a push approach, notifies all of the other software applications that need to be notified of the change in data, all at the same time.

For example: the e-commerce Web site of an enterprise receives an order for a product, completing a business event. An event agent detects this transaction and simultaneously notifies all other applications in the enterprise that need to know about the order, which can include such aspects as an inventory database, accounts receivable software, customer service applications, marketing and advertising monitors, and shipping software.

http://www.webopedia.com/TERM/E/EDA.html