Adding Rules to Improve Flexibility and Effectively Manage Complex Events Tim Bass, CISSP Principal Global Architect TIBCO Software Inc. Acknowledgement: Emerging Technology Group at TIBCO Software

Our Agenda Discuss a Few Classes of Complex Business Problems Provide a Brief Overview of Complex Event Processing Summarize TIBCO BusinessEvents™ Illustrate Rules in a Simple Fraud Detection Use Case Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case Wrap Up

Discuss a Few Classes of Complex Business Problems

Provide a Brief Overview of Complex Event Processing

Summarize TIBCO BusinessEvents™

Illustrate Rules in a Simple Fraud Detection Use Case

Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case

Wrap Up

How Would We Solve These Problems? Fraud Detection Millions of users logging in to access their financial information IP addresses change between logins, maybe even during login How do you identify known patterns of “suspicious” behavior? Train Scheduling Trains require critical resources (crew, terminals, track networks) at critical times Train schedules are only ETAs, which leads to misallocation of the above– How do we do this better? School Attendance Thousands of students, hundreds of schools and districts How do you “call home” in case of emergency? How do you do this elegantly and in a scaleable fashion?

Fraud Detection

Millions of users logging in to access their financial information

IP addresses change between logins, maybe even during login

How do you identify known patterns of “suspicious” behavior?

Train Scheduling

Trains require critical resources (crew, terminals, track networks) at critical times

Train schedules are only ETAs, which leads to misallocation of the above– How do we do this better?

School Attendance

Thousands of students, hundreds of schools and districts

How do you “call home” in case of emergency?

How do you do this elegantly and in a scaleable fashion?

A Deeper Look into Events … What Do We Really Have Here? Positive Events User X logs in to a financial website from a given IP Address Train Y arrives at a terminal Student Z calls in sick Negative Events User X has been inactive for 5 minutes Train Y hasn’t arrived at the terminal by the ETA Student Z hasn’t shown up yet Sets of Events User X unsuccessfully logs in 3 consecutive times Train Y was 5 mins late at one stop, but made it early to the next Some students haven’t shown up, and their bus has reported a breakdown

Positive Events

User X logs in to a financial website from a given IP Address

Train Y arrives at a terminal

Student Z calls in sick

Negative Events

User X has been inactive for 5 minutes

Train Y hasn’t arrived at the terminal by the ETA

Student Z hasn’t shown up yet

Sets of Events

User X unsuccessfully logs in 3 consecutive times

Train Y was 5 mins late at one stop, but made it early to the next

Some students haven’t shown up, and their bus has reported a breakdown

A Deeper Look into Events … What Do We Really Have Here? - Continued Time Sensitivity A user doesn’t usually log in from different Internet addresses with seconds between attempts – fraudsters do A train should take 40 minutes to travel a given track segment Distributed Event Sources A student didn’t sign-in at school, and her bus reported a breakdown a mile away Successful login attempts coming from different continents How do we interact with and process this Event Cloud?

Time Sensitivity

A user doesn’t usually log in from different Internet addresses with seconds between attempts – fraudsters do

A train should take 40 minutes to travel a given track segment

Distributed Event Sources

A student didn’t sign-in at school, and her bus reported a breakdown a mile away

Successful login attempts coming from different continents

How do we interact with and process this Event Cloud?

Our Agenda Discuss a Few Classes of Complex Business Problems Provide a Brief Overview of Complex Event Processing Summarize TIBCO BusinessEvents™ Illustrate Rules in a Simple Fraud Detection Use Case Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case Wrap Up

Discuss a Few Classes of Complex Business Problems

Provide a Brief Overview of Complex Event Processing

Summarize TIBCO BusinessEvents™

Illustrate Rules in a Simple Fraud Detection Use Case

Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case

Wrap Up

Need for Complex Events Processing (CEP) “ The events we have access to are not always tailored to the problems we are trying to solve. Therefore, we need a technology that enables us to progress in stages. The first stage is recognizing relevant patterns of events in the sources of events we do have access to and can monitor. The second stage is aggregating information in those events to build up information that is needed to solve our problems.” Dr. David Luckham - Stanford University Author, The Power of Events

“ The events we have access to are not always tailored to the problems we are trying to solve. Therefore, we need a technology that enables us to progress in stages.

The first stage is recognizing relevant patterns of events in the sources of events we do have access to and can monitor.

The second stage is aggregating information in those events to build up information that is needed to solve our problems.”

Dr. David Luckham - Stanford University Author, The Power of Events

What Is CEP? “ recognizing relevant patterns of events…” User X unsuccessfully logs in 3 consecutive times Train Y was 5 minutes late at one stop, but made it early to the next Some students haven’t shown up, and their bus has reported a breakdown “ aggregating information in those events to build up useful information…” User X is behaving suspiciously Train Y will likely make the next stop earlier than planned Expect these students to be late CEP Vision Provide a technology to detect various business conditions by monitoring a flow of events and recognizing patterns as they occur Aggregate and correlate these events into higher level “business events” that can be used to trigger business process to handle the various detected conditions

“ recognizing relevant patterns of events…”

User X unsuccessfully logs in 3 consecutive times

Train Y was 5 minutes late at one stop, but made it early to the next

Some students haven’t shown up, and their bus has reported a breakdown

“ aggregating information in those events to build up useful information…”

User X is behaving suspiciously

Train Y will likely make the next stop earlier than planned

Expect these students to be late

CEP Vision

Provide a technology to detect various business conditions by monitoring a flow of events and recognizing patterns as they occur

Aggregate and correlate these events into higher level “business events” that can be used to trigger business process to handle the various detected conditions

What Does CEP Give You? Ability to observe & recognize event patterns Ability to aggregate event patterns into higher level event structures Ability to correlate / match events to business objects Ability to take action - process and drive business object state Ability to model process / state based timing expectations (e.g. timeouts / lack of event support)

Ability to observe & recognize event patterns

Ability to aggregate event patterns into higher level event structures

Ability to correlate / match events to business objects

Ability to take action - process and drive business object state

Ability to model process / state based timing expectations (e.g. timeouts / lack of event support)

More Specifically … React and Predict Business Situations in Real-time Alert your resources to be ready at the next stop Train Y will likely make the next stop earlier than planned Alert the school and emergency contacts Expect these students to be late Issued an automated challenge-response to user User X is behaving suspiciously (medium likelihood of fraud) Investigate for fraud manually User X is behaving suspiciously (high likelihood of fraud) Resulting Situation-Decision Detected Business Situation

Why CEP? Can I solve these business problems with a database? Database Triggers CEP When Situation Then Reaction Localized Event Cloud. Rigid Schemas Non Intuitive and Static Relationships. Lack of Temporal Aspects. Point in Time. Distributed Configuration Management Global Event Cloud. Flexible, Dynamic Schemas Intuitive, Static and Dynamic Relationships. Temporal Reasoning. Points Across Time. Centralized Configuration Management Integration Backbone

Localized Event Cloud.

Rigid Schemas

Non Intuitive and Static Relationships.

Lack of Temporal Aspects.

Point in Time.

Distributed Configuration Management

Global Event Cloud.

Flexible, Dynamic Schemas

Intuitive, Static and Dynamic Relationships.

Temporal Reasoning.

Points Across Time.

Centralized Configuration Management

Details of a CEP Engine Has Access to the Event Cloud JMS, RV, SmartSockets, TCP/IP, etc… Timers [Lack of Events] Applies Business Logic and Intelligence Rule Based Systems When {condition} => Then {action} [Rules] Optimized Condition Checking [RETE algorithm] Maintains State and Facts [Working Memory] Executes Rules based on addition, removal, modification of Facts [Forward/Backward Chaining]

Has Access to the Event Cloud

JMS, RV, SmartSockets, TCP/IP, etc…

Timers [Lack of Events]

Applies Business Logic and Intelligence

Rule Based Systems

When {condition} => Then {action} [Rules]

Optimized Condition Checking [RETE algorithm]

Maintains State and Facts [Working Memory]

Executes Rules based on addition, removal, modification of Facts [Forward/Backward Chaining]

Our Agenda Discuss a Few Classes of Complex Business Problems Provide a Brief Overview of Complex Event Processing Summarize TIBCO BusinessEvents™ Illustrate Rules in a Simple Fraud Detection Use Case Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case Wrap Up

Discuss a Few Classes of Complex Business Problems

Provide a Brief Overview of Complex Event Processing

Summarize TIBCO BusinessEvents™

Illustrate Rules in a Simple Fraud Detection Use Case

Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case

Wrap Up

What is TIBCO BusinessEvents? At its core, BE is a Rules Inference Engine: Receives events Correlates events Applies rules Can generate internal events (which could trigger more rules) Can send events out Also includes State Machine Conceptual Model

At its core, BE is a Rules Inference Engine:

Receives events

Correlates events

Applies rules

Can generate internal events (which could trigger more rules)

Can send events out

Also includes

State Machine

Conceptual Model

What is TIBCO BusinessEvents? Introduces a high performance, low latency rules and policy engine for enterprise messaging customers. Introduces real-time operational process performance and decision support in market leading TIBCO Enterprise Integration Platform. Introduces cross-application decision platform for business users in large and medium scale SOA and EDA initiatives. Introduces semantic models to compliment the high performance XML stack. Introduces a model and an inference engine to model complex, dynamic processes such as “complex order brokering”, “dynamic resource management”, “fraud detection”. Introduces a high performance “model to code” approach for building event driven applications. Enterprise Metadata (UML, XSD, XSLT, WSDL, Business Rules and Process Flow) UML Conceptual UML State Business Rules Business Users Event Analyzer

Introduces a high performance, low latency rules and policy engine for enterprise messaging customers.

Introduces real-time operational process performance and decision support in market leading TIBCO Enterprise Integration Platform.

Introduces cross-application decision platform for business users in large and medium scale SOA and EDA initiatives.

Introduces semantic models to compliment the high performance XML stack.

Introduces a model and an inference engine to model complex, dynamic processes such as “complex order brokering”, “dynamic resource management”, “fraud detection”.

Introduces a high performance “model to code” approach for building event driven applications.

Our Agenda Discuss a Few Classes of Complex Business Problems Provide a Brief Overview of Complex Event Processing Summarize TIBCO BusinessEvents™ Illustrate Rules in a Simple Fraud Detection Use Case Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case Wrap Up

Discuss a Few Classes of Complex Business Problems

Provide a Brief Overview of Complex Event Processing

Summarize TIBCO BusinessEvents™

Illustrate Rules in a Simple Fraud Detection Use Case

Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case

Wrap Up

Case Study: Fraud Detection Identify user logins from various Internet locations within a given time window. Identity user logins from different IP addresses within a given time window with delayed input (within the window.) Identify user login attempts from different geographic locations in less than a feasible or computable travel time between the locations. Identity multiple user login attempts and/or successes from a single Internet address within a given time window. Classify user behavior based on a profile and detect anomalies.

Identify user logins from various Internet locations within a given time window.

Identity user logins from different IP addresses within a given time window with delayed input (within the window.)

Identify user login attempts from different geographic locations in less than a feasible or computable travel time between the locations.

Identity multiple user login attempts and/or successes from a single Internet address within a given time window.

Classify user behavior based on a profile and detect anomalies.

Fraud Detection: Technical Requirements Acquire and correlate real time data Acquire and correlate static data from DB2 and Oracle TIBCO must be able handle 20,000 transaction per second Detect and alert fraud outlined in use cases Send fraud alert multi-distribution channels such as SMTP, e-mail and wireless Demonstrate that data can be aggregated from multiple sites. Millions of Users

Acquire and correlate real time data

Acquire and correlate static data from DB2 and Oracle

TIBCO must be able handle 20,000 transaction per second

Detect and alert fraud outlined in use cases

Send fraud alert multi-distribution channels such as SMTP, e-mail and wireless

Demonstrate that data can be aggregated from multiple sites.

Millions of Users

How BusinessEvents Helped Detection Use Cases => Temporal Rules operating on Log On Events correlated to User Concept Instances

Detection Use Cases => Temporal Rules operating on Log On Events correlated to User Concept Instances

How BusinessEvents Helped Multiple Engines deployed and Embedded Object Database used for memory management Millions of Users Channels and Events Send fraud alert via multi-distribution channels such as SMTP and wireless Modeled using Rules, Events and Concepts Detect and alert fraud outlined in use cases Plugging into the “Event Cloud” in a non-intrusive manner Demonstrate that data can be aggregated from multiple sites Native product support for basic DB Access Acquire and correlate static data from DB2 and Oracle “ Noise filtering”, messaging, optimized RETE network, and model- to-code paradigm TIBCO solution must be able handle 20,000 transaction per second Used Provided RV and Custom TCP/IP Channels Acquire and correlate real time data How TIBCO Implemented What Our Customer Required

Our Agenda Discuss a Few Classes of Complex Business Problems Provide a Brief Overview of Complex Event Processing Summarize TIBCO BusinessEvents™ Illustrate Rules in a Simple Fraud Detection Use Case Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case Wrap Up

Discuss a Few Classes of Complex Business Problems

Provide a Brief Overview of Complex Event Processing

Summarize TIBCO BusinessEvents™

Illustrate Rules in a Simple Fraud Detection Use Case

Discuss a More Complex Dynamic Resource Allocation Problem – a Train Scheduling Use Case

Wrap Up

A TIBCO Customer Case Study: Dynamic Resource Management Each train demands the 5 critical resources over time For a train plan to be effective, all trains must have all resources allocated As the train schedule moves, the resource demand moves Time  Train Schedule Network Capacity Allocation Car Assignment Terminal Resource Allocation Locomotive Assignment Crew Assignment  Train Demand Dimension   Resource Supply Dimension 

Each train demands the 5 critical resources over time

For a train plan to be effective, all trains must have all resources allocated

As the train schedule moves, the resource demand moves

Dynamic Resource Management: Functional Requirements Provide a Train Lineup with dynamic updating of train schedules Associate trains on Train Lineup with crews Improve train movement projections. Detect opportunities to improve quality of decisions. Assess the quality of the Operating Train Plan.

Provide a Train Lineup with dynamic updating of train schedules

Associate trains on Train Lineup with crews

Improve train movement projections.

Detect opportunities to improve quality of decisions.

Assess the quality of the Operating Train Plan.

Dynamic Resource Management: Technical Requirements Model and Manage Hundreds of Data Values and Relationships Between Individual Crews, Trains, Terminals, and Network Segments, Points, and other entities Model and Manage Hundreds of Rules depending on above relationships and data values Enable Dynamic Allocation of hundreds of Resources separated by hundreds of miles of track

Model and Manage Hundreds of Data Values and Relationships Between Individual Crews, Trains, Terminals, and Network Segments, Points, and other entities

Model and Manage Hundreds of Rules depending on above relationships and data values

Enable Dynamic Allocation of hundreds of Resources separated by hundreds of miles of track

How Did BusinessEvents Help? A State Machine to Model and Monitor the Train State Machine to model and monitor a train’s progress over a route => Automatically Converted to Rules by Deployment

State Machine to model and monitor a train’s progress over a route

=> Automatically Converted to Rules by Deployment

How Did BusinessEvents Help? A Concept Diagram to Model Relationships

How Did BusinessEvents Help? Custom Visualization with TIBCO General Interface AJAX BASED IDE FOR EVENT & SITUATION VISUALIZATION (RICH CLIENT WEB APPLICATION)

How Did BusinessEvents Help? TIBCO BusinessEvents for Robust Rules Processing Managed spatially distributed resources Spatially partition allocation management One instance of BusinessEvents per train or track TIBCO EMS as Communications Backbone TIBCO General Interface for Rich Web Applications

TIBCO BusinessEvents for Robust Rules Processing

Managed spatially distributed resources

Spatially partition allocation management

One instance of BusinessEvents per train or track

TIBCO EMS as Communications Backbone

TIBCO General Interface for Rich Web Applications

Wrap Up How Did BusinessEvents Help? Provides the Capability for Our Customers to Solve and Manage Their Most Challenging Business Problems

Thank You! Tim Bass, CISSP Principal Global Architect [email_address] Complex Event Processing at TIBCO With BusinessEvents™