Getting Started in CEP: How to Build an Event Processing Application Tim Bass Founder and CTO SilkRoad

Our Agenda UNDER CONSTRUCTION www.thecepblog.com

UNDER CONSTRUCTION

Business Event Clouds Adapted from Roy Schulte, Gartner Web 2.0 & social nets Click streams RSS, Atom feeds Poll Web page scrapes Orders ATM transactions Credit card authorizations and transactions Airline reservations Telco Call Records Financial trade “ticks” RFID networks Bar code scans Device & application alerts e-mail, IM Insurance claims Traditional OLTP Sensor Networks Web and Net Other Transaction Streams Factory floor Micropayments Web commerce Inquiries Large Companies Experience 10 4 to 10 7 Business Events Per Second

Our Agenda Summary of CEP/EP and Example Use Cases How To Build an Event Processing Application Wrap Up

Summary of CEP/EP and Example Use Cases

How To Build an Event Processing Application

Wrap Up

A Perspective on CEP/EP

CEP is a Key Enabler in Many Kinds of BAM Courtesy of Roy Schulte, Gartner Process Monitoring Unstructured Data Situational Awareness Track and Trace Comparison Prediction Reality Monitor metrics Sequence stitching Alert on anomalies Drill down to detail records Fraud Hot prospect Customer churn Pattern Matching Model Construct Execute Monitor B2B

Monitor metrics

Sequence stitching

Alert on anomalies

Drill down to detail records

Fraud

Hot prospect

Customer churn

Summary of Event Processing Use Cases Cyber and algorithmic trading — Finance, energy Compliance reporting and monitoring — MiFID, RegNMS, SOX Adaptive CRM — Call centers and web clicks Financial controls — "Track and trace," , surveillance Fraud detection — Web commerce, AML, credit cards, telco Track and Trace – Patients, packages, pharmaceuticals Military — Situational awareness, intelligence Security and Networks — Intrusion detection (IDS) and NMS Sensor networks — RFID, GPS and others Transportation operations — Trucks, airlines, ships or trains Service Level Agreements (SLAs) – Telco, B2B, networks

Cyber and algorithmic trading — Finance, energy

Compliance reporting and monitoring — MiFID, RegNMS, SOX

Adaptive CRM — Call centers and web clicks

Financial controls — "Track and trace," , surveillance

Fraud detection — Web commerce, AML, credit cards, telco

Track and Trace – Patients, packages, pharmaceuticals

Military — Situational awareness, intelligence

Security and Networks — Intrusion detection (IDS) and NMS

Sensor networks — RFID, GPS and others

Transportation operations — Trucks, airlines, ships or trains

Service Level Agreements (SLAs) – Telco, B2B, networks

Fraud Detection Use Case Courtesy of TIBCO Software, 8th Annual Japan's International Banking & Securities System Forum, Tokyo, Japan, Feb 2007 Overall 100 Million Hits Handled Between 3PM – 4 PM Peak Approx. 250 Million Hits Per Day Across the Three Sites Approx. 12,000 Hits Per Second During Peak Period Across the Three Sites Session Info Three Server Farms ~600-700 Application Servers

Algorithmic Trading Use Case Courtesy of Progress Apama apama event scenario Multi-Dimensional Any-To-Any Correlation ! ! ! ! time real-time data streams S&P500 NYSE NASDAQ FTSE Example Event Scenario ( S&P moves by 2%) AND ( IBM price within 2% of $59.18 OR MSFT price within 1% of $43.49 ) WITHIN ( any 2 minute time period ) THEN ( BUY IBM & SELL MSFT )

MMO Engine (HERO) Monitoring Courtesy of Simutronics, StreamBase and SL Corporation

Key Take Aways Event processing applications can be: Complex, processing simple events into complex events and situations with real-time sense-and-respond capabilities False detections should be very low Accurate detection should be very high Low latency stream processing Event processing should be fast and latency should be low Analytics are relatively simple (compared to complex situation detection) Complex situation detection and stream processing are: Complimentary in many, if not most, future CEP/EP applications Work hand-in-hand to solve complex problems

Event processing applications can be:

Complex, processing simple events into complex events and situations with real-time sense-and-respond capabilities

False detections should be very low

Accurate detection should be very high

Low latency stream processing

Event processing should be fast and latency should be low

Analytics are relatively simple (compared to complex situation detection)

Complex situation detection and stream processing are:

Complimentary in many, if not most, future CEP/EP applications

Work hand-in-hand to solve complex problems

Our Agenda Summary of CEP/EP and Example Use Cases How To Build an Event Processing Application Wrap Up

Summary of CEP/EP and Example Use Cases

How To Build an Event Processing Application

Wrap Up

Key EDA Concepts Aggregate events across multiple sources; correlate with historical data, refine Detect events across the enterprise in real-time. Normalize and contextualize. Manage resources, processes; Invoke actions in real-time Analyze & Visualize Sense Respond ©2007, Tim Bass

How To Use a Functional Reference Architecture as a Framework for EP 24 EVENT SOURCES EXTERNAL . . . Visualization, BAM, User Interaction DISTRIBUTED LOCAL EVENT SERVICES . . EVENT PROFILES . . DATA BASES . . OTHER DATA Adapted by Tim Bass from the JDL: Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001 EVENT PRE-PROCESSING LEVEL ONE EVENT TRACKING Functional Reference Architecture for CEP/EP DB MANAGEMENT Historical Data Profiles & Patterns LEVEL TWO SITUATION DETECTION LEVEL THREE PREDICTIVE ANALYSIS LEVEL FOUR ADAPTIVE BPM

How To Know Where You Are Headed Multi-level inference in a distributed event-decision architectures User Interface (Dashboards, BAM, Visualization, Portals) Human visualization, monitoring, interaction and situation management Level 4 – Process Refinement (Adaptive BPM) Decide on control feedback, for example resource allocation, sensor and state management, parametric and algorithm adjustment Level 3 – Impact Assessment (Predictive Analytics) Impact assessment, i.e. assess intent on the basis of situation development, recognition and prediction Level 2 – Situation Refinement (Situational Detection) Identify situations based on sets of complex events, state estimation, etc. Level 1 – Event Refinement (Event Track and Trace) Identify events & make initial decisions based on association and correlation Level 0 – Event Preprocessing Cleansing, transformation of raw event data to produce semantically understandable data Level of Inference Low Med High

Multi-level inference in a distributed event-decision architectures

User Interface (Dashboards, BAM, Visualization, Portals)

Human visualization, monitoring, interaction and situation management

Level 4 – Process Refinement (Adaptive BPM)

Decide on control feedback, for example resource allocation, sensor and state management, parametric and algorithm adjustment

Level 3 – Impact Assessment (Predictive Analytics)

Impact assessment, i.e. assess intent on the basis of situation development, recognition and prediction

Level 2 – Situation Refinement (Situational Detection)

Identify situations based on sets of complex events, state estimation, etc.

Level 1 – Event Refinement (Event Track and Trace)

Identify events & make initial decisions based on association and correlation

Level 0 – Event Preprocessing

Cleansing, transformation of raw event data to produce semantically understandable data

How To Scope Your Application Determine Event Sources Do you have access to the events? What are the security requirements? Determine Event Transport(s) Services Publish and subscribe messaging? JMS? Feed backed transport such as RSS or ATOM Request / Reply (Polling) and SOA Determine Your Adaptation Requirements SNMP File adapters such as Unix syslog Database Low Med

Determine Event Sources

Do you have access to the events?

What are the security requirements?

Determine Event Transport(s) Services

Publish and subscribe messaging? JMS?

Feed backed transport such as RSS or ATOM

Request / Reply (Polling) and SOA

Determine Your Adaptation Requirements

SNMP

File adapters such as Unix syslog

Database

How To Think About EP Analytics Do You Need Edge Processing? Rule-based edge filtering? Event routing and distribution services? High performance hardware services? Determined Event Processing Requirements? Event stream processing using continuous queries? Filtering, transformation and routing? State management over long periods of time? Statistical methods to deal with uncertainty? Modelling and visualization Tools Low Med

Do You Need Edge Processing?

Rule-based edge filtering?

Event routing and distribution services?

High performance hardware services?

Determined Event Processing Requirements?

Event stream processing using continuous queries?

Filtering, transformation and routing?

State management over long periods of time?

Statistical methods to deal with uncertainty?

Modelling and visualization Tools

Events Sources (1/2) Devices as Event Sources Routers, hubs, switches, and similar network or telecommunication devices Network appliances designed for alerting, like a firewall, sniffer or an IDS Sensors and sensor networks including RFID readers Log Files as Event Sources Application log files including syslog () Other flat file event logs Databases as Event Sources Change (insert, update, or delete) to a row of the source table events Events captured in real time during the transaction Data Feeds as Event Sources Market data feeds News feeds (RSS / Atom) Other feeds from feed handlers

Devices as Event Sources

Routers, hubs, switches, and similar network or telecommunication devices

Network appliances designed for alerting, like a firewall, sniffer or an IDS

Sensors and sensor networks including RFID readers

Log Files as Event Sources

Application log files including syslog ()

Other flat file event logs

Databases as Event Sources

Change (insert, update, or delete) to a row of the source table events

Events captured in real time during the transaction

Data Feeds as Event Sources

Market data feeds

News feeds (RSS / Atom)

Other feeds from feed handlers

Events Sources (2/2) Message-Oriented Middleware (MOM) as Event Sources JMS TIBCO RV IBM MQ Analytics as Event Sources Rules-engines Statistical analytics (e.g. Bayesian inference engines) Artificial neural networks Filters and signal processors (e.g. Kalman filters) CEP and ESP engines (could also be or include one or more of the above) Process and Resource Management Systems as Event Sources BPM systems CRM systems Tealeaf adapter for Customer Experience Monitoring/Analytics. Users as Event Sources E-mail, SMTP IM Timers as Event Sources

Message-Oriented Middleware (MOM) as Event Sources

JMS

TIBCO RV

IBM MQ

Analytics as Event Sources

Rules-engines

Statistical analytics (e.g. Bayesian inference engines)

Artificial neural networks

Filters and signal processors (e.g. Kalman filters)

CEP and ESP engines (could also be or include one or more of the above)

Process and Resource Management Systems as Event Sources

BPM systems

CRM systems

Tealeaf adapter for Customer Experience Monitoring/Analytics.

Users as Event Sources

E-mail, SMTP

IM

Timers as Event Sources

Events Transformation Services XML Transformations XPath (including JXPath) XSLT XQuery XStream DOMToXML and XMLToDOM Scripting Transformations PHP PERL Regular Expressions (Regex) Event Processing Agent Transformations Event Processing Languages (EPLs) Other Transformations Cryptographic Transformations Encoding Transformations CVS Transformations Vendor Specific Transformations

XML Transformations

XPath (including JXPath)

XSLT

XQuery

XStream

DOMToXML and XMLToDOM

Scripting Transformations

PHP

PERL

Regular Expressions (Regex)

Event Processing Agent Transformations

Event Processing Languages (EPLs)

Other Transformations

Cryptographic Transformations

Encoding Transformations

CVS Transformations

Vendor Specific Transformations

Event Preprocessing and Track and Trace Use Case Example: Fraud Detection Financial services / ecommerce application Hundreds of web servers, many server farms Real-time click-stream processing to detection fraud How To Get Started? Passive hardware sniffing? Pre-process header and payload information? Pre-filter? Session vector management? Your Turn, You Be the Designer

Use Case Example: Fraud Detection

Financial services / ecommerce application

Hundreds of web servers, many server farms

Real-time click-stream processing to detection fraud

How To Get Started?

Passive hardware sniffing?

Pre-process header and payload information?

Pre-filter?

Session vector management?

Your Turn, You Be the Designer

Correlation Across Numerous Platforms The Heart of Complex Event Processing Use Case Example: Fraud Detection Financial services / ecommerce application Pre-processing (edge processing) “solved” Data normalized and cleansed for data integration Numerous fraud-detection and security platforms How To Get Started? Cross-platform correlation rules? Rules, Bayesian networks (BN) or neural nets (e.g.) Your Turn, You Be the Designer

Use Case Example: Fraud Detection

Financial services / ecommerce application

Pre-processing (edge processing) “solved”

Data normalized and cleansed for data integration

Numerous fraud-detection and security platforms

How To Get Started?

Cross-platform correlation rules?

Rules, Bayesian networks (BN) or neural nets (e.g.)

Your Turn, You Be the Designer

An EP Functional Reference Architecture 24 EVENT SOURCES EXTERNAL . . . Visualization, BAM, User Interaction DISTRIBUTED LOCAL EVENT SERVICES . . EVENT PROFILES . . DATA BASES . . OTHER DATA Adapted by Tim Bass from the JDL: Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001 EVENT PRE-PROCESSING LEVEL ONE EVENT TRACKING Functional Reference Architecture for CEP/EP DB MANAGEMENT Historical Data Profiles & Patterns LEVEL TWO SITUATION DETECTION LEVEL THREE PREDICTIVE ANALYSIS LEVEL FOUR ADAPTIVE BPM

Other Design Factors to Consider Historical Data, Databases, Patterns Regulatory Issues Privacy Issues Capacity Planning Visualization BAM Work Flow and BPM EDA and SOA Design Principles Intelligent agents as services (hint, you need both)

Historical Data, Databases, Patterns

Regulatory Issues

Privacy Issues

Capacity Planning

Visualization

BAM

Work Flow and BPM

EDA and SOA Design Principles

Intelligent agents as services (hint, you need both)

Wrap Up and Your Questions Event processing varies in complexity Simple event processing with stand alone systems High performance stream processing and edge services Distributed heterogeneous architecture integrating SOA and EDA CEP/EP vendors vary in capability Stand alone systems, purpose built applications EP integrated with adapters, analytics, dashboards Complete extensible enterprise integration So, Please Consider a Framework to Help You Plan Where You are Going

Event processing varies in complexity

Simple event processing with stand alone systems

High performance stream processing and edge services

Distributed heterogeneous architecture integrating SOA and EDA

CEP/EP vendors vary in capability

Stand alone systems, purpose built applications

EP integrated with adapters, analytics, dashboards

Complete extensible enterprise integration

So, Please Consider a Framework to Help You Plan Where You are Going

Thank You! Tim Bass