Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection

Tim Bass, CISSP Director, Principal Global Architect Emerging Technologies Group Complex Event Processing (CEP) for Next-Generation Security Event Management, Fraud and Intrusion Detection April 17, 2007 (First Draft) London

Our Agenda

Brief Overview of TIBCO Software Inc.

PredictiveBusiness® and CEP

SEM, FDS and IDS Reference Architecture

Solutions Architecture and Case Study

Wrap Up & Open Discussion

Who We Are and What We Do We help our customers… Improve operational visibility, collaboration and ability to be proactive Increase operational efficiency and effectiveness Accelerate projects, initiatives and go-to-market cycles A leading provider of business integration and process management software.

How TIBCO Delivers for Customers Accelerate projects, initiatives, and go-to-market cycles Increase operational efficiency and effectiveness. Improve operational visibility, security, collaboration and responsiveness

TIBCO is Trusted by Thousands of Companies

47 of the World’s 100 Largest Companies are TIBCO Customers

* By annual revenues except for investment banking which is measured by assets Retail Banking — 17 of top 20 Consumer Package Goods — 5 of top 10 Energy — 5 of top 10 Hi-Tech Manufacturing — 15 of top 20 Investment Banking — 9 of top 10 Manufacturing (non High-tech) — 5 of top 10 Pharmaceutical — 6 of top 10 Telecommunications — 8 of top 10 Transportation — 4 of top 10

TIBCO History and Acquisitions IPO 1999 eXtensibility InConcert Staffware TIBCO Today Teknekron 2000 2002 2001 2003 2004 2005 2005

1,600+ employees

Consistently profitable

Worldwide presence

Recognized market leader

2500+ customers

Acquired by Reuters Est. 1980s Palo Alto Campus Est. 1997 2004

TIBCO Runs a Strong and Viable Business 14 consecutive quarters of yr/yr total revenue growth

$284M USD invested in R&D in past 4 years

$540M USD in cash + short term investments in the bank and growing

Market cap of $1.9 billion (USD)

Revenue Numbers FY 2004 – 2006 (in thousands of dollars) 15.8% $61,060 $73,715 $387,220 FY2004 16.4% $73,127 $67,081 $445,910 FY2005 16.6% $85,923 $90,558 $517,279 FY 2006 R&D SPEND AS A % OF REVENUE R&D SPEND PRE-TAX PROFIT REVENUE

Our Agenda

PredictiveBusiness TM Source: Ranadiv é , V., The Power to Predict , 2006.

Complex Event Processing " Events in several forms, from simple events to complex events, will become very widely used in business applications during 2004 through 2008 " --- Gartner July 2003

What is Complex Event Processing? Detecting Threats and Opportunities with PredictiveBusiness®

When Do You Need to Think About CEP?

“ CEP applies to a very broad spectrum of challenges in information systems. A short list includes:”

Business process automation

Computer systems to automate scheduling and control network-based processes and processing

Network monitoring and performance prediction

Detection intrusion, fraud and other network attacks .

The Power of Events , Addison Wesley, ISBN: 0-201-72789-7, 2002

Bloor Report on Event Processing Event Processing and Decision Making Automated Operational Decisions Automated Predictive Decisions Human Predictive Decisions Human Operational Decisions Decision Latency Event Complexity Process Complexity Pattern Matching and Inferencing Anti-Money Laundering Credit-Card Fraud Exchange Compliance Database Monitoring Algorithmic Trading Trade Desk Monitoring Customer Interaction Order Routing RFID Tariff Look-Up Rail Networks Search & Rescue Baggage Handling Liquidity Management

Our Agenda

Firewalls, Stand-Alone or Purpose-Built Fraud and Intrusion Detection Systems, Cryptography, Access Control, are Simply Not Sufficient.

Malicious Users are Using Legitimate Internet Application Protocols, such as HTTP, HTTPS and SOAP to Defraud Businesses.

A 2006 CyberSource reports that $2,800,000,000 (2.8B USD) was lost to on-line fraud in the US and Canada in 2005.

eCommerce online fraud continues to grow (US and Canada) at a 20% annual rate.

Risk for international transactions is 3 times the average risk.

Industry and Business Drivers A Sample of the Problems with Network Security and Fraud Detection

Rapidly detect threats with a low rate of false alarms and a high level of situational detection confidence …

Detection-Oriented Systems - Design Goals What are the overall design goals for detection systems? (Illustrative Purposes Only)

Classification of Intrusion and Fraud Detection Systems Traditional View Before Data Fusion Approach to FDS and IDS Distributed Fraud and Intrusion Detection Systems, Logs Detection Approach Systems Protected Architecture Data Sources Analysis Timing Detection Actions HIDS NIDS Hybrid Audit Logs Net Traffic System Stats Real Time Data Mining Anomaly Detection Signature Detection Centralized Distributed Active Passive Agent Based Security “Stovepipes” Centralized

Intrusion Detection and Data Fusion (2000) Next-Generation Intrusion Detection Systems Source: Bass, T., CACM, 2000

PredictiveBusiness TM

A Business Optimization Perspective What Classes of Rule-Based Problems Do Businesses Need to Solve? Rule-Based

Pattern Recognition

Anomaly Detection

Track and Trace

Monitoring (BAM)

Dynamic Resource Allocation

Adaptive Resource Allocation

Constraint Satisfaction (CSP)

Dynamic CSP

Adaptive Marketing

Dynamic CRM

Fault Management

Impact Assessment

Detection Prediction Scheduling

Fraud Detection

Intrusion Detection

Fault Detection

Rule-Based Access Control

Exception Management

Compliance Work Flow

Risk Management

Fault Analysis

Impact Assessment

Example PredictiveBusiness® Applications

Emerging Event-Decision Architecture Customer Profiles Purpose-Built Analytics Secure, Distributed Messaging Backbone Internet/Extranet Sensors Human Sensors Edge/POC Sensors Operations Center Other Reference Data Rule-Based Event Processors

Complex Event Processing Reference Architecture Next-Generation Functional Architecture for Fraud and Intrusion Detection 24 EVENT PRE-PROCESSING EVENT SOURCES EXTERNAL . . . LEVEL ONE EVENT TRACKING Visualization, BAM, User Interaction CEP Reference Architecture DB MANAGEMENT Historical Data Profiles & Patterns DISTRIBUTED LOCAL EVENT SERVICES . . EVENT PROFILES . . DATA BASES . . OTHER DATA LEVEL TWO SITUATION DETECTION LEVEL THREE PREDICTIVE ANALYSIS LEVEL FOUR ADAPTIVE BPM

CEP – Situation Detection Hierarchy 22 Adapted from: Waltz, E. & Llinas, J., Multisensor Data Fusion, 1990 Impact Assessment Situational Assessment Relationship of Events Identify Events Location, Times and Rates of Events of Interest Existence of Possible Event of Interest Data/Event Cloud Analysis of Situation & Plans Contextual and Causal Analysis, Rules Causal Analysis, Bayesian Belief Networks, Rules, NNs, Correlation, State Estimation, Classification Use of Distributed Sensors for Estimations Raw Sensor Data (Passive and Active) HIGH LOW MED

CEP High Level Architecture 22 Adapted from: Engelmore, R. S., Morgan, A.J., & and Nii, H. P., Blackboard Systems, 1988 & Luckham, D., The Power of Events, 2002 EVENT CLOUD (DISTRIBUTED DATA SET) KS KS KS KS KS KS KS KS KS KS KS KS KS KS

Sensors

Systems that provide data and events to the inference models and humans

Actuators

Systems that take action based on inference models and human interactions

Knowledge Processors

Systems that take in data and events, process the data and events, and output refined, correlated, or inferred data or events

HLA - Knowledge Sources KS KS KS

Complex Event Processing Reference Architecture Next-Generation Functional Architecture for Fraud and Intrusion Detection 24 EVENT PRE-PROCESSING EVENT SOURCES EXTERNAL . . . LEVEL ONE EVENT TRACKING Visualization, BAM, User Interaction CEP Reference Architecture DB MANAGEMENT Historical Data Profiles & Patterns DISTRIBUTED LOCAL EVENT SERVICES . . EVENT PROFILES . . DATA BASES . . OTHER DATA LEVEL TWO SITUATION DETECTION LEVEL THREE PREDICTIVE ANALYSIS LEVEL FOUR ADAPTIVE BPM

Structured Processing for Event-Decision

Multi-level inference in a distributed event-decision architectures

User Interface

Human visualization, monitoring, interaction and situation management

Level 4 – Process Refinement

Decide on control feedback, for example resource allocation, sensor and state management, parametric and algorithm adjustment

Level 3 – Impact Assessment

Impact assessment, i.e. assess intent on the basis of situation development, recognition and prediction

Level 2 – Situation Refinement

Identify situations based on sets of complex events, state estimation, etc.

Level 1 – Event Refinement

Identify events & make initial decisions based on association and correlation

Level 0 – Event Preprocessing

Cleansing of event-stream to produce semantically understandable data

Level of Inference Low Med High

CEP Level 0 – Event Preprocessing

Cleanse/Refine/Normalize Data for Upstream Processing

Calibrate Raw Event Cloud:

Web Server Farm Event Stream Example -

Group HTTP REQUESTS and RESPONSES

Reduce and Extract Required Data from Transaction

Format into Event for Upstream Processing

Agent-Based Log File Event Steam Example -

Parse Log File for Sensor Information

Match Patterns and Convert Tokens to JMS Properties

Reduces System Load by Preprocessing Events

Enables Upstream to Concentrate on Most Relevant Events

Focuses on Objects/Events

CEP Level 1 – Event Refinement

Problem: Which Events in the Event Stream Are “Interesting”?

Event Refinement Example (Association & Classification):

Hypothesis Generation (HG)

Processing incoming events, data and reports

Hypothesis: This Group of Events May Need to be Tracked

Output: Scorecard or Matrix

Hypothesis Evaluation (HE)

Evaluates Scorecard/Matrix for likelihood evaluation

Rank Evaluation: These Events have a Higher Likelihood

Output: Fills Scorecard/Matrix with relative likelihood estimation

Hypothesis Selection (HS)

Evaluates Scorecard/Matrix for best fit into scenario

Evaluation: Provide an Estimate (Name) of the Scenario Activity

Output: Assignment of scenario - activity estimate to event

CEP Level 2 – Situation Refinement

What is the Context of the Identified Events?

Focuses on Relationships and States Between Events

Situation Refinement

Event-Event Relationship Networks

Temporal and State Relationships

Geographic or Topological Proximity

Environmental Context

Example: Brand currently used by phishing site in Internet increasing probability of fraud and identity theft

Event / Activity Correlation – Relational Networks

Pattern, Profile and Signature Recognition Processing

CEP Level 3 – Impact Assessment

Predict Intention of Subject (Fraudster example)

Make changes to account identity information?

Transfer funds out of account?

Test for access and return at later time?

Estimate Capabilities of Fraudster

Organized Gang or Individual Fraudster?

Expert or Novice?

Estimate Potential Losses if Successful

Identify Other Threat Opportunities

CEP Level 4 – Process Refinement

Evaluate Process Performance and Effectiveness

Exception Detection, Response Efficiency and Mitigation

Knowledge Development

Identify Changes to System Parameters

Adjust Event Stream Processing Variables

Fine Tune Filters, Algorithms and Correlators

Determine If Other Source Specific Resources are Required

Recommend Allocation and Direction of Resources

Database Management Examples

Reference Database

User Profiles

Activity and Event Signatures and Profiles

Environmental Profiles

Inference Database

Subject Identification

Situation and Threat Assessment

Knowledge Mining

Referential Mapping Database Examples

Mapping Between IP Address and Domain

Mapping Between Known Anonymous Proxies

User Interface / Interaction

Operational Visualization at all “Levels”

Dynamic Graphical Representations of Situations

Supports the Decision Making Process of Analytics Personnel

Process and Resource Control

Supports Resource Allocation and Process Refinement

Display Control & Personalization

Different Operator Views Based on Job Function and Situation

Business Optimization Summary A Simplified View of the CEP Reference Architecture Flexible SOA and Event-Driven Architecture

Our Agenda

TIBCO’s Real-Time Agent-Based SEM Approach A Multisensor Data Fusion Approach to Security Event Management Distributed Fraud and Intrusion Detection Systems, Logs Detection Approach Systems Protected Architecture Data Sources Analysis Timing Detection Actions HIDS NIDS Hybrid Audit Logs Net Traffic System Stats Real Time Data Mining Anomaly Detection Signature Detection Centralized Distributed Active Passive Agent Based Enterprise Correlation of Security Events

Security Event Management High Level Event-Driven Architecture (EDA) for SEM (CEP and BPM) JAVA MESSAGING SERVICE (JMS) DISTRIBUTED EVENTS (TIBCO EMS) HIGH PERFORMANCE RULES-ENGINE (TIBCO BE) HIGH PERFORMANCE RULES-ENGINE (TIBCO BE) HIGH PERFORMANCE RULES-ENGINE (TIBCO BE) SENSOR NETWORK RULES NETWORK FDS BW JMS LOGFILE JMS BW LOGFILE JMS BW LOGFILE JMS BW IDS JMS BW FDS JMS BW SQL DB BW JMS ADB SQL DB BW JMS ADB MESSAGING NETWORK SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM SYSTEM BPM Compliance Workflow (TIBCO iProcess)

TIBCO BusinessEvents™ Solutions Overview BusinessEvents™ Solutions Space Data: Events & Databases -Real-Time & Historical Data Models: Statistical Financial Optimization Comms: Pub/Sub Messaging Queues Topics UIs Knowledge: Facts & Rules

TIBCO BusinessEvents™ Overview

High performance, low latency business rules engine.

Top down business process modeling.

Real-time event processing.

Cross-application and cross-process integration.

Analytical and predictive models.

Modeling Tools, Statefulness, Business Rules and Process Integration UML Conceptual UML State Business Rules Business Users Event Analyzer

TIBCO BusinessEvents™ Overview Collection, Normalization Metric of Managed Objects, Normalized Non-Contextual Events Metadata Repository Event Management, Correlation, Aggregation, Inference and Analysis Correlated, Analyzed, Contextual Dialogue Events Rules, Knowledge, Patterns, Models Visualization, Reporting, Alert Management Application Interface Feeds Visualization: Detection Metrics Agents Synthetic Warehouse Visualization: Process View Dialogue Manager Inference Engine FDS/IDS Logfiles Edge Devs Semantic Model Events Rules Design Environment State Model Sensors

TIBCO BusinessEvents™ Awards 2006 Best Complex Event Processing Software Winner: TIBCO 2006 Event Processing General Purpose Gold Award Winner

CEP and BusinessEvents™ Case Study: Real-Time On-Line Fraud Detection Requirements

Identify characteristics of fraud, such as continuous behavior changes, and identify new patterns of fraud

Stop new account setups from fraudulent IP addresses

Stop online registrations from fraudulent IP addresses

Verify user identity in every transaction based on click-behavior

Identify multiple users trying to login from same IP address

Identify single user logins from multiple IP addresses within a time span

Prevent phishing by tracking IP addresses that mass download institutional web pages

Prevent phishing, pharming and man-in-the-middle attacks by checking against a list for fraudulent IP’s in real-time

On-Line Fraud Detection Use Case Architecture and Capacity Planning Approx. 12,000 Hits Per Second During Peak Period Across the Three Sites – One Instance Of TIBCO BusinessEvents™ Capable of Handling Maximum Hits Overall 100 Million Hits Handled Between 3PM – 4 PM Peak Approx. 250 Million Hits Per Day Across the Three Sites TIBCO EMS™ TIBCO Business Events™ Session Info Three Server Farms ~600-700 Application Servers

Characteristics of Solutions Architecture