Rapid Model Refresh (RMR) in Online Fraud Detection Engine

Rapid Model Refresh (RMR)in Online Fraud Detection Engine
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies. © 2010 SAS Institute Inc. All rights reserved. S55547.0410

Agenda
Overview
Traditional Tactics Fighting Fraud
Best Practice in PayPal Fraud Detection
Rapid Model Refresh (RMR)
Extensions and Future

Online Fraud in Financial Services
Evolution in Financial Services
Paper-Based
In-Branch
Perceptible Footprint
… …
Electronic
Cyber Spaces
Invisible Marketplace
… …
Emerging Fraud Trends
Old-Fashion
Isolated Individual
Limited-Scope Damage
Traceable Patterns
… …
Tech-Savvy
Organized Gang
Multi-Billion Loss
Dynamic Trends
… …

Industry Fact
Online Revenue Loss Due to Fraud
Source: Cybersource

Agenda
Objectives

Traditional Mitigation Tactics
Heuristic Approach
Detect Anomalies
Identify Patterns
Set Review Criterion
Model-Based Score
Rely on Statistical Models (Logit Models / Neural Nets)
Generate Suspicion Score
Rank Order Transactions
Rule-Based System
Employ Machine Learning Algorithms
Generate Rule Sets for Segmentation
Target High-Risk Segments

Pros and Cons
Heuristic
Review-Based & Labor Intensive
Local Solutions without Global View
Integrate Domain Knowledge
Easy to Implement
Scoring
Long Time-to-Market
Static perspective of Fraud Trends
Successful Industrial Applications
Ideal for Large-Scale Domains
Rule-Induction
Require Frequent Refreshes
Burden of High-Volume Rules
Fits Dynamic Online Nature
Rapid Development & Deployment

Next … …
Now What?

Agenda
Objectives
Best Practices in PayPal Fraud Detection

PayPal's Way to Fight Frauds
PayPal Loss Trend from 200X through 200Y

Multi-Level Detection Engine
Agent Review
Risk Scoring
Rule Induction
Analysts built decision trees on high-risk transactions ranked order by risk scoring.
Most risky segments are further identified by balancing between bad and pass-through rate.
Modelers developed scoring models with logistic regression / neural network
Risk score is assigned to each transaction through the system.
Low-risk transactions will be passed through.
Most risky transactions identified by rule sets are sent into review queues.
Queued transactions are prioritized and routed to agents in specific domains.
Case review and investigation are conducted.

Implementation Challenges
Problems
Realities
Fast-Growing International Footprint
Overwhelming Number of Segments & Models
Extremely Rich Data from Diversified Sources
Information Overload instead of Data Mining
Ever-Complicated IT Infrastructure
High Exposures to System Risks
Dynamic Fraud Trends & Smarter Fraudsters
Escalating Model Decay & Deterioration

Data-Driven Model (DDM) Strategy
Dynamic Rule Induction
Automatic Model Development
Real-Time Deployment
Conceptual
DDM
Modular Data Processing
Daily Monitoring
Implemented by

Agenda
Objectives

What’s RMR?
Three Common Layers
Data
Layer
Deployment
Layer
Algorithm
Layer
Packaged Processing
Optimized Queries
Repeatable Stream
Arbitrary Models
Standard Evaluation
Version Controlled
Model Specs. to XML
Deploy in Real-Time
Batched Monitor

RMR – Data Layer
Model Development SAS Data
Fine
Layer
SAS as Wrapper around Shell / SED / BTEQ Scripts
Modular SAS Macros & Parameterized Scripts
Variables Creation / Imputation / Transformation
Web Logs
Enterprise Database
3rd-Party Sources
Coarse
Layer

Data Layer at A Glance
Data Manipulation
Variable Transformation
20+ SAS Macros
Update Parameters in Scripts
SED Stream Editor
SAS Workflow
Create Dynamic SQL
Parallel Execution
Shell Scripts
Submit SQL
BTEQ Interface with Teradata

Code Snippet in Data Layer
1
2
3
Use SED update parameters in the query
Submit the query to Teradata through BTEQ
Append the log into a output file

RMR – Algorithm Layer
Best Models to Production
Swap Analysis for Rule Sets
Model Evaluation (KS / AUC / … )
Stump
Exhaustive Search for Best Cutoffs
Bumping
Stochastic Search for Best Tree(s)
Champion
Generalized Linear Model
Arbitrary Challengers
Neural Nets
Bagging Trees
… …
Supported by SAS / STAT & SAS / Enterprise Miner

A Peek into Algorithm Layer
25% Validation
25% Testing
WoE Vars
GLM
SASEvaluation Macros
SAS EDA Macros
Best Model
50% Training
NNET
Bagging
Binned Vars
… …
TreeX
Tree2

One Tree, Endless Possibilities
Use Cases of Decision Tree in RMR’s View
Bagging
Simple Average of Massive Number of Trees
Take Advantages of RMR Deployment Layer and Parallel Computing
Use as A Challenger to Traditional Logistic Regression
Bumping
Stochastic Search from Massive Number of Trees
Improve Estimation while Retain Simple Tree Structure
Use to Enhance Vallina-Version Tree Development
Stump
Exhaustive Search on 1-Dimension Space, e.g. Score
Induce 1-Level Binary Tree by Minimizing Gini Impurity
Use to Find the Best Score Cutoff while Balancing Review Rate

Pick Winner from Multiple Candidates
Generically Support Arbitrary Number of Score Inputs for Massive Models Evaluation and Deployment

RMR – Deployment Layer
Model Specifications
Perl
Convert to XML / PMML
Inject into Web Engine
Collect Web Logs in DB
SAS
Monitor Daily Scoring Stability
Shell
Email Reports to Stakeholders

A Use Case: Score Monitoring
Objectives:
System Breakage
Score Shift
Driver Table
Log Table
Lookup Tables
Daily Web Log
Baseline Distribution
Model / Segment / Owner Lookups
SAS Daily Job Scheduled by Cron
Population Stability Reports in Html

Sample Reports
Overall
Detailed

Agenda
Objectives

Evolution of RMR Paradigm
Future
Past
Now
Expert Process
Programmers Pull Data
Statisticians Build Predictive Model
Engineers Hard-Code Specification into On-Line Environment
Meets Minimum Benefit Schedule.
Mechanized Process
Population and Performance Criterion Identified
A Suite of Challenger Models Built Automatically
Model Specifications Published in Live Scoring Platform
New Models Deployed in Periodic Batch
Online Process
Models Developed & Deployed with Most Recent Online Data Dynamically
Re-deployment of New Models not Needed

2-Path Directions
SAS / Teradata in-DB Analytics
Hadoop with R Integration

Rapid Model Refresh (RMR) in Online Fraud Detection Engine

by liuwensui on May 14, 2011

Accessibility

Tags

Upload Details

Usage Rights

Statistics

Rapid Model Refresh (RMR) in Online Fraud Detection Engine — Presentation Transcript