Vision 2014: Inside The Box Business Scoring Validations and Model Governance

398 views
255 views

Published on

Given an environment of heightened attention to score use and validating score performance, this session will cover case studies on score validations and how this process can be used for risk management, compliance and meeting regulatory standards.

Published in: Business, Economy & Finance
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
398
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Vision 2014: Inside The Box Business Scoring Validations and Model Governance

  1. 1. ©2014 Experian Information Solutions, Inc. All rights reserved. Experian and the marks used herein are service marks or registered trademarks of Experian Information Solutions, Inc. Other product and company names mentioned herein are the trademarks of their respective owners. No part of this copyrighted work may be reproduced, modified, or distributed in any form or manner without the prior written permission of Experian. Experian Public. Inside the box — business scoring validations and model governance Chuck Noel Compass Bank Torsten Gerwien Experian John Krickus Experian #vision2014
  2. 2. 2©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.  Model build and validation process  Metrics for validating model performance  Case study – BBVA Compass  Model governance policy and best practices Agenda
  3. 3. 3©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Model build and validation process John Krickus Experian Business Information Services
  4. 4. 4©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Bad definition: Firms filing bankruptcy / firms where GT 75% of trade dollars are 91+ and / or negative trade comments for multiple quarters and at the end Financial stability model process Performance window Month 0 Observation point SOURCE: Experian Business Information Services What archived data from the observation point, when all records had “clean” data, would have predicted the “bads” at the end of the performance window in month 12? All with “clean” data; no bad performance Month 12 3% “bad” rate “Bads” observed in sample
  5. 5. 5©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. 1) Population sampling 75/25 split is applied: 75% is for model development and 25% for model validation 2) Population segmentation 3) Proof of concept (POC) analysis to test alternative performance definitions, alternative segmentation scheme 4) Data treatment, variable selection and regression for each individual model ► Data treatment: Variable capping/false zero correction/variable transformations ► Variable selection: STEPDISC procedure in SAS is employed to do the variable selection ► Statistical regression 5) Finalize model and prepare documentation Modeling build process Development steps
  6. 6. 6©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 2 3 4 5 6 7 8 9 10 Cumulativepercentcaptured Score decile, worst to best Cumulative good Cumulative bad KS = 33.8  KS is a generally accepted measure of a model’s ability to separate two populations KS (Kolmogorov-Smirnoff) example
  7. 7. 7©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Metrics for validating model performance Torsten Gerwien Experian Decision Analytics
  8. 8. 8©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Stability analysis – score distribution The overall score distribution didn't change significantly; the difference in the two distributions mainly exhibited that sample in 2010 has higher scores at low end and lower scores at the high end Score distribution comparison in percentile – IP214 CML New (9/10/11/12-2010)Old (9/2008) Diff 100 90 80 70 60 50 40 30 20 10 0 Score 6 5 4 3 2 1 0 -1 -2 -3 -4 Distributiondifference(points) 5 90 95 1007055 60 655035 40 453015 20 2510 75 80 85 Percentile
  9. 9. 9©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Stability analysis – divergence measure Score divergence result Conclusion: No significant change Sample Obs # Mean Median Min Max Var Old 383,973 50.56 50 1.00 100 820.34 New 652,683 49.23 49 1.00 100 718.71 Divergence = 0.00 Def: DIV = ((NEW_MEAN – ORG_MEAN) ** 2) / (0.5*(NEW_VAR + ORG_VAR))
  10. 10. 10©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Performance of industry-specific vs. all industry  80% bad capture for industry model vs. 50% for all-industry model (bottom 20% bads)  Bad definition: Two 60-day late pays or one 90-day late card payment  Model built off of Small Business Credit ShareSM financial trade and general business database, bad rate: 7.81% Small Business Credit ShareSM: Bad capture bottom 20%, private card SBCS = Small Business Credit ShareSM
  11. 11. 11©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Superior ROI from Custom Model capabilities Experian business / SBCS data and analytic expertise  Combination of full range of business information and SBCS data from financial contributors  Experian Decision Analytics expertise  All combined in a Custom Model value offering delivered in weeks  Improving risk management results by 53% 3% lift 53% lift 41% lift46% lift SBCS = Small Business Credit ShareSM
  12. 12. 12©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. KS, bad capture for worst 10%, 20% and 30% of population 23 30% 43% 52% 29 29% 50% 59% 22 30% 40% 50% 31 31% 44% 59% 0% 10% 20% 30% 40% 50% 60% 70% KS 10% 20% 30% IPV1 IPV2 FSR IPV2+FSR  Intelliscore PlusSM v2 (IPV2)+ FSR matrix provides the highest KS statistic  However, bad capture at the worst 20% shows stronger performance for IPV2  IPV2+FSR matrix optimized for bad account and $ capture Performance results KS and bad account capture
  13. 13. 13©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. BBVA Compass Validation results Chuck Noel SVP Global Risk Management BBVA Compass
  14. 14. 14©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Validation objectives Objectives  Determine the ability of Experian information to predict Compass Bank medium size exposure bads for ongoing account management  Validated the performance of multiple Experian commercial risk models  Developed custom decision tree model to identify common characteristics of bads  Validated the predictiveness of Experian commercial triggers BBVA Compass is a Sunbelt-based bank operating 687 branches in Texas, Alabama, Arizona, California, Florida, Colorado and New Mexico. BBVA Compass ranks among the Top-25 largest U.S. commercial banks based on deposit market share
  15. 15. 15©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Performance results KS and bad capture  The financial models (FC / FCC / FRC) had higher KS  The generic models (Intelliscore PlusSM v2 and FSR) have higher score (hit) rate  Focus will be on Intelliscore PlusSM v2 for custom decision tree Bad Capture Score 10% 20% 30% KS Intelliscore PlusSM v2 39.73% 50.00% 55.21% 30.9 FSR5 30.55% 37.67% 51.37% 30.7 FTC 34.01% 52.38% 59.18% 33.8 FCC 45.67% 57.67% 67.33% 39.1 FRC 30.83% 61.25% 67.50% 42.9
  16. 16. 16©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.  Decision tree model is used to illustrate benefits of customization  A decision tree is a statistical model that determines the optimal attributes for separating the good accounts from bad accounts  Each ‘branch node’ consists of an attribute which can be split on the attribute’s value ranges to further separate goods from bads  Attributes assessed for Compass model: ► Intelliscore PlusSM v2 risk score ► Business age ► Trade count ► Trade balance ► Trade payment behavior ► Collections ► Tax lines ► Judgments Custom decision tree (DT) model
  17. 17. 17©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Decisions tree example Numbers are example only, KS result is real Score 1-24 Total = 200 1 Bad = 5 0 Good = 195 Within no trade segment, new “branch” Low Intelliscore 1-24, 10% of applicants Bad rate for this segment of 2.5% 5X greater bad rate then portfolio Total = 8,000 1 Bad = 40 0 Good = 7,960 Start with total portfolio of 8,000 accounts Bad rate of 0.5% or 40 accounts No Trade Total = 2,000 1 Bad = 15 0 Good = 1,885 Segment identified, higher bad rate 0.75% No trade indicates 50% higher risk KS improved 40% by using decision tree branches vs. generic Intelliscore PlusSM v2 score
  18. 18. 18©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Model governance policy and best practices Torsten Gerwien Experian Decision Analytics
  19. 19. 19©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.  Monitors attributes for soundness  Tracks the impact of changes on existing attributes  Provides regulatory audit support  All of which includes: ► Framework for effective attribute development and use ► Documentation management for robust version tracking  Business data attributes: ► Business Aggregates, Small Business Credit ShareSM Aggregates ► Delivered in batch file monthly (Commercial Risk DatabaseSM), archives, online via NetConnect Attribute risk governance Attribute governance is necessary for compliance
  20. 20. 20©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. OCC Bulletin 2000-16 2000 OCC Bulletin 2011-12 2011 .  The Supervisory Guidance on Model Risk Management OCC Bulletin 2011–12 extends the scope beyond model validation to policies, practices, standards for: ► Model development ► Model use and implementation ► Model governance and controls Banks of all sizes are impacted Model Risk Governance
  21. 21. 21©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. For additional information, please contact: Torsten.Gerwein@experian.com John.Krickus@experian.com Hear the latest from Vision 2014 in the Daily Roundup: www.experian.com/vision/blog @ExperianVision | #vision2014 Follow us on Twitter
  22. 22. 22©2014 Experian Information Solutions, Inc. All rights reserved. Experian Public. Visit the Experian Expert Bar to learn more about the topics and products covered in this presentation.

×