Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Framework Driven Approach to Model Risk Management (


Published on

Model risk and the importance of model risk management has gotten significant attention in the last few years. As financial companies increase their reliance on quants and quantitative models for decision making, they are increasingly exposed to model risk and are looking for ways to mitigate it. The financial crisis of 2008 and various high profile financial accidents due to model failures has brought model risk management to the forefront as an important topic to be addressed. Many regulatory efforts (Solvency II, Basel III, Dodd-Frank etc.) have been initiated obligating banks and financial institutions to incorporate formal model risk management programs to address model risk. Regulatory agencies have issued guidance letters and supervisory insights to assist companies in developing model risk management programs. In the United States, as the Dodd-Frank act is implemented, newer guidance letters have been issued that emphasize model risk management. Despite these efforts, in practice, financial companies continue to struggle in formulating and developing a model risk management program. A lot of companies acknowledge and understand the model risk management guidelines in spirit but have practical challenges in implementing these guidance letters. In our prior article on model risk , we discussed many drivers to address model risk and challenges in integrating model risk into the quant development process. In this talk, we will discuss ten best practices for the implementation of an effective model risk management program. These best practices have evolved from discussions with industry experts and consulting projects we have worked with in the recent years to create robust risk management programs. These best practices meant to provide practical tips for companies embarking on a formal model risk management program or enhancing their model risk methodologies to address the new realities

Published in: Data & Analytics
  • Be the first to comment

A Framework Driven Approach to Model Risk Management (

  1. 1. A FRAMEWORK DRIVEN APPROACH TO MODEL RISK MANAGEMENT Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC. Sri Krishnamurthy, CFA Founder and CEO QuantUniversity LLC. DATA ANALYTICS FOR FINANCIAL SERVICES, JULY 22 2014, BOSTON, MA
  2. 2. AGENDA About Model Risk Analytics Model Risk : A Brief Introduction A Framework driven approach to Model Risk Management Quantifying Model Risk Stress and Scenario Testing in Model Risk Management 1 2 3 4 5 Agenda
  3. 3. • Founder of QuantUniversity LLC. • Advisory and Consultancy for Model Risk • Prior Experience at MathWorks, Citigroup and Endeca and 25+ customers • Regular Columnist for the Wilmott Magazine • Author of forthcoming book “Financial Modeling: A case study approach” published by Wiley • Charted Financial Analyst and Certified Analytics Professional • Teaches Analytics in the Babson College MBA program Sri Krishnamurthy Founder and CEO SPEAKERBIO
  5. 5. OURSOLUTION Comprehensive Model Risk Assessment Framework Model Risk Quantification Methodology Fully auditable cloud-based Model Risk Management platform 5 5 10 15 20 25 4 4 8 12 16 20 3 3 6 9 12 15 2 2 4 6 8 10 1 1 2 3 4 5 1 2 3 4 5 Likelihoodof occurrence Impact RiskScores Model Governance Model Lifecycle Management Model Risk and Validation Makes measuring risk and prioritizing issues easier Complies with audits and regulatory needs Proven framework for a Model risk platform MODELRISKANALYTICS  Customer Validation  Industry Acceptance  Novel and fast growing
  7. 7. MODELRISKINTHENEWS Financial accidents have cost companies millions of dollars and was blamed for the financial crisis of 2008 Concerned about systemic risk, regulators have stepped up regulations to setup model risk programs All Banks, Insurance Companies and Credit Rating agencies in the US and EU are affected by these regulations As financial institutions depend on models for decision making, Model Risk Management is critical 8Source: E&Y Survey 69 banks & 6 insurance companies
  8. 8. CHALLENGES Quantitative models are complex: Measuring model risk is not easy Quantitative systems are complex : Many stakeholders Novelty: Lack of guidance and ambiguity on regulations Quants Risk Portfolio Management IT 1 2 3 Financial institutions face challenges implementing Model Risk Programs SEC = ?
  9. 9. MODELRISK “Model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk Defined Input Assumptions/Data Processing component Output/ Reporting component
  10. 10. MODELVALIDATION “Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1] “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk and Validation Defined
  12. 12. MODELRISKMANAGEMENT Elements of a Model Risk Management framework 1. Model Governance structure: Addresses regulatory requirements, roles, responsibilities, oversight, control and escalation procedures 2. Model Lifecycle management: Addresses the processes involved in the design, development, testing, deployment and use of models. Also addresses testing and documentation plans and change management. 3. Model Review and Validation Process: Addresses internal and external model review, validation and ongoing monitoring of models (both qualitative and quantitative)
  13. 13. MODELGOVERNANCESTRUCTURE Model Governance Structure Model Governance Structure Regulatory guidance and best practices Model Classification Roles and Responsibil ities Oversight and Controls
  14. 14. MODELLIFECYCLEMANAGEMENT Model Lifecycle Management
  15. 15. MODELREVIEWANDVALIDATION Model Review and Validation Policy: Model Policy Review Structure: Model Process Review Content: Model Review
  16. 16. LEVERAGINGTECHNOLOGYANDANALYTICSFOREFFECTIVEMODELRISKMANAGEMENT Leveraging technology and analytics for effective Model Risk Management 1.Quantifying Model Risk: • Classification and Measurement of Model Risk 2.Leveraging technology to scale stress and scenario testing
  18. 18. CHALLENGES Organizational Structure Organization Enterprise Risk Management Compliance Model Research and Development End Users IT How to engage all departments strategically to have a comprehensive view of Model Risk ?
  19. 19. HOWTODOIT Theory to Practice : How to cross the chasm ? Image Courtesy: • Theory • Regulations • Local Laws • Practical IT systems • Company policies • Company culture and Best practices
  20. 20. CLASSIFYINGMODELRISK Classifying Model Risk Class 3 Example: Monte-carlo simulation engine Class 2 Example: Linked-spreadsheet model with dependencies Class 1 Example: Simple Spreadsheets Complexity 1. Class 1 Models: Simple Models typically involving less complex atomic calculations 2. Class 2 Models: Models more complicated than Class 1 models 3. Class 3 Models: Typically involves sophisticated mathematical/statistical techniques
  22. 22. SCORINGGUIDELINES Scoring Guidelines
  23. 23. RISKGRADING Sample Risk grading considering impact and likelihood of occurrence 5 5 10 15 20 25 4 4 8 12 16 20 3 3 6 9 12 15 2 2 4 6 8 10 1 1 2 3 4 5 1 2 3 4 5 Likelihood of occurrence Impact Risk Scores Red High Risk Yellow Moderate Risk Green Low Risk High Impact- High likelihood of occurrence : Needs adequate model risk control measures to mitigate risk High Impact – Low likelihood of occurrence: Address through model risk control measures and contingency plans Low Impact – High likelihood of occurrence : Lower priority model risk control measures Low Impact – Low likelihood of occurrence: Least priority model risk control measures
  24. 24. LEVERAGINGMACHINELEARNING Machine learning techniques applied to Quantifying Model risk 1. Clustering to bucket “similar” risks • Identifying training opportunities and best practices for model development 2. K-Nearest Neighbor (k-NN) to automatically derive risk scores • Leveraging expert scoring to help prioritize issues 3. Conjoint analysis • Identifying what combination of a limited number of attributes is most influential on respondent choice or decision making
  26. 26. STRESSTEST&SCENARIOTEST Stress Tests and Scenario Tests Figure courtesy:
  27. 27. DEFINITIONS Definitions 1. Scenarios : “A scenario is a possible future environment, either at a point in time or over a period of time.” “Considers the impact of a combination of events“ 2. Sensitivity Analysis: “A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “ 3. Stress Testing: Analysis of the impact of single extreme events (or risk factors) Ref:
  28. 28. CONSIDERATIONSFORMANAGINGUNCERTAINTY Considerations for managing uncertainity Case One: Single unknown parameter Example : Volatility Case Two: Two Unknown Parameters Example : Uncertain about mean μ as well as the standard deviation σ Case Three: Unknown correlation(s) Case Four: Mixing parameter and Distribution risk Ref: Wiley: Measuring Market Risk, 2nd Edition - Kevin Dowd
  29. 29. HANDLINGLARGEDIMENSIONALDATASETS Handling large dimensional data • Variable Selection methods (where a subset of dimensions is chosen) • Reducing dimensions (where variables are transformed to a smaller set of new variables) Ref: Dealing with Dimensionality in large datasets
  30. 30. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING Characteristics of Stress and Scenario testing 1. Difficult to build parametric models – Simulation driven approach necessary 2. Parameter space can explode easily 3. Tests independent of each other (Embarrassingly parallel) 4. Complete test-coverage – Useless 5. Human intervention required 6. Tests to be designed and customized for the companies needs considering portfolios, organization structure and regulatory obligations Ref:
  31. 31. LEVERAGINGTECHNOLOGYTOSCALESTRESSANDSCENARIOTESTING Leveraging Technology to scale Stress and Scenario Testing • Advances in technology in the last two decades have significantly enhanced the toolsets quants have to develop, test and scale innovative quantitative applications • Simulation and stress testing in risk management are vastly scalable due to innovations in parallel and distributed computing • Restricting the number of tests due to lack of technological resources not an excuse Ref: Gaining the Technology Edge:
  32. 32. LEVERAGETECHNOLOGYTOSCALEANALYTICS Leverage technology to scale analytics 1.64 bit systems : Addressable space ~ 8TB 2.Multi-core processors : Explicit and Implicit Multi-threading 3.Parallel and Distributed Computing : Leverage commodity/Specialized hardware to scale problems 4.General-purpose computing on graphics processing units : Use graphics cards to scale your algorithms 5.Cloud Computing Ref: Gaining the Technology Edge:
  33. 33. BESTPRACTICESTOLEVERAGETECHNOLOGYFORSTRESSTESTING Best practices to leverage technology for Stress testing 1. Test Plan to optimize models for technology use 2. Parameter mapping plan to scale stress tests based on use cases 3. Aggregation of Test results 1. Pass/Fail 2.Automatic validation and Benchmarking of results 3.Analytics to infer the state and maturity of the model 4.Model Monitoring Test plan 1.Monitoring of parameters 2.Monitoring of test adequacy 5.Integrate Testing in the Model Lifecycle Development process
  34. 34. REFERENCES Quantifying Model Risk Wilmott Magazine January 2014 The Decalogue Wilmott Magazine July 2014 Copies can be downloaded at :
  35. 35. SUMMARY Model Risk : A Brief Introduction A Framework driven approach to Model Risk Management Quantifying Model Risk Stress and Scenario Testing in Model Risk Management 1 2 3 4 Summary
  36. 36. UPCOMINGCOURSES Upcoming courses 1. Data Science and Analytics for Quants 2. Model Risk Management Covers analytical techniques and Big Data methodologies in finance. Covers Model Risk Management framework, analytical techniques and case studies Register your interest at
  37. 37. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC.