Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FitchLearning QuantUniversity Model Risk Presentation


Published on

In this lecture, we discuss the importance of a framework driven model risk approach and discuss 4 aspects required to operationalize model risk including quantifying uncertainty, quantifying model risk, importance of model verification in model risk management and leveraging technology to scale stress and scenario testing

Published in: Economy & Finance
  • Be the first to comment

FitchLearning QuantUniversity Model Risk Presentation

  1. 1. A FRAMEWORK BASED APPROACH TO MODEL RISK MANAGEMENT Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC. Sri Krishnamurthy, CFA Founder and CEO QuantUniversity LLC. FITCHLEARNING PRESENTATION, NOVEMBER 18 2014, NEW YORK
  2. 2. AGENDA About Model Risk Analytics Model Risk : A Brief Introduction A Framework driven approach to Model Risk Management Quantifying Model Risk Stress and Scenario Testing in Model Risk Management 1 2 3 4 6 Agenda Role of Model Verification in Model Risk Management 5 Demo 7 Summary of Best Practices in Model Risk Management 8 9 Understanding the nature of Uncertainity
  3. 3. •Founder of QuantUniversity LLC. •Advisory and Consultancy for Model Risk •Prior Experience at MathWorks, Citigroup and Endecaand 25+ financial services customers •Regular Columnist for the WilmottMagazine •Author of forthcoming book “Financial Modeling: A case study approach” published by Wiley •Charted Financial Analyst and Certified Analytics Professional •Teaches Analytics in the Babson CollegeMBA program and at Northeastern University Sri Krishnamurthy Founder and CEO SPEAKERBIO
  6. 6. MODEL RISK INTHENEWS Financial accidents have cost companies millions of dollars and was blamed for the financial crisis of 2008 Concerned about systemic risk, regulators have stepped up regulations to setup model risk programs All Banks, Insurance Companies and Credit Rating agencies in the US and EU are affected by these regulations As financial institutions depend on models for decision making, Model Risk Management is critical 7 Source: E&Y Survey 69 banks & 6 insurance companies
  7. 7. Image courtesy: DENIAL! However, when it comes to Model Risk, many companies are still in denial.
  8. 8. CHALLENGES Quantitative models are complex: Measuring model risk is not easy Quantitative systems are complex : Many stakeholders Novelty: Lack of guidance and ambiguity on regulations Quants Risk Portfolio Management IT 1 2 3 Financial institutions face challenges implementing Model Risk Programs ?
  9. 9. HOWTODOIT Theory to Practice : How to cross the chasm ? Image Courtesy: •Theory •Regulations •Local Laws •Practical IT systems •Company policies •Company culture and Best practices
  11. 11. MODEL “Model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates” [1] Ref: [1] . Supervisory LetterSR11-7 on guidance onModel Risk Model Defined Input Assumptions/Data Processing component Output/ Reporting component
  12. 12. MODELVALIDATION “Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1] “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1] Ref: [1] . Supervisory LetterSR11-7 on guidance onModel Risk Model Risk and Validation Defined
  13. 13. PROBLEMS Problems adopting Model Risk Management 1.Too many guidelines ; Interpretation left to companies 2.Biggest challenge is to operationalize many of the guidelines and best practices 3.Lack of formal frameworks for Model Risk Management 4.Customization needed for each company’s operations 5.Novelty
  14. 14. MODELRISKMANAGEMENT Elements of a Model Risk Management framework 1.ModelGovernancestructure:Addressesregulatoryrequirements,roles, responsibilities,oversight,controlandescalationprocedures 2.ModelLifecyclemanagement:Addressestheprocessesinvolvedinthedesign,development,testing,deploymentanduseofmodels.Alsoaddressestestinganddocumentationplansandchangemanagement. 3.ModelReviewandValidationProcess:Addressesinternalandexternalmodelreview,verification,validationandongoingmonitoringofmodels(bothqualitativeandquantitative)
  15. 15. MODELGOVERNANCESTRUCTURE Model Governance Structure Model Governance Structure Regulatory guidance and best practices Model Classification Roles and Responsibilities Oversight and Controls
  16. 16. MODELLIFECYCLEMANAGEMENT Model Lifecycle Management
  17. 17. MODELREVIEWANDVALIDATION Model Review and Validation Policy: Model Policy Review Structure: Model Process Review Content: Model Review Model Verification Model Validation
  18. 18. OPERATIONALIZINGMODELRISKMANAGEMENT Operationalizing Model Risk Management 1. Understand the nature of uncertainty 2. Quantifying Model Risk: Classification and Measurement of Model Risk 3. Role of Model Verification for Model Risk Management 4. Leveraging technology to scale stress and scenario testing
  20. 20. Quantifying Uncertainty is hard
  21. 21. Kennedy, Marc C., and Anthony O'Hagan. "Bayesian calibration of computer models."Journal of the Royal Statistical Society: Series B (Statistical Methodology)63.3 (2001): 425-464. Sources of Uncertainty Parameter uncertainty Interpolation uncertainty Experimental uncertainty Algorithmic uncertainty Structural uncertainty Parameter variability
  22. 22. Uncertainty Aleatoryuncertainty Or Statistical uncertainty Epistemic uncertainty Or Systemic uncertainty Roy, Christopher J., and William L. Oberkampf. "A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing."Computer Methods in Applied Mechanics and Engineering200.25 (2011): 2131-2144. Given sufficient samples of the stochastic process, can be characterized via a pdf Uncertainty due to lack of knowledge by the modelers conducting the analysis or analysts involved in validation
  23. 23. UNDERSTANDTHENATUREOFUNCERTAINTYINMODELS Understand the nature of uncertainty in models 1.To evaluate model risk, it is essential to understand the nature of uncertainty that is in the model. 2.Are inputs uncertain? 3.Is the model deterministic or stochastic? 4.What kind of assumptions and simplifications do we have in the model? Ex: Correlations, Distribution assumptions 5.Do we have data to verify if the model’s outputs mimic reality?
  25. 25. CHALLENGES Organizational Structure Organization Enterprise Risk Management Compliance Model Research and Development End Users IT How to engage all departments strategically to have a comprehensive view of Model Risk ?
  26. 26. CLASSIFYINGMODELRISK Classifying Model Risk Class 3 Example: Monte-carlo simulation engine Class 2 Example: Linked-spreadsheet model with dependencies Class 1Example: Simple Spreadsheets Complexity 1.Class 1 Models: Simple Models typically involving less complex atomic calculations 2.Class 2 Models: Models more complicated than Class 1 models 3.Class 3 Models: Typically involves sophisticated mathematical/statistical techniques
  27. 27. Kaplan, Stanley, and B. John Garrick. "On the quantitative definition of risk."Riskanalysis1.1 (1981): 11-27.
  29. 29. SCORINGGUIDELINES Scoring Guidelines
  30. 30. RISK GRADING Sample Risk grading considering impact and likelihood of occurrence 5 5 10 15 20 25 4 4 8 12 16 20 3 3 6 9 12 15 2 2 4 6 8 10 1 1 2 3 4 5 1 2 3 4 5 Likelihood of occurrence Impact Risk Scores Red High Risk Yellow Moderate Risk Green Low Risk High Impact- High likelihood of occurrence : Needs adequate model risk control measures to mitigate risk High Impact – Low likelihood of occurrence: Address through model risk control measures and contingency plans Low Impact – High likelihood of occurrence : Lower priority model risk control measures Low Impact – Low likelihood of occurrence: Least priority model risk control measures
  31. 31. LEVERAGINGMACHINELEARNING Machine learning techniques applied to Quantifying Model risk 1.Clustering to bucket “similar” risks •Identifying training opportunities and best practices for model development 2.K-Nearest Neighbor (k-NN) to automatically derive risk scores •Leveraging expert scoring to help prioritize issues 3.Conjoint analysis •Identifying what combination of a limited number of attributes is most influential on respondent choice or decision making
  32. 32. Quantifying Model Risk Wilmott Magazine January 2014
  34. 34. MODELVERIFICATIONVSMODELVALIDATION Model Verification vs Model Validation Verification is defined as: “The process of determining that a model or simulation implementation and its associated data accurately represent the developer’s conceptual description and specifications”. Validation is defined as: “The process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model”. Ref: DoDModeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), DoDInstruction 5000.61, December 9, 2009.
  35. 35. MODELVERIFICATIONPROCESS Elements of Model Verification The Model Verification process
  36. 36. 300 400 500 600 700 800 900 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Strike Maturity Vol Fitted Vol Vol vs. K, T Model Interpolation and Extrapolation What if Model Application domain outside of Model Validation domain ?
  37. 37. THEMODELVERIFICATION The Model Verification Process 1. Scoping the Model Verification Process •Model Scope •Model Specification -> Model Design -> Model Implementation •Acceptance criteria 2. Model Implementation Checks •The Levers for the model: Input /Output Analysis •Failure modes •Determining the degree of correctness 3. Model Policy and Process Checks 4. Model Verification Reporting
  38. 38. The Role of Model Verification in Model Risk Management Oct 2014 To be published in Wilmott Magazine Nov 2014
  40. 40. STRESS TEST & SCENARIO TEST Stress Tests and Scenario Tests Figure courtesy:
  41. 41. DEFINITIONS Definitions 1.Scenarios : “A scenario is a possible future environment, either at a point in time or over a period of time.” “Considers the impact of a combination of events“ 2.Sensitivity Analysis: “A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “ 3.Stress Testing: Analysis of the impact of single extreme events (or risk factors) Ref:
  42. 42. STRESSTESTINGANDMODELRISKMANAGEMENT Regulatory efforts SR 11-7 says “Banks benefit from conducting model stress testing to check performance over a wide range of inputs and parameter values, including extreme values, to verify that the model is robust” In fact, SR14-03explicitly calls for all models used for Dodd- Frank Act Company-Run Stress Tests must fall under the purview of Model Risk Management. In addition SR12-07calls for incorporating validation or other type of independent review of the stress testing framework to ensure the integrity of stress testing processes and results. Stress Testing and Model Risk Management
  43. 43. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING Characteristics of Stress and Scenario testing 1.Difficult to build parametric models –Simulation driven approach necessary 2.Parameter space can explode easily 3.Tests independent of each other (Embarrassingly parallel) 4.Complete test-coverage –Useless 5.Human intervention required 6.Tests to be designed and customized for the companies needs considering portfolios, organization structure and regulatory obligations Ref:
  44. 44. LEVERAGINGTECHNOLOGYTOSCALESTRESSANDSCENARIOTESTING Leveraging Technology to scale Stress and Scenario Testing •Advances in technology in the last two decades have significantly enhanced the toolsets quants have to develop, test and scale innovative quantitative applications •Simulation and stress testing in risk management are vastly scalable due to innovations in parallel and distributed computing •Restricting the number of tests due to lack of technological resources not an excuse Ref: Gaining the Technology Edge:
  45. 45. LEVERAGETECHNOLOGYTOSCALEANALYTICS Leverage technology to scale analytics 1.64 bit systems : Addressable space ~ 8TB 2.Multi-core processors : Explicit and Implicit Multi-threading 3.Parallel and Distributed Computing : Leverage commodity/Specialized hardware to scale problems 4.General-purpose computing on graphics processing units : Use graphics cards to scale your algorithms 5.Cloud Computing Ref: Gaining the Technology Edge:
  46. 46. MODELRISKMITIGATIONTHROUGHSYSTEMATICSTRESSTESTING Model Risk Mitigation through systematic stress testing 1.Model Implementation •Does it actually work for all intended use cases? 2.Model parameter testing •Number of parameters •How many Scenarios 3.Model Applicability 4.Model Benchmarking against Reference Implementation •Python vs MATLAB 5.Model Migration (version) •Regression Testing v1.0 to v2.0 6.Model Use case validation •Can we use the results to make decisions?
  47. 47. DEMO
  48. 48. Value-at-Risk & Conditional Value-at-Risk •VaR : The predicted maximum loss of a portfolio with a specified probability level (e,g., 95%) over a certain period of time (e.g. one day) •CVaR(Expected Shortfall) : The expected value of the loss given that the loss exceeds VaR Ref: Optimization Methods in Finance by Gerard Cornuejols, RehaTutuncu, Cambridge University Press Image courtesy:
  49. 49. How to Implement VaR and CVaR? Methodology: •Historical •Variance-Covariance method •Monte-Carlo simulations All 3 models are implemented in: •MATLAB •Python •R
  50. 50. Methods to compute VaR and CVaR •Historical method Image Courtesy: 1.Compute Daily Returns and sort them in ascending order 2.For a given confidence level (α, e.g. 95%) , find VaRα(X) such that: P(X<= VaRα(X)) = α 3.Compute CVaRby taking the average loss of the tail
  51. 51. Methods to compute VaR and CVaR •Variance-Covariance Method Image Courtesy: 1.Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (μ & σ) 2.For a given confidence level (α, e.g. 95%) , find VaRα(X) such that: P(X<= VaRα(X)) = α Example 95% => -1.65*σ 3.Compute CVaRby taking the average loss of the tail (See Yamaiand Yoshiba
  52. 52. Methods to compute VaR and CVaR •Monte-Carlo Simulations Image Courtesy: 1.Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (μ & σ) 2.Run n Monte-Carlo simulations with random numbers drawn from a normal distribution described by (μ & σ) 3.For a given confidence level (α, e.g. 95%) , find VaRα(X) such that: P(X<= VaRα(X)) = α 4.Compute CVaRby taking the average loss of the tail
  53. 53. Model Summary VaR Model Historical Method Python R MATLAB Variance- Covariance Method Python R MATLAB Monte-Carlo Simulations Python R MATLAB Given: 1.Historical Daily price time series for a specified time period 2.Constituents of a 3-asset long-only portfolio Compute: VaR and CVaR
  54. 54. Model Verification criteria 1.Model Benchmarking •MATLAB vs Python vs R 2.Parameter sweeps •Different Confidence Intervals (90, 95, 99) 3.Model Convergence •How many simulations needed ? (100, 500, 1000) 4.How do different methods compare? •Historical vs Variance-Covariance vs Monte-Carlo methods
  55. 55. WHEREHAVEWEUSEDTHISAPPROACH? Example 1: External Model Verification •Large bank, no formal model validation team •Model to estimate future cash flows factoring defaults and many other parameters •20+ parameters, 100+ assets •Impossible to manually stress test the model thoroughly Solution : •Identified important parameters and ran more than 10000 tests automatically •Identified multiple issues where model failed especially when handling edge cases
  56. 56. WHEREHAVEWEUSEDTHISAPPROACH? Example 2: Energy Forecasting and Risk Management •Energy company with more than 100,000 customers •Model to estimate future energy usage based on historical usage, temperature forecasts etc. •5+ parameters, 100K+ assets •Data-Driven model to help source wholesale energy and to hedge exposures Solution : •Working on building 100K+ models that will be clustered later •Need to run the model monthly factoring changing portfolio characteristics and new market information
  57. 57. FEATURES 1.Asynchronous 2.Language-agnostic model parameter specification 3.Can support massive scale of tests 4.Systematic and Test results archived for reproducibility Coming soon! Model Definition Language •To specify model parameters ModelRiskEngine Optimization to leverage infrastructure in the cloud •Resource constraints: •Budget & Time constraints •Priority queues and jobs •Dynamic scaling and load balancing
  58. 58. MODELRISKANALYTICSBETAPROGRAM If you are interested in being a part of the Beta program: If you are interested in knowing more about the Model Risk Analytics system Or Want to be part of the Beta program, email: sri@quantuniversity.comto be added to the interest listModel Risk Analytics Beta program
  60. 60. BESTPRACTICESFORMODELRISKMANAGEMENT Best Practices in Model Risk Management (MRM) 1. Adopt a Framework driven approach to MRM 2. Customize the MRM program 3. Clearly define roles and responsibilities 4. Integrate MRM effectively into the model life cycle 5. Don’t reinvent the wheel 6. All models weren’t born equal 7. A checklist is your friend 8. Monitor the health of the models and program 9. Leverage the domain knowledge of your models 10. Own the MRM program
  61. 61. The Decalogue Wilmott Magazine July 2014
  62. 62. MODELRISKMANAGEMENTCOURSE Coming soon: A 2-day course in Model Risk Management with hands-on case studies Email: sri@quantuniversity.comto be added to the interest listModel Risk Management for Quants
  63. 63. REFERENCES Quantifying Model Risk Wilmott Magazine January 2014 The Decalogue Wilmott Magazine July 2014 To get copies of these articles, go to: OR Email: The Role of Model Verification in Model Risk Management Oct 2014
  64. 64. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not bedistributed or used in any other publication without the prior written consent of QuantUniversity LLC.