Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Model Risk Management : Best Practices


Published on

Model risk and the importance of model risk management has gotten significant attention in the last few years. As financial companies increase their reliance on quants and quantitative models for decision making, they are increasingly exposed to model risk and are looking for ways to mitigate it. The financial crisis of 2008 and various high profile financial accidents due to model failures has brought model risk management to the forefront as an important topic to be addressed. Many regulatory efforts (Solvency II, Basel III, Dodd-Frank etc.) have been initiated obligating banks and financial institutions to incorporate formal model risk management programs to address model risk. In this talk, we will discuss the key aspects of model verification and validation and discuss best practices for stress and scenario testing for model verification and validation for MATLAB-based models. These best practices are meant to provide practical tips for companies embarking on a formal model risk management program or enhancing their model risk methodologies to address the new realities.

Published in: Economy & Finance
  • Be the first to comment

Model Risk Management : Best Practices

  1. 1. MODEL RISK MANAGEMENT: BEST PRACTICES FOR MATLAB MODELS Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC. Sri Krishnamurthy, CFA Founder and CEO August 14th 2016
  2. 2. AGENDA About QuantUniversity Stress Testing and Model Risk: A Brief Introduction The Model Risk Analytics Solution 1 2 3 4 Agenda Case Study5 Challenges to implementing Model Risk Management
  4. 4. INTRODUCTION • Founder of QuantUniversity. • Advisory and Consultancy for Financial Analytics and Model Risk • Prior Experience at MathWorks, Citigroup and Endeca and 25+ financial services and energy customers. • Charted Financial Analyst and Certified Analytics Professional • Teaches Analytics in the Babson College MBA program and at Northeastern University, Boston Sri Krishnamurthy Founder and CEO 4
  6. 6. STRESSTESTSINTHENEWS “Of the 33 banks the Fed tested—which included the largest U.S. banks, like Bank of America, Citigroup, and Wells Fargo.. all were deemed strong enough to weather a severe economic meltdown without any help from the government.”
  7. 7. MODELVALIDATION “Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1] “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk and Validation
  8. 8. STRESSTESTINGANDMODELRISKMANAGEMENT Regulatory efforts EIOPA: Europe-wide stress test for the insurance sector What’s driving Stress Testing and Model Risk Management efforts? 215_stress_test_2014_specifications.pdf
  9. 9. STRESSTEST&SCENARIOTEST Stress Tests and Scenario Tests Figure courtesy:
  10. 10. DEFINITIONS Definitions 1. Scenarios : “A scenario is a possible future environment, either at a point in time or over a period of time.” “Considers the impact of a combination of events“ 2. Sensitivity Analysis: “A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “ 3. Stress Testing: • Analysis of the impact of single extreme events (or risk factors) • Extreme but plausible Ref:
  11. 11. STRESSTESTINGANDMODELRISKMANAGEMENT Model verification: • Migration of code from an older code-base/ from another language • Regression Testing Model Deployment: • Model tests to be done prior to deployment Model Replication: • Models that replicate alternate models or functionality/strategy Model Back testing: • Test how a model performs under different conditions with historical data What’s involved with Model Risk Management efforts?
  12. 12. MODELLIFECYCLEMANAGEMENTKEYTOMRM How’s Model Risk Management implemented in the company? Ref: The Decalogue
  14. 14. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING Characteristics of Stress and Scenario testing 1. Difficult to build parametric models – Simulation driven approach necessary 2. Parameter space can explode easily 3. Large number (10s of thousands) of tests needed 4. Human intervention required 5. Company needs are different : Customization required 6. Significant coordination and engagement with multiple groups needed Ref:
  15. 15. IMPLEMENTINGSYSTEMATICSTRESSTESTINGISCHALLENGING Challenges to systematic stress testing 1. Model Implementation • Does it actually work for all intended use cases? 2. Model parameter testing • How many parameters ? • How many scenarios ? 3. Model Applicability 4. Model Benchmarking against Reference Implementation • R vs Proprietary vs MATLAB 5. Model Migration (version) • Regression Testing v1.0 to v2.0 6. Model use case validation • Can we use the results to make decisions?
  16. 16. MODELLIFECYCLEMANAGEMENTISSTILLEVOLVING Challenges in Model Life Cycle Management 1. Model risk management considered as a regulatory/compliance task 2. No one-size-fits-all solution 3. Lack of transparency, standardization and centralized model management 4. Disparate and disconnected systems (SVN, Sharepoint, Excel sheets, custom systems) 5. Difficult to track and audit changes 6. Metrics to quantify model risk not established
  18. 18. LEVERAGETECHNOLOGYTOSCALEANALYTICS Leverage technology to scale analytics 1.64 bit systems : Addressable space ~ 8TB 2.Multi-core processors : Explicit and Implicit Multi-threading 3.Parallel and Distributed Computing : Leverage commodity/specialized hardware to scale problems 4.Micro services-based solution for fast deployment and replication 5.Cloud Computing Ref: Gaining the Technology Edge:
  19. 19. ADDRESSINGSTRESSTESTINGCHALLENGES How can we address challenges in Stress testing? 1. Leverage hardware for scaling 1. Enable developers to plan for stress tests 2. Enable developers to scale without having to rewrite code 2. Enable developers to run their code 1. Locally 2. On Enterprise Servers 3. Cloud
  20. 20. ADDRESSINGSTRESSTESTINGCHALLENGES How can we address challenges in Stress testing? 3. Facilitate exhaustive and robust testing without having to get lost in details 1. Enable testers to specify their tests and configurations 2. Enable benchmarking and comparisons with other models 4. Facilitate deployment of models built using different frameworks including legacy code 1. Provide a choice of models (which could be built in different languages) 2. Enable benchmarking and comparisons with models built using other languages
  21. 21. ADDRESSINGMODELLIFECYCLEMANAGEMENTCHALLENGES How can we address challenges of Model Lifecycle Management? 1. Provide a single comprehensive environment to manage model validation tests 1. Enable integration with source control systems 2. Enable model validators to create, run and assess tests 2. Facilitate all stages of model lifecycle management in a single environment 1. Allow access to all model related artifacts (documentation, tests and results) 2. Provide ability to collaborate with all stakeholders responsible for model risk 3. Enable audit trails and analytics to manage model risk
  22. 22. MODELRISKANALYTICSSYSTEM ModelRisk Analytics System
  23. 23. MODELRISKANALYTICSSOLUTION Model Risk Analytics Solution 1. Distributed System to support Model Verification, Validation, Stress and Back Testing 2. Supports MATLAB and other open source packages like R, Python, SVN 3. Architecture leverages state-of-the-art asynchronous computing modes 4. Package management 1. Using Custom Client/ Images 2. Using Docker(Different containers for different apps) 5. Supports synchronous and asynchronous modes 6. Built for the cloud but can work on any cloud/internal infrastructure (private or public)
  24. 24. CASE STUDY
  25. 25. Value-at-Risk & Conditional Value-at-Risk • VaR : The predicted maximum loss of a portfolio with a specified probability level (e.g., 95%) over a certain period of time (e.g. one day) • CVaR (Expected Shortfall) : The expected value of the loss given that the loss exceeds VaR Ref: Optimization Methods in Finance by Gerard Cornuejols, Reha Tutuncu, Cambridge University Press Image courtesy: )
  26. 26. How to Implement VaR and CVaR? Methodology: • Historical • Variance-Covariance method • Monte-Carlo simulations Models are implemented in: • MATLAB – Production Model • Python – Alternate model developed by a 3rd-party vendor • R – Reference/Benchmark model from a paper
  27. 27. Methods to compute VaR and CVaR • Historical method Image Courtesy: 1. Compute Daily Returns and sort them in ascending order 2. For a given confidence level (e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α 3. Compute CVaR by taking the average loss of the tail
  28. 28. Methods to compute VaR and CVaR • Variance-Covariance Method Image Courtesy: 1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ) 2. For a given confidence level (e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α Example 95% => -1.65* σ 3. Compute CVaR by taking the average loss of the tail (See Yamai and Yoshiba ation/mes/2002/me20-1-3.pdf )
  29. 29. Methods to compute VaR and CVaR • Monte-Carlo Simulations Image Courtesy: 1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ) 2. Run n Monte-Carlo simulations with random numbers drawn from a normal distribution described by (µ & σ) 3. For a given confidence level (e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α 4. Compute CVaR by taking the average loss of the tail
  30. 30. Model Summary VaR Model Historical Method Python R MATLAB Variance- Covariance Method Python R MATLAB Monte-Carlo Simulations Python R MATLAB Given: 1. Historical Daily price time series for a specified time period 2. Constituents of a 3-asset long-only portfolio Compute: VaR and CVaR
  31. 31. Model Verification criteria 1. Model Benchmarking • MATLAB production model vs Alternate Python model vs Reference R model 2. Parameter sweeps • Different Confidence Intervals (90%, 95%, 99%) 3. Model Convergence • How many simulations needed ? (100, 500, 1000) 4. How do different methods compare? • Historical vs Variance-Covariance vs Monte-Carlo methods
  32. 32. VarModel.m
  33. 33. ModelRisk Studio facilitates easy structuring of tests
  34. 34. Input parameters can be specified for each model
  35. 35. ModelRisk Analytics allows you to run the model in multiple environments
  36. 36. The Model Testing results can be viewed in an interactive dashboard
  37. 37. Model Issues can be tracked, risks evaluated and addressed appropriately
  38. 38. FEATURES Coming soon! ModelRisk Model Definition Language • To specify model parameters ModelRisk Engine Optimization to leverage infrastructure in the cloud • Resource constraints: • Budget & Time constraints • Priority queues and jobs • Dynamic scaling and load balancing ModelRisk Audit to enable periodic testing and ongoing monitoring of your models
  39. 39. MODELRISKANALYTICSBETAPROGRAM If you are interested in being a part of the Beta program: If you are interested in knowing more about the Model Risk Analytics system Or Want to be part of the Beta program, email: to be added to the interest list Model Risk Analytics Beta program
  40. 40. MODELRISKMANAGEMENTWORKSHOP Also available: A 2-day workshop in Model Risk Management with hands-on case studies Email: to be added to the interest list Stress Testing and Model Risk Management for Quants
  41. 41. ANOMALYDETECTIONWORKSHOP Coming soon: A 2-day course in methods and best practices in Anomaly Detection September 19th, 20th , New York Visit for more details Anomaly Detection Core Techniques and Best Practices
  42. 42. WHITEPAPERS To get copies of these articles, go to: OR Email:
  43. 43. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC.