Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- A Framework Driven Approach to Mode... by QuantUniversity 3375 views
- Missing data handling by QuantUniversity 1960 views
- Credit risk meetup by QuantUniversity 555 views
- Energy analytics with Apache Spark ... by QuantUniversity 1241 views
- Scaling Analytics with Apache Spark by QuantUniversity 1037 views
- Outlier analysis for Temporal Datasets by QuantUniversity 5169 views

1,372 views

Published on

Published in:
Economy & Finance

No Downloads

Total views

1,372

On SlideShare

0

From Embeds

0

Number of Embeds

87

Shares

0

Downloads

67

Comments

0

Likes

2

No embeds

No notes for slide

- 1. MODEL RISK MANAGEMENT: BEST PRACTICES FOR MATLAB MODELS Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC. Sri Krishnamurthy, CFA Founder and CEO www.QuantUniversity.com August 14th 2016
- 2. AGENDA About QuantUniversity Stress Testing and Model Risk: A Brief Introduction The Model Risk Analytics Solution 1 2 3 4 Agenda Case Study5 Challenges to implementing Model Risk Management
- 3. ANALYTICS, RISK MANAGEMENT & STRESS TESTING FOR FINANCIAL INSTIT UTIONS - ADVISORY SERVICES - PLATFORM TO STRESS TESTING, ANALYTICS AND RISK ASSESSMENT - ARCHITECTURE REVIEW, TRAINING AND AUDITS
- 4. INTRODUCTION • Founder of QuantUniversity. • Advisory and Consultancy for Financial Analytics and Model Risk • Prior Experience at MathWorks, Citigroup and Endeca and 25+ financial services and energy customers. • Charted Financial Analyst and Certified Analytics Professional • Teaches Analytics in the Babson College MBA program and at Northeastern University, Boston Sri Krishnamurthy Founder and CEO 4
- 5. STRESS TESTING AND ANALYTICS– A BRIEF INTRODUCTION
- 6. STRESSTESTSINTHENEWS “Of the 33 banks the Fed tested—which included the largest U.S. banks, like Bank of America, Citigroup, and Wells Fargo.. all were deemed strong enough to weather a severe economic meltdown without any help from the government.”
- 7. MODELVALIDATION “Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1] “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk and Validation
- 8. STRESSTESTINGANDMODELRISKMANAGEMENT Regulatory efforts EIOPA: Europe-wide stress test for the insurance sector What’s driving Stress Testing and Model Risk Management efforts? https://eiopa.europa.eu/Publications/Surveys/eiopa-14- 215_stress_test_2014_specifications.pdf
- 9. STRESSTEST&SCENARIOTEST Stress Tests and Scenario Tests Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 10. DEFINITIONS Definitions 1. Scenarios : “A scenario is a possible future environment, either at a point in time or over a period of time.” “Considers the impact of a combination of events“ 2. Sensitivity Analysis: “A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “ 3. Stress Testing: • Analysis of the impact of single extreme events (or risk factors) • Extreme but plausible Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 11. STRESSTESTINGANDMODELRISKMANAGEMENT Model verification: • Migration of code from an older code-base/ from another language • Regression Testing Model Deployment: • Model tests to be done prior to deployment Model Replication: • Models that replicate alternate models or functionality/strategy Model Back testing: • Test how a model performs under different conditions with historical data What’s involved with Model Risk Management efforts?
- 12. MODELLIFECYCLEMANAGEMENTKEYTOMRM How’s Model Risk Management implemented in the company? Ref: The Decalogue http://quantuniversity.com/w9.html
- 13. CHALLENGES TO IMPLEMENTING MODEL RISK MANAGEMENT
- 14. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING Characteristics of Stress and Scenario testing 1. Difficult to build parametric models – Simulation driven approach necessary 2. Parameter space can explode easily 3. Large number (10s of thousands) of tests needed 4. Human intervention required 5. Company needs are different : Customization required 6. Significant coordination and engagement with multiple groups needed Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 15. IMPLEMENTINGSYSTEMATICSTRESSTESTINGISCHALLENGING Challenges to systematic stress testing 1. Model Implementation • Does it actually work for all intended use cases? 2. Model parameter testing • How many parameters ? • How many scenarios ? 3. Model Applicability 4. Model Benchmarking against Reference Implementation • R vs Proprietary vs MATLAB 5. Model Migration (version) • Regression Testing v1.0 to v2.0 6. Model use case validation • Can we use the results to make decisions?
- 16. MODELLIFECYCLEMANAGEMENTISSTILLEVOLVING Challenges in Model Life Cycle Management 1. Model risk management considered as a regulatory/compliance task 2. No one-size-fits-all solution 3. Lack of transparency, standardization and centralized model management 4. Disparate and disconnected systems (SVN, Sharepoint, Excel sheets, custom systems) 5. Difficult to track and audit changes 6. Metrics to quantify model risk not established
- 17. MODELRISK ANALYTICS SOLUTION
- 18. LEVERAGETECHNOLOGYTOSCALEANALYTICS Leverage technology to scale analytics 1.64 bit systems : Addressable space ~ 8TB 2.Multi-core processors : Explicit and Implicit Multi-threading 3.Parallel and Distributed Computing : Leverage commodity/specialized hardware to scale problems 4.Micro services-based solution for fast deployment and replication 5.Cloud Computing Ref: Gaining the Technology Edge: http://www.quantuniversity.com/w5.html
- 19. ADDRESSINGSTRESSTESTINGCHALLENGES How can we address challenges in Stress testing? 1. Leverage hardware for scaling 1. Enable developers to plan for stress tests 2. Enable developers to scale without having to rewrite code 2. Enable developers to run their code 1. Locally 2. On Enterprise Servers 3. Cloud
- 20. ADDRESSINGSTRESSTESTINGCHALLENGES How can we address challenges in Stress testing? 3. Facilitate exhaustive and robust testing without having to get lost in details 1. Enable testers to specify their tests and configurations 2. Enable benchmarking and comparisons with other models 4. Facilitate deployment of models built using different frameworks including legacy code 1. Provide a choice of models (which could be built in different languages) 2. Enable benchmarking and comparisons with models built using other languages
- 21. ADDRESSINGMODELLIFECYCLEMANAGEMENTCHALLENGES How can we address challenges of Model Lifecycle Management? 1. Provide a single comprehensive environment to manage model validation tests 1. Enable integration with source control systems 2. Enable model validators to create, run and assess tests 2. Facilitate all stages of model lifecycle management in a single environment 1. Allow access to all model related artifacts (documentation, tests and results) 2. Provide ability to collaborate with all stakeholders responsible for model risk 3. Enable audit trails and analytics to manage model risk
- 22. MODELRISKANALYTICSSYSTEM ModelRisk Analytics System
- 23. MODELRISKANALYTICSSOLUTION Model Risk Analytics Solution 1. Distributed System to support Model Verification, Validation, Stress and Back Testing 2. Supports MATLAB and other open source packages like R, Python, SVN 3. Architecture leverages state-of-the-art asynchronous computing modes 4. Package management 1. Using Custom Client/ Images 2. Using Docker(Different containers for different apps) 5. Supports synchronous and asynchronous modes 6. Built for the cloud but can work on any cloud/internal infrastructure (private or public)
- 24. CASE STUDY
- 25. Value-at-Risk & Conditional Value-at-Risk • VaR : The predicted maximum loss of a portfolio with a specified probability level (e.g., 95%) over a certain period of time (e.g. one day) • CVaR (Expected Shortfall) : The expected value of the loss given that the loss exceeds VaR Ref: Optimization Methods in Finance by Gerard Cornuejols, Reha Tutuncu, Cambridge University Press Image courtesy: http://www.imes.boj.or.jp/english/publication/mes/2002/me20-1-3.pdf )
- 26. How to Implement VaR and CVaR? Methodology: • Historical • Variance-Covariance method • Monte-Carlo simulations Models are implemented in: • MATLAB – Production Model • Python – Alternate model developed by a 3rd-party vendor • R – Reference/Benchmark model from a paper
- 27. Methods to compute VaR and CVaR • Historical method Image Courtesy: http://www.investopedia.com/articles/04/092904.asp 1. Compute Daily Returns and sort them in ascending order 2. For a given confidence level (e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α 3. Compute CVaR by taking the average loss of the tail
- 28. Methods to compute VaR and CVaR • Variance-Covariance Method Image Courtesy: http://www.investopedia.com/articles/04/092904.asp 1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ) 2. For a given confidence level (e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α Example 95% => -1.65* σ 3. Compute CVaR by taking the average loss of the tail (See Yamai and Yoshiba http://www.imes.boj.or.jp/english/public ation/mes/2002/me20-1-3.pdf )
- 29. Methods to compute VaR and CVaR • Monte-Carlo Simulations Image Courtesy: http://www.investopedia.com/articles/04/092904.asp 1. Compute Daily Returns and fit a Normal distribution to obtain mean and Standard Deviation (µ & σ) 2. Run n Monte-Carlo simulations with random numbers drawn from a normal distribution described by (µ & σ) 3. For a given confidence level (e.g. 95%) , find VaRα (X) such that: P(X<= VaR α(X)) = α 4. Compute CVaR by taking the average loss of the tail
- 30. Model Summary VaR Model Historical Method Python R MATLAB Variance- Covariance Method Python R MATLAB Monte-Carlo Simulations Python R MATLAB Given: 1. Historical Daily price time series for a specified time period 2. Constituents of a 3-asset long-only portfolio Compute: VaR and CVaR
- 31. Model Verification criteria 1. Model Benchmarking • MATLAB production model vs Alternate Python model vs Reference R model 2. Parameter sweeps • Different Confidence Intervals (90%, 95%, 99%) 3. Model Convergence • How many simulations needed ? (100, 500, 1000) 4. How do different methods compare? • Historical vs Variance-Covariance vs Monte-Carlo methods
- 32. VarModel.m
- 33. ModelRisk Studio facilitates easy structuring of tests
- 34. Input parameters can be specified for each model
- 35. ModelRisk Analytics allows you to run the model in multiple environments
- 36. The Model Testing results can be viewed in an interactive dashboard
- 37. Model Issues can be tracked, risks evaluated and addressed appropriately
- 38. FEATURES Coming soon! ModelRisk Model Definition Language • To specify model parameters ModelRisk Engine Optimization to leverage infrastructure in the cloud • Resource constraints: • Budget & Time constraints • Priority queues and jobs • Dynamic scaling and load balancing ModelRisk Audit to enable periodic testing and ongoing monitoring of your models
- 39. MODELRISKANALYTICSBETAPROGRAM If you are interested in being a part of the Beta program: If you are interested in knowing more about the Model Risk Analytics system Or Want to be part of the Beta program, email: sri@quantuniversity.com to be added to the interest list Model Risk Analytics Beta program
- 40. MODELRISKMANAGEMENTWORKSHOP Also available: A 2-day workshop in Model Risk Management with hands-on case studies Email: sri@quantuniversity.com to be added to the interest list Stress Testing and Model Risk Management for Quants
- 41. ANOMALYDETECTIONWORKSHOP Coming soon: A 2-day course in methods and best practices in Anomaly Detection September 19th, 20th , New York Visit www.analyticscertificate.com/Anomaly for more details Anomaly Detection Core Techniques and Best Practices
- 42. WHITEPAPERS To get copies of these articles, go to: http://www.quantuniversity.com/articlerequest.html OR Email: sri@quantuniversity.com
- 43. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy www.QuantUniversity.com Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC.

No public clipboards found for this slide

Be the first to comment