Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- Model Risk Management : Best Practices by QuantUniversity 1010 views
- Deep learning Tutorial - Part II by QuantUniversity 5642 views
- Deep learning - Part I by QuantUniversity 7732 views
- Deep learning and Apache Spark by QuantUniversity 6425 views
- Koja je Vaša strategija za obraćanj... by Tatjana Mamula 1092 views
- FitchLearning QuantUniversity Model... by QuantUniversity 979 views

2,925 views

Published on

Published in:
Data & Analytics

No Downloads

Total views

2,925

On SlideShare

0

From Embeds

0

Number of Embeds

1,462

Shares

0

Downloads

0

Comments

0

Likes

2

No embeds

No notes for slide

- 1. A FRAMEWORK DRIVEN APPROACH TO MODEL RISK MANAGEMENT Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC. Sri Krishnamurthy, CFA Founder and CEO QuantUniversity LLC. www.QuantUniversity.com DATA ANALYTICS FOR FINANCIAL SERVICES, JULY 22 2014, BOSTON, MA
- 2. AGENDA About Model Risk Analytics Model Risk : A Brief Introduction A Framework driven approach to Model Risk Management Quantifying Model Risk Stress and Scenario Testing in Model Risk Management 1 2 3 4 5 Agenda
- 3. • Founder of QuantUniversity LLC. • Advisory and Consultancy for Model Risk • Prior Experience at MathWorks, Citigroup and Endeca and 25+ customers • Regular Columnist for the Wilmott Magazine • Author of forthcoming book “Financial Modeling: A case study approach” published by Wiley • Charted Financial Analyst and Certified Analytics Professional • Teaches Analytics in the Babson College MBA program Sri Krishnamurthy Founder and CEO SPEAKERBIO
- 4. COMPREHENSIVE MODEL RISK MANAGEMENT FOR FINANCIAL INSTITUTIONS - ADVISORY SERVICES - PLATFORM TO MANAGE MODEL RISK - TRAINING AND AUDITS
- 5. OURSOLUTION Comprehensive Model Risk Assessment Framework Model Risk Quantification Methodology Fully auditable cloud-based Model Risk Management platform 5 5 10 15 20 25 4 4 8 12 16 20 3 3 6 9 12 15 2 2 4 6 8 10 1 1 2 3 4 5 1 2 3 4 5 Likelihoodof occurrence Impact RiskScores Model Governance Model Lifecycle Management Model Risk and Validation Makes measuring risk and prioritizing issues easier Complies with audits and regulatory needs Proven framework for a Model risk platform MODELRISKANALYTICS Customer Validation Industry Acceptance Novel and fast growing
- 6. MODEL RISK – A BRIEF INTRODUCTION
- 7. MODELRISKINTHENEWS Financial accidents have cost companies millions of dollars and was blamed for the financial crisis of 2008 Concerned about systemic risk, regulators have stepped up regulations to setup model risk programs All Banks, Insurance Companies and Credit Rating agencies in the US and EU are affected by these regulations As financial institutions depend on models for decision making, Model Risk Management is critical 8Source: E&Y Survey 69 banks & 6 insurance companies
- 8. CHALLENGES Quantitative models are complex: Measuring model risk is not easy Quantitative systems are complex : Many stakeholders Novelty: Lack of guidance and ambiguity on regulations Quants Risk Portfolio Management IT 1 2 3 Financial institutions face challenges implementing Model Risk Programs SEC = ?
- 9. MODELRISK “Model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk Defined Input Assumptions/Data Processing component Output/ Reporting component
- 10. MODELVALIDATION “Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. “ [1] “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. ” [1] Ref: [1] . Supervisory Letter SR 11-7 on guidance on Model Risk Model Risk and Validation Defined
- 11. A FRAMEWORK DRIVEN APPROACH TO MODEL RISK MANAGEMENT
- 12. MODELRISKMANAGEMENT Elements of a Model Risk Management framework 1. Model Governance structure: Addresses regulatory requirements, roles, responsibilities, oversight, control and escalation procedures 2. Model Lifecycle management: Addresses the processes involved in the design, development, testing, deployment and use of models. Also addresses testing and documentation plans and change management. 3. Model Review and Validation Process: Addresses internal and external model review, validation and ongoing monitoring of models (both qualitative and quantitative)
- 13. MODELGOVERNANCESTRUCTURE Model Governance Structure Model Governance Structure Regulatory guidance and best practices Model Classification Roles and Responsibil ities Oversight and Controls
- 14. MODELLIFECYCLEMANAGEMENT Model Lifecycle Management
- 15. MODELREVIEWANDVALIDATION Model Review and Validation Policy: Model Policy Review Structure: Model Process Review Content: Model Review
- 16. LEVERAGINGTECHNOLOGYANDANALYTICSFOREFFECTIVEMODELRISKMANAGEMENT Leveraging technology and analytics for effective Model Risk Management 1.Quantifying Model Risk: • Classification and Measurement of Model Risk 2.Leveraging technology to scale stress and scenario testing
- 17. QUANTIFYING MODEL RISK
- 18. CHALLENGES Organizational Structure Organization Enterprise Risk Management Compliance Model Research and Development End Users IT How to engage all departments strategically to have a comprehensive view of Model Risk ?
- 19. HOWTODOIT Theory to Practice : How to cross the chasm ? Image Courtesy: http://rednomadoz.blogspot.com.au/ • Theory • Regulations • Local Laws • Practical IT systems • Company policies • Company culture and Best practices
- 20. CLASSIFYINGMODELRISK Classifying Model Risk Class 3 Example: Monte-carlo simulation engine Class 2 Example: Linked-spreadsheet model with dependencies Class 1 Example: Simple Spreadsheets Complexity 1. Class 1 Models: Simple Models typically involving less complex atomic calculations 2. Class 2 Models: Models more complicated than Class 1 models 3. Class 3 Models: Typically involves sophisticated mathematical/statistical techniques
- 21. MODELRISKASSESSMENTFRAMEWORK Model Risk Assessment
- 22. SCORINGGUIDELINES Scoring Guidelines
- 23. RISKGRADING Sample Risk grading considering impact and likelihood of occurrence 5 5 10 15 20 25 4 4 8 12 16 20 3 3 6 9 12 15 2 2 4 6 8 10 1 1 2 3 4 5 1 2 3 4 5 Likelihood of occurrence Impact Risk Scores Red High Risk Yellow Moderate Risk Green Low Risk High Impact- High likelihood of occurrence : Needs adequate model risk control measures to mitigate risk High Impact – Low likelihood of occurrence: Address through model risk control measures and contingency plans Low Impact – High likelihood of occurrence : Lower priority model risk control measures Low Impact – Low likelihood of occurrence: Least priority model risk control measures
- 24. LEVERAGINGMACHINELEARNING Machine learning techniques applied to Quantifying Model risk 1. Clustering to bucket “similar” risks • Identifying training opportunities and best practices for model development 2. K-Nearest Neighbor (k-NN) to automatically derive risk scores • Leveraging expert scoring to help prioritize issues 3. Conjoint analysis • Identifying what combination of a limited number of attributes is most influential on respondent choice or decision making
- 25. STRESS TESTING AND SCENARIO TESTING TO EVALUATE MODEL RISK
- 26. STRESSTEST&SCENARIOTEST Stress Tests and Scenario Tests Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 27. DEFINITIONS Definitions 1. Scenarios : “A scenario is a possible future environment, either at a point in time or over a period of time.” “Considers the impact of a combination of events“ 2. Sensitivity Analysis: “A sensitivity is the effect of a set of alternative assumptions regarding a future environment. “ 3. Stress Testing: Analysis of the impact of single extreme events (or risk factors) Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 28. CONSIDERATIONSFORMANAGINGUNCERTAINTY Considerations for managing uncertainity Case One: Single unknown parameter Example : Volatility Case Two: Two Unknown Parameters Example : Uncertain about mean μ as well as the standard deviation σ Case Three: Unknown correlation(s) Case Four: Mixing parameter and Distribution risk Ref: Wiley: Measuring Market Risk, 2nd Edition - Kevin Dowd
- 29. HANDLINGLARGEDIMENSIONALDATASETS Handling large dimensional data • Variable Selection methods (where a subset of dimensions is chosen) • Reducing dimensions (where variables are transformed to a smaller set of new variables) Ref: Dealing with Dimensionality in large datasets http://www.quantuniversity.com/w8.html
- 30. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING Characteristics of Stress and Scenario testing 1. Difficult to build parametric models – Simulation driven approach necessary 2. Parameter space can explode easily 3. Tests independent of each other (Embarrassingly parallel) 4. Complete test-coverage – Useless 5. Human intervention required 6. Tests to be designed and customized for the companies needs considering portfolios, organization structure and regulatory obligations Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf
- 31. LEVERAGINGTECHNOLOGYTOSCALESTRESSANDSCENARIOTESTING Leveraging Technology to scale Stress and Scenario Testing • Advances in technology in the last two decades have significantly enhanced the toolsets quants have to develop, test and scale innovative quantitative applications • Simulation and stress testing in risk management are vastly scalable due to innovations in parallel and distributed computing • Restricting the number of tests due to lack of technological resources not an excuse Ref: Gaining the Technology Edge: http://www.quantuniversity.com/w5.html
- 32. LEVERAGETECHNOLOGYTOSCALEANALYTICS Leverage technology to scale analytics 1.64 bit systems : Addressable space ~ 8TB 2.Multi-core processors : Explicit and Implicit Multi-threading 3.Parallel and Distributed Computing : Leverage commodity/Specialized hardware to scale problems 4.General-purpose computing on graphics processing units : Use graphics cards to scale your algorithms 5.Cloud Computing Ref: Gaining the Technology Edge: http://www.quantuniversity.com/w5.html
- 33. BESTPRACTICESTOLEVERAGETECHNOLOGYFORSTRESSTESTING Best practices to leverage technology for Stress testing 1. Test Plan to optimize models for technology use 2. Parameter mapping plan to scale stress tests based on use cases 3. Aggregation of Test results 1. Pass/Fail 2.Automatic validation and Benchmarking of results 3.Analytics to infer the state and maturity of the model 4.Model Monitoring Test plan 1.Monitoring of parameters 2.Monitoring of test adequacy 5.Integrate Testing in the Model Lifecycle Development process
- 34. REFERENCES Quantifying Model Risk Wilmott Magazine January 2014 The Decalogue Wilmott Magazine July 2014 Copies can be downloaded at : http://www.quantuniversity.com/w6.html http://www.quantuniversity.com/w9.html
- 35. SUMMARY Model Risk : A Brief Introduction A Framework driven approach to Model Risk Management Quantifying Model Risk Stress and Scenario Testing in Model Risk Management 1 2 3 4 Summary
- 36. UPCOMINGCOURSES Upcoming courses 1. Data Science and Analytics for Quants 2. Model Risk Management Covers analytical techniques and Big Data methodologies in finance. Covers Model Risk Management framework, analytical techniques and case studies Register your interest at www.quantuniversity.com
- 37. Thank you! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy www.QuantUniversity.com Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC.

No public clipboards found for this slide

Be the first to comment