Model risk and the importance of model risk management has gotten significant attention in the last few years. As financial companies increase their reliance on quants and quantitative models for decision making, they are increasingly exposed to model risk and are looking for ways to mitigate it. The financial crisis of 2008 and various high profile financial accidents due to model failures has brought model risk management to the forefront as an important topic to be addressed. Many regulatory efforts (Solvency II, Basel III, Dodd-Frank etc.) have been initiated obligating banks and financial institutions to incorporate formal model risk management programs to address model risk. Regulatory agencies have issued guidance letters and supervisory insights to assist companies in developing model risk management programs. In the United States, as the Dodd-Frank act is implemented, newer guidance letters have been issued that emphasize model risk management. Despite these efforts, in practice, financial companies continue to struggle in formulating and developing a model risk management program. A lot of companies acknowledge and understand the model risk management guidelines in spirit but have practical challenges in implementing these guidance letters. In our prior article on model risk , we discussed many drivers to address model risk and challenges in integrating model risk into the quant development process. In this talk, we will discuss ten best practices for the implementation of an effective model risk management program. These best practices have evolved from discussions with industry experts and consulting projects we have worked with in the recent years to create robust risk management programs. These best practices meant to provide practical tips for companies embarking on a formal model risk management program or enhancing their model risk methodologies to address the new realities

1. A FRAMEWORK DRIVEN APPROACH TO MODEL RISK MANAGEMENT
Information, data and drawings embodied in this presentation are strictly confidential and are supplied on the understanding that
they will be held confidentially and not disclosed to third parties without the prior written consent of QuantUniversity LLC.
Sri Krishnamurthy, CFA
Founder and CEO
QuantUniversity LLC.
www.QuantUniversity.com
DATA ANALYTICS FOR FINANCIAL SERVICES, JULY 22 2014, BOSTON, MA

2. AGENDA
About Model Risk Analytics
Model Risk : A Brief Introduction
A Framework driven approach to Model Risk Management
Quantifying Model Risk
Stress and Scenario Testing in Model Risk Management
1
2
3
4
5
Agenda

3. • Founder of QuantUniversity LLC.
• Advisory and Consultancy for Model Risk
• Prior Experience at MathWorks, Citigroup and
Endeca and 25+ customers
• Regular Columnist for the Wilmott Magazine
• Author of forthcoming book
“Financial Modeling: A case study approach”
published by Wiley
• Charted Financial Analyst and Certified Analytics
Professional
• Teaches Analytics in the Babson College MBA
program
Sri Krishnamurthy
Founder and CEO
SPEAKERBIO

4. COMPREHENSIVE MODEL RISK MANAGEMENT FOR FINANCIAL INSTITUTIONS
- ADVISORY SERVICES
- PLATFORM TO MANAGE MODEL RISK
- TRAINING AND AUDITS

5. OURSOLUTION
Comprehensive Model Risk Assessment Framework
Model Risk Quantification Methodology
Fully auditable cloud-based Model Risk Management platform
5 5 10 15 20 25
4 4 8 12 16 20
3 3 6 9 12 15
2 2 4 6 8 10
1 1 2 3 4 5
1 2 3 4 5
Likelihoodof occurrence
Impact
RiskScores
Model
Governance
Model
Lifecycle
Management
Model Risk
and Validation
Makes measuring risk and
prioritizing issues easier
Complies with audits and
regulatory needs
Proven framework for a
Model risk platform
MODELRISKANALYTICS
 Customer Validation
 Industry Acceptance
 Novel and fast growing

6. MODEL RISK – A BRIEF INTRODUCTION

8. MODELRISKINTHENEWS
Financial accidents have cost companies millions of dollars and was blamed for the financial crisis of 2008
Concerned about systemic risk, regulators have stepped up regulations to setup model risk programs
All Banks, Insurance Companies and Credit Rating agencies in the US and EU are affected by these regulations
As financial institutions depend on models for decision making, Model Risk Management is critical
8Source: E&Y Survey 69 banks & 6 insurance companies

9. CHALLENGES
Quantitative models are complex: Measuring model risk is not easy
Quantitative systems are complex : Many stakeholders
Novelty: Lack of guidance and ambiguity on regulations
Quants Risk
Portfolio
Management IT
1
2
3
Financial institutions face challenges implementing Model Risk Programs
SEC = ?

10. MODELRISK
“Model refers to a quantitative method, system, or
approach that applies statistical, economic, financial,
or mathematical theories, techniques, and
assumptions to process input data into quantitative
estimates” [1]
Ref:
[1] . Supervisory Letter SR 11-7 on guidance on Model Risk
Model Risk Defined
Input
Assumptions/Data
Processing
component
Output/
Reporting
component

11. MODELVALIDATION
“Model risk is the potential for adverse
consequences from decisions based on incorrect or
misused model outputs and reports. “ [1]
“Model validation is the set of processes and activities
intended to verify that models are performing as
expected, in line with their design objectives and
business uses. ” [1]
Ref:
[1] . Supervisory Letter SR 11-7 on guidance on Model Risk
Model Risk and Validation Defined

12. A FRAMEWORK DRIVEN APPROACH TO MODEL RISK MANAGEMENT

13. MODELRISKMANAGEMENT
Elements of a Model Risk Management framework
1. Model Governance structure: Addresses regulatory requirements, roles,
responsibilities, oversight, control and escalation procedures
2. Model Lifecycle management: Addresses the processes involved in the
design, development, testing, deployment and use of models. Also
addresses testing and documentation plans and change management.
3. Model Review and Validation Process: Addresses internal and external
model review, validation and ongoing monitoring of models (both
qualitative and quantitative)

14. MODELGOVERNANCESTRUCTURE
Model Governance Structure
Model
Governance
Structure
Regulatory
guidance
and best
practices
Model
Classification
Roles and
Responsibil
ities
Oversight and
Controls

15. MODELLIFECYCLEMANAGEMENT
Model Lifecycle Management

16. MODELREVIEWANDVALIDATION
Model Review and Validation
Policy:
Model Policy Review
Structure:
Model Process Review
Content:
Model Review

17. LEVERAGINGTECHNOLOGYANDANALYTICSFOREFFECTIVEMODELRISKMANAGEMENT
Leveraging technology and analytics for effective Model Risk Management
1.Quantifying Model Risk:
• Classification and Measurement of Model Risk
2.Leveraging technology to scale stress and scenario
testing

18. QUANTIFYING MODEL RISK

19. CHALLENGES
Organizational Structure
Organization
Enterprise
Risk
Management
Compliance
Model
Research and
Development
End Users
IT
How to engage all departments strategically to have a
comprehensive view of Model Risk ?

20. HOWTODOIT
Theory to Practice : How to cross the chasm ?
Image Courtesy: http://rednomadoz.blogspot.com.au/
• Theory
• Regulations
• Local Laws
• Practical IT
systems
• Company
policies
• Company
culture and
Best practices

21. CLASSIFYINGMODELRISK
Classifying Model Risk
Class 3
Example: Monte-carlo
simulation engine
Class 2
Example: Linked-spreadsheet
model with dependencies
Class 1
Example: Simple Spreadsheets
Complexity
1. Class 1 Models: Simple Models typically involving less complex atomic
calculations
2. Class 2 Models: Models more complicated than Class 1 models
3. Class 3 Models: Typically involves sophisticated
mathematical/statistical techniques

22. MODELRISKASSESSMENTFRAMEWORK
Model Risk Assessment

23. SCORINGGUIDELINES
Scoring Guidelines

24. RISKGRADING
Sample Risk grading considering impact and likelihood of occurrence
5 5 10 15 20 25
4 4 8 12 16 20
3 3 6 9 12 15
2 2 4 6 8 10
1 1 2 3 4 5
1 2 3 4 5
Likelihood of occurrence
Impact
Risk Scores
Red High Risk
Yellow Moderate Risk
Green Low Risk
High Impact- High likelihood of occurrence : Needs adequate model risk control measures to
mitigate risk
High Impact – Low likelihood of occurrence: Address through model risk control measures
and contingency plans
Low Impact – High likelihood of occurrence : Lower priority model risk control measures
Low Impact – Low likelihood of occurrence: Least priority model risk control measures

25. LEVERAGINGMACHINELEARNING
Machine learning techniques applied to Quantifying Model risk
1. Clustering to bucket “similar” risks
• Identifying training opportunities and best practices for model
development
2. K-Nearest Neighbor (k-NN) to automatically derive risk scores
• Leveraging expert scoring to help prioritize issues
3. Conjoint analysis
• Identifying what combination of a limited number of attributes is
most influential on respondent choice or decision making

26. STRESS TESTING AND SCENARIO TESTING TO EVALUATE MODEL
RISK

27. STRESSTEST&SCENARIOTEST
Stress Tests and Scenario Tests
Figure courtesy: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

28. DEFINITIONS
Definitions
1. Scenarios :
“A scenario is a possible future environment, either at a point in
time or over a period of time.”
“Considers the impact of a combination of events“
2. Sensitivity Analysis:
“A sensitivity is the effect of a set of alternative assumptions
regarding a future environment. “
3. Stress Testing:
Analysis of the impact of single extreme events (or risk factors)
Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

29. CONSIDERATIONSFORMANAGINGUNCERTAINTY
Considerations for managing uncertainity
Case One: Single unknown parameter
Example : Volatility
Case Two: Two Unknown Parameters
Example : Uncertain about mean μ as well as the
standard deviation σ
Case Three: Unknown correlation(s)
Case Four: Mixing parameter and Distribution risk
Ref: Wiley: Measuring Market Risk, 2nd Edition -
Kevin Dowd

30. HANDLINGLARGEDIMENSIONALDATASETS
Handling large dimensional data
• Variable Selection methods (where a subset of dimensions is chosen)
• Reducing dimensions (where variables are transformed to a smaller set
of new variables)
Ref: Dealing with Dimensionality in large datasets
http://www.quantuniversity.com/w8.html

31. CHARACTERISTICSOFSTRESSANDSCENARIOTESTING
Characteristics of Stress and Scenario testing
1. Difficult to build parametric models – Simulation driven
approach necessary
2. Parameter space can explode easily
3. Tests independent of each other (Embarrassingly parallel)
4. Complete test-coverage – Useless
5. Human intervention required
6. Tests to be designed and customized for the companies
needs considering portfolios, organization structure and
regulatory obligations
Ref: http://www.actuaries.org/CTTEES_SOLV/Documents/StressTestingPaper.pdf

32. LEVERAGINGTECHNOLOGYTOSCALESTRESSANDSCENARIOTESTING
Leveraging Technology to scale Stress and Scenario Testing
• Advances in technology in the last
two decades have significantly
enhanced the toolsets quants have to
develop, test and scale innovative
quantitative applications
• Simulation and stress testing in risk
management are vastly scalable due
to innovations in parallel and
distributed computing
• Restricting the number of tests due
to lack of technological resources not
an excuse
Ref: Gaining the Technology Edge:
http://www.quantuniversity.com/w5.html

33. LEVERAGETECHNOLOGYTOSCALEANALYTICS
Leverage technology to scale analytics
1.64 bit systems : Addressable space ~ 8TB
2.Multi-core processors : Explicit and Implicit Multi-threading
3.Parallel and Distributed Computing : Leverage
commodity/Specialized hardware to scale problems
4.General-purpose computing on graphics processing units :
Use graphics cards to scale your algorithms
5.Cloud Computing
Ref: Gaining the Technology Edge:
http://www.quantuniversity.com/w5.html

34. BESTPRACTICESTOLEVERAGETECHNOLOGYFORSTRESSTESTING
Best practices to leverage technology for Stress testing
1. Test Plan to optimize models for technology use
2. Parameter mapping plan to scale stress tests based on use
cases
3. Aggregation of Test results
1. Pass/Fail
2.Automatic validation and Benchmarking of results
3.Analytics to infer the state and maturity of the model
4.Model Monitoring Test plan
1.Monitoring of parameters
2.Monitoring of test adequacy
5.Integrate Testing in the Model Lifecycle Development
process

35. REFERENCES
Quantifying Model Risk
Wilmott Magazine
January 2014
The Decalogue
Wilmott Magazine
July 2014
Copies can be downloaded at :
http://www.quantuniversity.com/w6.html
http://www.quantuniversity.com/w9.html

36. SUMMARY
Model Risk : A Brief Introduction
A Framework driven approach to Model Risk Management
Quantifying Model Risk
Stress and Scenario Testing in Model Risk Management
1
2
3
4
Summary

37. UPCOMINGCOURSES
Upcoming courses
1. Data Science and Analytics for Quants
2. Model Risk Management
Covers analytical techniques and Big Data methodologies in
finance.
Covers Model Risk Management framework, analytical
techniques and case studies
Register your interest at www.quantuniversity.com

38. Thank you!
Sri Krishnamurthy, CFA, CAP
Founder and CEO
QuantUniversity LLC.
srikrishnamurthy
www.QuantUniversity.com
Contact
Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be
distributed or used in any other publication without the prior written consent of QuantUniversity LLC.