This document discusses enhancing approaches to aggregating model risk at financial institutions. It begins by noting that regulators expect banks to assess model risk in the aggregate. While quantitative approaches have been explored, most institutions use qualitative scorecard approaches. The document then outlines ways to improve upon typical qualitative scorecard approaches, including: [1] basing finding risk ratings on materiality to the institution not just the model, [2] using a standardized issue catalog to structure residual risk findings, [3] properly reflecting model interdependencies, and [4] measuring aggregate model risk at the functional area level rather than just by individual models.

1. Model Risk Aggregation
Xiaoling (Sean) Yu
SVP, Director of Model Validation
October 18, 2017

2. Disclaimer
The views expressed in this presentation are those of the
speaker and do not necessarily reflect the views of my
current or any previous employers in all respects.
1

3. Overview
• Assessing and managing model risk in the aggregate is an explicit regulatory expectation.
• Qualitative scorecard based approach instead of quantitative approaches is used by most
institutions for model risk aggregation
• Illustration of the typical qualitative scorecard based approach
• Aspects of the qualitative approach that can be enhanced
o What is Model Risk
o Consistent finding risk rating across models
o Standardized issue catalog
o Properly capturing of model interdependency
o Measuring by Functional Area
2

4. Regulatory Expectation
• Assessing and managing model risk in the aggregate is explicit required in SR 11-7/OCC
2011-12:
o “As part of their overall responsibilities, a bank's board and senior management should establish a
strong model risk management framework that … should be grounded in an understanding of
model risk—not just for individual models but also in the aggregate.”
o “In the same manner as for other major areas of risk, senior management, directly and through
relevant committees, is responsible for regularly reporting to the board on significant model risk,
from individual models and in the aggregate, and on compliance with policy.”
• OCC’s Heightened Standards also expects large banks to assess and manage model risk at
aggregate level:
• “Risk appetite means the aggregate level and types of risk the board of directors and
management are willing to assume to achieve a covered bank’s strategic objectives and business
plan”
• “Risk profile means a point-in-time assessment of a covered bank’s risks, aggregated within and
across each relevant risk category, using methodologies consistent with the risk appetite
statement”
3

5. General Industry Practices
• Quantitative approaches have been explored, but have not gained sufficient traction
o To mimic AMA operational risk modeling approach, or to quantify model uncertainty based on
results of various performance analyses
o They are too complex to implement, and also cannot avoid subjectivity due to limited data
• Qualitative scorecard based approach that leverages established model risk management
processes is much easy to implement and currently widely used in the industry
o Important inherent model risk factors (i.e. model complexity, uncertainty of inputs/assumptions,
and materiality of model impacts) are typically reflected in model tier assessment
o The effectiveness of model risk mitigation and residual risk are reflected in the findings of 2nd
LOD model validation and control review, as well as the effectiveness of 1st LOD finding
remediation and ongoing monitoring process
• The qualitative approach also serves the risk appetite statement and risk profile assessment
well. According to the recent ABA survey of 22 CCAR banks:
o 90% of the banks have established model KRIs and model risk tolerance thresholds
o 60% of the banks incorporate aggregated model risk into the bank’s risk appetite statement
4

6. Qualitative Model Risk Scorecard: Illustration
5
Inherent Risk Mitigation Residual Risks
Development
• Data Deficiencies
(availability and quality)
• Unsound Conceptual
Framework
• Bias/Flaws in Specification
or Assumptions
• Poor Model Performance
1st LOD
Testing and
Control
Validation Findings:
• Data Limitations
• Conceptual Soundness
• Inappropriate Assumptions
• Model Performance
• Inadequate Testing/Control
Validation Deficiencies
2nd LOD
Model
Validation
Implementation
• Inadequate Model
Documentation
• Implementation Errors
1st LOD
Acceptance
Tests
Validation Findings:
• Implementation Defects
• Acceptance Test
Deficiencies
Validation Deficiencies
2nd LOD
Model
Validation
OperationandUse
• Production Data Errors
• Incorrect Model Execution
• Improper Model Change
• Inappropriate Use of the
Model
• Environment Change
1st LOD
Ongoing
Monitoring and
Controls
Periodic Review Findings:
• Control Deficiencies
• Ongoing Monitoring
Deficiencies
• Model Performance
Deteriorations
Periodic Review Deficiencies
2nd LOD Periodic Review
Model Tier
Materiality
High Medium Low
Complexity
High I II III
Medium II III IV
Low III IV IV
Finding
Risk
Rating
Materiality
High High
Medium Medium
Low Low
Model Risk
Rating
Not
Approved
Conditionally
Approved
Approved
Metrics of Aggregate Model Risk
Current Risks:
• Material (High/Medium) Findings
• Timely Finding Remediation
• Conditionally Approved Models
• Not Approved Models in Use under
Exception Approval
Emerging Risks:
• New Models under Development
• Resource Adequacy

7. What is Model Risk
• Model risk is defined as “the potential for adverse consequences from decisions based on
incorrect or misused model outputs and reports” (SR 11-7/OCC 2011-12).
• “All models are wrong; some models are useful” (George E. P. Box). A model is an
approximation of the reality and therefore always deviates from the reality. Model risk is
not the deviation itself, but the misperception of the deviation and the adverse
consequences of this misperception.
• Not to say it’s less important, but model risk is a secondary risk, since models are
leveraged to reduce uncertainties (e.g. credit risk, market risk, valuation, etc.) encountered
in business operations.
• Assessing model risk should not focus on the absolute deviances/errors of the model,
but the clear understanding of the model’s value/limitations in reducing the primary
risks.
6

8. Enhancements: Finding Risk Rating
Challenge:
• While finding risk rating is defined by materiality, it mainly reflects materiality to the model, which makes
it difficult to aggregate findings across models.
• The typical solution is to weight findings by model tier. The caveat is that while findings capture the
residual model risks, model tier typically reflects inherent model risk (of model impacts and complexity).
Enhancement:
• Assigning finding risk rating based on materiality to the institution as well as to the model.
7
With the finding, does the
model still create value ?
If No, High Risk
Finding
If Yes, what’s the
impact to the
institution
Medium Risk
Finding
Low Risk Finding
High Risk
Finding
Medium Risk
Finding
Low Risk
Finding
Tier I
Models ✓ ✓
Tier II
Models ✓ ✓ ✓
Tier III
Models ✓ ✓
Tier IV
Models ✓ ✓

9. Enhancements: Standardized Issue Catalog
Challenge:
• Model risk findings reflect residual risks of different aspects and in different staged of the model
lifecycle. Aggregating model risk findings without clear structure may provide a high level view that
cannot help effectively manage the risk.
Enhancement:
• Developing a standardized issue catalog to capture the various types of residual model risk across model
lifecycle.
• Organizing the issues into meaningful structure to support effective model risk management at different
level.
• There are two fundamental types of issues – the ones related with model performance currently and the
ones related with the deficiencies of mitigations/controls that may affect model performance in the
future.
8
Performance Deficiencies
(Known Unknowns)
• Data
• Conceptual Soundness
• Assumptions
• Outcome Analysis
• Implementation
Control Deficiencies
(Unknown Unknowns)
• Development Testing/Controls
• Documentation
• Implementation Testing
• Operation Controls
• Ongoing Monitoring

10. Enhancements: Model Interdependency
Challenge:
• Model interdependency is not fully reflected in model risk assessment and aggregation
• Model risk finding is typically rated at model level, sometimes with its impacts on direct downstream
models considered.
• Model interdependency is also normally not reflected in model tier assignment.
Enhancement:
• Model inventory needs to fully capture the model interdependency information, and to reflect position
of each model within the full model network and model streams of specific functional areas.
• Model interdependency should be reflected in model tier assignment and finding risk rating
determination.
9

11. Enhancements: Measuring by Functional Areas
Challenge:
• Ideally, the materiality of model risk should be “quantified” by a common measure (e.g. ROE). But
given the broad uses of models, it’s hard to converting their materiality to the institution into a common
measure.
• There are also sometimes multiple set of models leveraged for the same functional use. For instance,
both CCAR and Economic Capital models could be used in capital adequacy assessment and portfolio
management. Simply adding up their residual risks in the aggregation may overstate model risk.
Enhancement:
• To group models into major functional areas, with proper weights assigned to each area. Identify sub-
areas within each functional areas and allocate the weight of the major functional area down to avoid
overstating model risk in aggregation.
10
Functional Area Weight Sub-Area Weight
Capital Adequacy 5
CCAR 3
EC 2
Valuation and Pricing 3
ALM 4
Compliance 5
Marketing 1

12. 11
THANK YOU