Model risk and validation are important processes for banks that rely on models. There are several potential sources of model risk over a model's lifecycle from data issues to changes over time that impact applicability. Effective validation ensures models are performing as intended and identifies limitations. It should include independent review and testing using quantitative and qualitative techniques on a regular basis to verify models continue to meet requirements.

1. Model Risk & Model
Validation
Rajib Chakravorty
The views expressed in this presentation are that of my own and they
do not represent anything of my current assignment or my current employer

2. Agenda
 What is Model & Model Risk Management
 Modelling Process
 Sources of Model Risk
 Regulatory Timelines
 What is Model Validation
 Model Validation Example
 Guidelines for Model validation
 Summary

3. The world we live in is vastly different from the world we
think we live in.
Nassim Nicholas Taleb
We know from chaos theory that even if you had a perfect
model of the world, you'd need infinite precision in order
to predict future events. With socio-political or economic
phenomena, we don't have anything like that.
Nassim Nicholas Taleb
In the risk management and compliance space, Taleb
argued that our corporations, industries, and economies
have become very fragile – a breeding ground for a Black
Swan event to occur and to have devastating and lasting
impact.
While statistical risk and pricing models may do a good
job when markets are calm, they lay the seeds for their
own destruction – it is inevitable that such models be
proven wrong. The riskometer is a myth (Danielsson
2009).

4. What is a Model?
“The term model refers to a quantitative method, system, or
approach that applies statistical, economic, financial, or
mathematical theories, techniques, and assumptions to process
input data into quantitative estimates. A model consists of three
components:
an information input component, which delivers assumptions and data to
the model;
a processing component, which transforms inputs into estimates;
and a reporting component, which translates the estimates into useful
business information.” *
*Source : SR Letter 11-7 Attachment
Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency
Examples of Models
Most organisations begin their inventory of models with the identification of model “classes”
aligned with business activities where model-based decision making occurs and which therefore
are potential sources of model risk.
Examples of model classes:
►Credit Risk (e.g., PD, LGD, Underwriting, Behavioral PD, Exposure)
►Treasury (e.g., ALM, liquidity risk)
►Stress Testing (e.g., credit loss forecasting, PPNR)
►Market Risk (e.g. Value-at-Risk)
►Trading Counterparty Credit Risk
►Operational Risk (e.g. Basel II models)
►Asset management (e.g. Portfolio Optimisation)
►Economic Capital

5. Model Risk Management
“The expanding use of models in all aspects of banking reflects the
extent to which models can improve business decisions, but models
also come with costs. There is the direct cost of devoting resources to
develop and implement models properly. There are also the
potential indirect costs of relying on models, such as the possible
adverse consequences (including financial loss) of decisions based on
models that are incorrect or misused.
Those consequences should be addressed by active management of
model risk.”
SR Letter 11-7
Attachment
Board of Governors of the Federal Reserve System
Office of the Comptroller of the Currency

6. Example of Model Failure
 Statistical models that attempt to predict equity market
prices based on historical data. So far, no such model is
considered to consistently make correct predictions over
the long term. One particularly memorable failure is that
of Long Term Capital Management, a fund that hired
highly qualified analysts, including a Nobel Prize winner
in economics, to develop a sophisticated statistical
model that predicted the price spreads between
different securities. The models produced impressive
profits until a spectacular debacle that caused the then
Federal Reserve chairman Alan Greenspan to step in to
broker a rescue plan by the Wall Street broker dealers in
order to prevent a meltdown of the bond market.

7. -Get Representative Sample – will be
used to identify Analytic Solution to the
Business Problem
- Apply Global Exclusion Criterion
- Perform Data Integrity Checks
- Agree on Operational Definitions of
Dependent Attribute
- Create Development / Validation
Samples*
SupplierInputProcessOutputCustomer
Client Identification of an
Analytic Solution
Algorithm / Code to
Implement the
Analytic Solution
- Representative Sample at
the required level –
Account, Transaction,
Relationship etc.
- Behavioral Information like:
Response, Profit,
Conversion, Lapsation,
Quote etc.
- Operational Definition of
above attributes.
- Overlay of External
Database Information
Client
Start
Data Preparation
- Perform Univariate / Bivariate Analysis
- Cap Outliers
- Treat Coded Values Appropriately
- Missing Treatment (Mean / Median /
Regression Based Approach etc)
- Convert Character Attributes to
Numeric Attributes - Dummy Creation /
Ordinal Values etc.
-Multicollinearity Removal – Identify set
of attributes with no serious
multicollinearity
Build Model – Identify the Best Set
of Significant (>95%) Attributes that
explain the phenomena:
- Logistic track: Concordance / AIC
/ HL Goodness of fit / KS/
Parameter Significance / VIF / CI /
Odds Ratio / Rank Ordering
- OLS track: R Square (adj) /
Parameter Significance / VIF / CI /
Rank Ordering
Validate Model
- Refit the model on Validation Sample
- Score the model on Validation / Full
Population.
- Check Signs with Bivariate Analysis.
Implementation Strategy
-Identify the Implementation
Strategy.
- Create a Code to implement it
at Customers Database.
- Test and Implement
-Control Strategy
- Finalize a Tracking
/ Control Mechanism
End Model Validates
on all
parameters
Y
N
Modeling - High Level Process Map

8. Sources of Model Risk In Model Life-Cycle
Start Model
Scoping
Data Preparation
Independent
Data
Validation &
Review
Model
Development
Independent
Model
Validation &
Review
Model Approval
Implementation
& Deployment
On-going
Monitoring
& Periodic Re-
Validation
Incomplete or
inaccurate model
development
data
Inconsistent
model set-up /
assumptions
Inconsistent
with business
objectives
Flawed model
theory,
approach, or
assumptions
Identifying and
choosing factors
that can be used to
estimate risks
Data
problems and
selection bias
Changes in
market
conditions
Model
Life
Cycle

9. Risk – Economic Capital Modeling
4. Integrate Risks to
Compute Firm’s
Economic Value
Distribution
3. Assess Dependencies
Among Risks
• Determine the Correlations
1. Identify Risks
2. Determine Firm’s Exposure to
Each Risk Driver
• Build Economic Value
Distributions
Risk
A. CREDIT B. INTEREST C. OTHER
MULTIPLE RISK DEPENDENCIES
1 in a 100
Expected Value
Probability
50%
1%
Economic Value
D. INTEGRATION
Credit Risk
•Data Reconciliation, Obligor Research, Running Credit Tools (KMV/ CreditMetrics)
Interest Rate Risk
•Scenario generation, In-house models for term structure of interest rates, Duration-
Convexity analysis

10. Data – Fit for Purpose ?
 Is the data suitable for the modelling task?
 Reliability in data collection; eg how reliable is a self-
assessment of income?
 Or, data based on an existing portfolio of predominantly
older customers is used to build a model for a card
targeting young customers.
 A data set of accepted loan applications, to build a
scorecard across all new applications.
 Market prices reflect the value of assets at any given time, but
that does not mean they provide a good signal of the state of
the economy or are a good input into forecast models. The
reason is that market prices reflect the constraints facing
market participants.

11. Robustness of data
 There are some problem domains where risk factors and
distributions on variables are stable over time.
 However, consumer credit does not remain stable over time.
 Credit risk changes over the business cycle.
 Credit usage behaviour changes over time.
 Banks’ risk appetite changes over time.
 Innovations in technology and product development change
risk.
 All of these time-varying factors affect the applicability of
credit risk models over time.

12. Possible fundamental limitations
of predictive model based on data
 History cannot always predict future: using relations derived from historical data
to predict the future implicitly assumes there are certain steady-state conditions or
constants in the complex system. This is almost always wrong when the system
involves people.
 The issue of unknown unknowns: in all data collection, the collector first defines
the set of variables for which data is collected. However, no matter how extensive
the collector considers his selection of the variables, there is always the possibility
of new variables that have not been considered or even defined, yet critical to the
outcome.
 Self-defeat of an algorithm: after an algorithm becomes an accepted standard of
measurement, it can be taken advantage of by people who understand the algorithm
and have the incentive to fool or manipulate the outcome. This is what happened to
the CDO rating. The CDO dealers actively fulfilled the rating agencies input to reach
an AAA or super-AAA on the CDO they are issuing by cleverly manipulating variables
that were "unknown" to the rating agencies' "sophisticated" models.
Source : https://en.wikipedia.org/wiki/Predictive_modelling

13. What is Model Validation for Basel
Compliance ?
Model validation is the set of processes and activities intended
to verify that models are performing as expected, in line with
their design objectives and business uses. Effective validation
helps ensure that models are sound. It also identifies potential
limitations and assumptions and assesses their possible impact.
“Validation…requires verification of the minimum requirements
for the IRB approach.”*
“Validation should focus on…oversight and control procedures
that are in place…prompt reassessment of the IRB parameters
when the actual outcomes diverge materially from expected
results.”**
*Source: BCBS WP No.14 – Feb 2005
**Source: BCBS NL No.4 – Jan 2005

14. Validation Approach – Rating System
Internal Validation by
Individual Bank
Validation of Rating
System
Validation of Rating
Process
BenchmarkingBacktesting
Data
Quality
PD
Report Problem
& Handling
Internal Use by
Credit Officers
Risk
Components
Model
Design
LGD EAD
Broad Approach
to Validation
Source: BCBS Working Paper No. 14– Feb 2005

15. Total Population Base
(N obs)
Development Sample
(n1 obs)
Validation Sample
(n2 obs)
Validation could be done in 2 ways:
 Validation Re-run
 Scoring the Validation sample
 Rerun the model on the validation sample.
 Check the chi-sq values and level of significances
and p-values for each explanatory variable.
 The p-values should not change significantly from
the development sample to the validation sample.
 Check the signs of the parameter estimates. They
should not change from development sample to the
validation sample.
 Check rank ordering. Both Development and
validation samples should rank order.
Validation sample scoringValidation Re-run
 Score the validation sample using the parameter
estimates obtained from the scorecard developed on
the development sample.
 Check rank ordering. Both development and
validation samples should rank order.
Scorecard Validation - Example

16. Lorenz Curve, Joint Lorenz, Gini Coefficient
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Random Development
Lorenz Curve
Lorenz curve indicates the lift provided by the
model over random selection.
Gini coefficient represents the area covered under
the Lorenz curve. A good model would have a Gini
coefficient between 0.2 - 0.35
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Random Development Validation
Joint Lorenz curve compares the lift provided by
the development sample and Validation sample.
For a stable and robust scorecard – the Lorenz
curves should be overlapping and similar in
distribution.

17. Approx. Bad Good Bad Overall Info Weight of Prob. Chi- % of % of Cum. % Cum. % % of Cum. % Gini Calc K-S
score %-ile (B) (G) Rate Odds Odds Evidence Bad square all Bad allGood Bad Good obs. obs. Calcs
~10% 3 915 0.3% 0.0 0.1 (2.7) 0.00 37.3 0.7% 10.4% 0.7% 10.4% 10% 10% 0.00 9.7%
~20% 6 912 0.7% 0.0 0.1 (2.0) 0.01 31.7 1.4% 10.4% 2.2% 20.9% 10% 20% 0.00 18.7%
~30% 7 910 0.8% 0.0 0.2 (1.8) 0.01 29.9 1.7% 10.4% 3.9% 31.2% 10% 30% 0.00 27.4%
~40% 13 905 1.4% 0.0 0.3 (1.2) 0.01 20.4 3.1% 10.3% 7.0% 41.6% 10% 40% 0.01 34.6%
~50% 19 898 2.1% 0.0 0.4 (0.8) 0.02 12.7 4.6% 10.2% 11.6% 51.8% 10% 50% 0.02 40.2%
~60% 30 888 3.3% 0.0 0.7 (0.3) 0.03 3.3 7.2% 10.1% 18.8% 61.9% 10% 60% 0.04 43.1%
~70% 30 888 3.3% 0.0 0.7 (0.3) 0.03 3.3 7.2% 10.1% 26.1% 72.1% 10% 70% 0.05 46.0%
~80% 61 856 6.7% 0.1 1.5 0.4 0.07 9.8 14.7% 9.8% 40.8% 81.9% 10% 80% 0.11 41.0%
~90% 78 840 8.5% 0.1 2.0 0.7 0.08 33.8 18.8% 9.6% 59.7% 91.4% 10% 90% 0.16 31.8%
~100% 167 750 18.2% 0.2 4.7 1.6 0.18 399.5 40.3% 8.6% 100.0% 100.0% 10% 100% 0.39 0.0%
Totals 414 8,762 4.5% 0.05 1.0 581.70 0.79 46.0%
B G B/G B/(B+G) Gini coefficient =0.2925
Gini Coefficient and KS Measure of the Model: Development Data set
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Random Development
Logistic Solution – KS, Gini & Lorenz

18. Guiding Principles for IRB Validation of Models
Ensure integrity of IRB processes & systems
Confirm predictiveness of PD, LGD, EAD
All IRB components
Models
Inputs (Data) & Outputs (Estimates)
Rating Process
Control & Oversight Mechanisms (e.g., Internal Audit, Use)
Independent validation team
Qualitative and Quantitative techniques
Review of documents
Meet with various depts. (e.g., risk mgmt, audit, etc.)
Determine model type & rating philosophy
Check logic behind model (programs)
Review sample data
Benchmarking (i.e., compare w/ external sources)
Backtesting (i.e., estimates v. actual)
Regular and Periodic Basis
At least once per year
Changes in model, data or portfolio
Initial model development
TIMING (WHEN)
PURPOSE (WHY)
SCOPE (WHAT)
MEMBERS (WHO)
METHOD (HOW)

19. Summary
 The models used in the bank do carry a risk which need to be
identified and reported along with the mitigation plan or
metrics
 The quantification of ‘Model Risk’ is still not very clearly
articulated and policies vary widely among banks
 Model development and validation process need to be
standardised across the bank with independent audit
mechanism wherever possible.
 The feedback mechanism of Model risk should incorporate the
‘Risk Appetite’ and ‘Tolerance’ levels as prescribed in the Risk
Management policies
 Model risk needs to be quantified for the enterprise which
means it needs to be aggregated for all models across
different types of risks faced by the bank