Forecasting operational risk losses for CCAR poses a number of challenges, not least of which is uncertainty about the best methodology for modeling the relationship between a bank’s future losses and the macroeconomic environment.

1. Aon Risk Solutions
Aon Global Risk Consulting
Risk. Reinsurance. Human Resources.
Variable selection for CCAR operational risk
projections: model formula or expert judgment?
Forecasting operational risk losses for CCAR poses a number of challenges, not least of which
is uncertainty about the best methodology for modeling the relationship between a bank’s
future losses and the macroeconomic environment.
The economic scenarios produced by the Federal Reserve provide the
cornerstone of the DFAST and CCAR stress testing exercises. Established
models exist for market risk and credit risk to link losses to economic variables,
but the best approach for operational risk is less clear.
It is widely acknowledged that many of the material drivers of operational risk
losses are independent of changes to the economic environment. But given
that banks are required to use the scenarios as a starting point, how should we
go about looking for the relationships that underpin these models?
Do we start with theory or data?
The question that banks need to answer is this: “Should variable selection for
regression analysis in operational risk CCAR models be purely objective and
guided by program algorithms or subjectively guided by economic theory and
logic?”
The Federal Reserve summed up the difficulty in their 2015 CCAR instructions:
“Regardless of the estimation methodology, BHCs should have a transparent,
repeatable, well-supported process that generates credible estimates that are
consistent with assumed scenario conditions.”
But:
“BHCs have found it challenging to identify meaningful relationships between
operational losses and macroeconomic factors… [and] should not try to force
the use of unstable and/or unobservable correlations.”
Best practices in this area continue to develop as regulators push for continued
improvement with each stress testing cycle. From conversations with our
clients, it is clear that there is no preferred approach. In fact, our experience is
that regulators are looking for banks to consider both objective and subjective
aspects of the variable selection process.
We’re here to
empower results:
Please reach out to a
member of Aon’s Financial
Institutions Risk Advisory
team to learn more.
Evan Sekeris
Director & Global Lead
+1.310.490.0243
evangelos.sekeris@aon.com
Jin Kang
Associate Director
+1.917.946.4963
jin.kang@aon.com
Derrick Oracki
Associate Director
+1.202.429.8539
derrick.oracki@aon.com

2. Aon Risk Solutions
Aon Global Risk Consulting
Variable selection for operational risk CCAR projections: model formula or modeler judgment? 2
Objective variable selection – the
algorithmic approach
We can think of “objective variable selection” as any
process where relationships between
macroeconomic variables and operational losses are
found using computer algorithms. Given the
absence of well-established theories on the
relationships between operational risk losses and
the economy, it comes as no surprise that many
banks rely heavily on such approaches when faced
with the task of tying their loss projections to the
Federal Reserve’s scenarios.
Banks may choose from many statistical algorithms,
each with their own strengths and weaknesses.
Most fall into one of two categories:
 Algorithms that evaluate the goodness of fit
between candidate models and quarterly
operational risk losses. Examples include
stepwise, ridge and lasso regression.
 Algorithms that focus on minimizing the
correlation between the macroeconomic
variables, without explicitly considering the
historical losses. Examples include spectral
decomposition and principal components
analysis.
Such algorithms can uncover strong relationships in
the data that a purely judgmental approach may
miss. Regulators will look for justification when an
institution chooses not to consider a macroeconomic
variable that has strong correlation with historical
losses.
A further advantage is that algorithms represent
transparent and repeatable processes that are
straightforward to document and implement.
Validation teams favor algorithmic approaches for
their reproducible answers and objective results.
For these reasons, many institutions rely heavily on
such approaches when faced with the task of tying
their loss supervisory scenario projections to the
Federal Reserve’s scenarios.
However, despite their advantages, algorithmic
approaches can never see the data as more than
columns of numbers. They are prone to indicating
relationships that are spurious, unstable, and
inconsistent with assumed scenario conditions over
time despite good fit statistics. Institutions who adopt
a purely algorithmic approach should expect to be
pushed to explain the theoretical justification behind
their selected models. Such justification can be
difficult to provide.
Subjective variable selection – the
theory and logic approach
Theory and logic-based approaches seek to address
some of the issues with pure algorithmic models.
Following a pure theory and logic approach would
involve developing a model framework as follows:
 Specify a functional form of theoretical variables
based on established theory;
 Then, obtain observable variables to represent
the theoretical inputs;
 Finally, estimate parameters and test the
statistical significance of these relationships.
Before we get to the obvious problems with such an
approach, let us consider some of the advantages:
 First and foremost, this approach removes the
problem of justifying the model selection on
theoretical grounds.
 Theory-based models are also more in keeping
with the idealized way in which we would like to
construct models, since they start with a
hypothesis that we then test against data.
 Although the process of arriving at the
hypothesized relationships between
macroeconomic factors and operational losses
is complex, it should lead to a more in-depth
understanding of an institution’s operational risk
exposure. We should not forget that this is the
spirit of the Federal Reserve’s stress testing
framework.

3. Aon Risk Solutions
Global Risk Consulting | Financial Institutions Risk Advisory
Variable selection for operational risk CCAR projections: model formula or expert judgment? 3
Unfortunately for those tasked with developing
operational risk models for CCAR and DFAST
projections, well-established theories linking
operational risk losses to economic factors simply do
not exist. Instead, what we’re left with is the problem
of establishing a relationship of the very general
form: Expected Losses = f(macroeconomic health).
But what is this relationship? “Macroeconomic
health” is an unhelpfully broad term. It could easily
encompass many observable variables, including
general measures like GDP, inflation and
unemployment along with more specific factors that
could potentially have a more direct causal
relationship will operational risk losses.
As a result, variable selection in a theory and logic
driven process requires significant exploratory
analysis and the application of significant subjective
expert judgment. As with any good model, the
process must be thorough, robust, transparent,
repeatable and well supported. Carrying out such a
process requires an extensive time commitment.
Supporting documentation must be sufficient to
allow a suitably qualified third party to understand,
critique and repeat the process, and as such must
contain information such as:
• Explanation of all subjective expert
judgments applied.
• The sensitivity of results to these expert
judgments and modeler decisions.
• The underlying logic that the validation team
should follow to reproduce the decisions.
• Explanations of how modelers should adjust
such decisions in the future in response to
updated data.
Perhaps most troubling of all (at least from the point
of view of validators and regulators), it is perfectly
likely that different modelers will obtain different
results.
A further, disadvantage of such approaches is that
they open institutions up to the opposite question
that we see with purely algorithmic approaches. If
your theory and logic-based approach ignores a
variable that exhibits a stronger relationship with
historical losses than the one in your selected
model, you should be prepared to justify why you did
not used this first variable for your projections.
Looking beyond regression for
operational risk in CCAR and DFAST
The Federal Reserve’s CCAR 2015 instructions
state that “BHCs have found it challenging to identify
meaningful relationships between operational losses
and macroeconomic factors”, citing limited datasets
and extensive judgment in assigning dates to loss
events as sources of the challenge. The instructions
go on to note that “given the challenges noted
above, BHCs should not try to force the use of
unstable and/or unobservable correlations and
should instead use a conservative approach to
project increased operational risk losses from
significant operational risk events that could
plausibly occur during a stressed macroeconomic
environment.”
Everything that we have discussed so far has
focused on the use of regression models. However,
the Federal Reserve’s guidance states that “BHCs
are encouraged to explore multiple loss-projection
techniques as long as the overall methodology
ultimately leads to reasonable, significant loss
projections.” In particular, the guidance goes on to
discuss a number of alternative techniques,
including regression analysis, loss distribution
analysis (LDA), historical averages, and scenarios
analysis and states that institutions may use all of
them, so long as they are all logical, well supported,
and effectively stress material, inherent risks.
Consistent with these instructions, we recommend a
model framework that explores a number of
methodologies for each unit of measure and a
robust approach for identifying the “best” model for
the final projection. Such an approach allows for
explicit and transparent consideration of indications

4. Aon Risk Solutions
Global Risk Consulting | Financial Institutions Risk Advisory
Variable selection for operational risk CCAR projections: model formula or expert judgment? 4
from multiple methodologies and produces a wealth
of developmental evidence to help advance the
development of theory for the operational loss
models.
Implications for regression analysis:
where do we see the industry heading?
We have seen that operational risk behaves in a
fundamentally different way from other risks
modeled by institutions. As such, different
approaches to modeling are required.
With the dust settling from the 2015 capital planning
and stress testing cycle, it is clear that there is no
silver bullet on the horizon. Regulators are looking
for institutions to show that their framework uses
both objective and subjective approaches for model
selection.
In response to the limitations of regression-based
approaches, regulators are pushing institutions to
consider hybrid projection methodologies that
included historical averages and scenario analysis.
Of course, scenario analysis is a critical component
of any framework for considering idiosyncratic risks.
As internal loss datasets grow, regulators will look
for frameworks that rely less on historical averages,
and instead base projections on a methodology that
is appropriate for the risk in question.
We consider hybrid frameworks to be the sensible
approach to tackling operational risk stress testing.
When CCAR/DFAST first appeared on the horizon,
the industry instinctively reached for regression
models as the natural way of relating operational
losses to the economic cycle. With the experience
gained from several model build and submission
cycles, perhaps now is the time for a fundamental
review of the current approaches. If it is obvious
that the economic cycle is not the main driving force
behind many operational risks, then why should we
force our modelers to find such relationships?
Why work with Aon?
Leading institutions know that the key to an effective and successful enterprise risk management framework lies
in ensuring compliance with evolving regulatory expectations whilst simultaneously adding tangible value to the
business. We understand this too. Our team has experience of being on both sides of regulatory conversations,
and of developing risk management tools and processes both as consultants and in the industry.
A “one size fits all” is often counterproductive when working with complex financial institutions. We believe that
the key to successfully supporting our clients is for us to understand their unique issues and approach to risk
management. Our clients see us as trusted advisors who provide in-depth advice blending regulatory insights
with practical recommendations and support.
About Aon
Aon plc (NYSE:AON) is the leading global provider of risk management, insurance and reinsurance brokerage,
and human resources solutions and outsourcing services. Through its more than 66,000 colleagues worldwide,
Aon unites to empower results for clients in over 120 countries via innovative and effective risk and people
solutions and through industry-leading global resources and technical expertise. Aon has been named repeatedly
as the world’s best broker, best insurance intermediary, reinsurance intermediary, captives manager and best
employee benefits consulting firm by multiple industry sources. Visit www.aon.rcom for more information on Aon
and www.aon.com/manchesterunited to learn about Aon’s global and principle partnership with Manchester
United.