SlideShare a Scribd company logo
GXN-1534
Balancing model performance and
complexity in real-world analytics
applications
Dr. Jeffrey R. Bohn
Chief Science Officer
Head of GX Labs
jbohn@statestreet.com
Limited Access
GXN-1534
Outline
•  Introduction
•  CDO “case study”: Balancing complexity and performance
•  Defining complexity
•  Driving complexity
•  Simplicity and complexity
•  Model performance
•  Choosing a model
•  Communicating complexity
•  Recommendations
2
GXN-1534GXN-1534
The wisdom of Archilochus (Greek
lyric poet from Paros, lived 680
BCE to 645 BCE)
The fox knows many things, but
the hedgehog knows one big
thing.
3
GXN-1534GXN-1534
Thinking
Theories
4
GXN-1534
CDO “case study”
5
GXN-1534
Big Short: Collateralized debt obligations (CDOs)
•  Simple in concept: Portfolio of many assets plus cash waterfall changes risk profile
•  Complex in structure: Rules in cash waterfall; over-collateralization and interest-
coverage triggers
•  Simple risk ratings: Diversity score (binomial expansion) or Gaussian copula plus
“shading” based on a committee
•  Complex market structure: Borrowers, appraisers, mortgage brokers, commercial banks,
mortgage servicers, investment bankers, fund managers, rating agencies, institutional
investors, regulators (Fed, NAIC, SEC) lawyers and many consultants
•  Simplicity in incentives: Money for nothing– or-- at the least-- mis-representation of
actual risk relative to fees and spreads
Communication regarding CDO risk to senior executives at financial institutions was
mostly not cognitively compelling.
“Real risk was not volatility; real risk was stupid investment decisions.” Lewis (2010)
6
GXN-1534
Failure of a simple CDO model
•  Materially underestimated risk of mezzanine and senior tranches
•  Often over-simplified waterfall rules– e.g., equity prioritized
•  Unrealistic assumptions with respect to default rates
•  Herd mentality on CDO ratings
•  Correlation dynamics not well represented– i.e., correlation went toward 1 in crisis.
7
GXN-1534
Portfolio Simulation: A more complex, but better model
Sampling across the Latent Factors to simulation various risk conditions
N securities
Portfolio
Valuation
Model
…
Random draws
Single Position
K latent factors
Factor
draws
Factor realizationsFactor Betas
idiosyncratic
value of
ith security
K = 30 factors
N ≈ 10k ~ 500k positions/idiosyncratic draws
J ≈ 1m ~ 10m iterations
How much computing power needed?
J iterations
Distribution
Repeat J times
value of
portfolio for
one iteration
equity
fixed income
other
Repeat N times
Source: State Street Global Exchange℠
Limited Access
Figure provided for illustrative purposes only.
8
GXN-1534
Multi-asset-class portfolio-risk modeling
Use-objective focuses on incorporating risk strategy into portfolio management
probability
Start of TailExpected
Tail Loss
stress losses
Portfolio value
Expected
Value
Risk-appetite assessment
•  Model guides executives as to whether allocation is prudent from a risk perspective
Sub-portfolio/manager/hedging evaluation
•  Model assists in evaluating how well components of portfolio contribute to overall risk/return
Scenario analysis
•  Model provide input into strategic discussions on portfolio construction/management
Figure provided for illustrative purposes.
9
GXN-1534
Interpreting simulated scenarios
Reverse Stress Testing
Values at horizon Search through
factor space
Inflation
S&P
Oil Price
GDP
Rates
Macro scenarios
What factor realizations
generate a portfolio value
in the specified confidence
interval of portfolio
distribution?
Source: State Street Global Exchange℠
Limited Access
Figure provided for illustrative purposes.
10
GXN-1534
Driving complexity
11
GXN-1534GXN-1534
Defining complexity
•  Epistemic: Hard to understand conceptually
•  Computational: Algorithm hard to understand and/or implement
•  Dynamic: System changes over time– sometimes as a function of itself or its users
•  Human minds are memory and predicting mechanisms: Does the complexity arise
from quantity of “insight chunks” that need to be remembered or from the quantity of
steps/inter-relationships in the predictive model?
•  Simple heuristics may arise from highly complex models/systems. Complexity may arise
from incorporating nuances into interpretations/decisions based on a given heuristic.
12
GXN-1534GXN-1534
Consilience: A frequent driver
toward complexity
	A	"jumping	together”	of	knowledge	by	the	
linking	of	facts	and	fact-based	theory	across	
disciplines	to	create	a	common	groundwork	of	
explana=on.	
	
Wilson	(1999)	p.	8	
13
GXN-1534
What makes decision-support analytics complex?
•  Multi-variate optimization problem– often with controls and results playing out over
different time horizons.
•  Uncertainty– “risk” defined by known distributions and “Knightian uncertainty” defined by
unknown models/data-generating processes
•  Interconnectedness– both explicit and implicit.
•  Hierarchies of relationships and relevant assumptions
•  Separation of point predictions/estimates and distribution estimates
14
GXN-1534GXN-1534
Drivers of model complexity
•  Nonstationarity:
–  Macro-economic regime changes
–  Firms changing leverage
–  Governments changing regulation
•  Heterogeneous return distributions by asset class
–  Non-normal
–  Skewed
–  Fat tails
–  Liquidity
•  Capturing obscure, but material risks
–  Correlated exposure to a latent factor (e.g., US housing market)
–  Sectorally diversified investment-grade bond portfolio may constitute material
concentration risk
–  Wrong-way, counterparty risk to an over-hedged energy company
•  Human reaction to changing
- Bank management reaction to stress environment by changing underwriting standards
- Not selling in a down market and distressed-selling in an up market not recognized
- Inability or unwillingness to understand a model and its implications may lead to value-
destroying behavior
- Traders gaming particularly sensitive aspects of a risk model
		15
GXN-1534
Shift in risk modeling favors more model complexity
Normal Distribution
Assumptions
Linear estimation is
good enough
Current	Mainstream	
Paradigm	(examples)	
Focus on 2nd moment Linear Regressions
Bias toward tractable
“closed form”
solutions
Examples FastData™
BigComputation™
Machine Learning
Empirical orientation
Enablers
Skewed, Fat-tailed
distributions
Focus on Skewness
and Tail, generated by
simulations
Deep Learning
Need to recognize
non-linearities
Empirical approach
with recognition of
non-linearity
Examples
New Approaches
Source: State Street Global Exchange℠
16
GXN-1534
Simplicity and complexity
17
GXN-1534GXN-1534
Simple/Complex Enough
Albert Einstein displayed the
following aphorism in his office:
“Things that are difficult to do are
being done from the wrong centers
and are not worth doing.”
-- Diaconis (2003)
Albert Einstein may have said
something like “[Theories],
[Education] or [Things] should be as
simple as possible, but no simpler.”
18
GXN-1534GXN-1534
Simplicity may not always be the primary or best criterion
•  Also attributed to Einstein: "Any intelligent fool can make things bigger, more complex, and more
violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction.”
•  Simplicity should be one goal, not a hard criterion.
•  Usefulness should also be a goal and should be a hard criterion: Thus, simple enough, but no
simpler. This thought can lead to…
•  Occam’s Razor: “Among competing hypotheses, the one with the fewest assumptions should be
selected.”
–  William of Ockham’s actual quote (I think): “Numquam ponenda est pluralitas sine
necessitate [Plurality must never be posited without necessity]”
–  Bertrand Russell’s version: "Whenever possible, substitute constructions out of known
entities for inferences to unknown entities.”
•  Simplicity in concept may belie complexity in reality– e.g., biological evolution, construct a
portfolio that finds highest return/risk
•  Not likely to be a global principle– important that model “suitably” explains/predicts data
–  “Suitably” implies matching model with use objective
–  Over time, more complex models may explain data better and open new vistas e.g., atomic
theory, plate tectonics, Black-Scholes, light theory (Newton’s “simpler” particle’s versus
Huygen’s waves)– bottom-up, factor-based portfolio simulations?
19
GXN-1534GXN-1534
Model performance
20
GXN-1534
Hume’s problem: When will a model perform?
•  When is it reasonable to think the future will be like the past?
•  Rephrase– what part of a model’s/system’s/algorithm’s structure will operate in the future
in the same way it has in the past?
–  Means are difficult to estimate and historical observation may be a poor guide
–  Volatility may be easier (than means) to estimate, but still may change so much that
historical observation is likely to be a poor guide.
–  Underlying correlation structure may change-- but not as much as volatility
–  Co-skew?
–  Co-kurtosis?
How much can/should non-linear-process-inducing feedback loops and tipping points be
included in a model and can historical observation help?
How does model complexification affect strategies for communicating to quantitatively-
informed business executives?
21
GXN-1534GXN-1534
How do we know a portfolio-risk system/model works?
“Science is no inexorable march to truth, mediated by the collection of
objective information and the destruction of ancient superstition.
Scientists, as ordinary human beings, unconsciously reflect in their
theories the social and political constraints of their times.”
-- Stephen Jay Gould
Karl Popper’s process applied to portfolio-risk analysis:
•  Problem situation: “Risk-appetite consistent allocation”
•  Tentative theories: “Hypothesize scenarios”
•  Error elimination: “Stress the stress test i.e., simulate”
However, falsifying every scenario is not possible; further, Gould argues to
beware theory-laden analyses.
22
GXN-1534
Complex models perform better: Evolution
•  Darwin (1859): Natural selection is a key mechanism of evolution based on the
differential survival and reproduction of individuals due to phenotype differences.
•  Romanes (1895): Neo-darwinism refers to germ-plasm theory advocated by Wallace and
Weissmann.
•  Huxley (1942): Modern synthesis drew together multiple fields of biology marrying
natural selection, genetics, natural population analysis, systematics, etc.
•  Today’s models of evolution are much more complex:
–  Genetic networks composed of tens to hundreds of genes interact
–  “Regulatory” genes behave conditional on the environment
–  Some hereditary variations are nonrandom in origin
–  Some acquired information is inherited (epigenetic inheritance)
–  Evolutionary change can result from instruction as well as selection
Jablonka and Lamb (2014)
23
GXN-1534
Complex models perform better: Risk analytics
•  Capital-asset pricing model
•  Markowitzian portfolio theory
•  Arrow-Debreu
•  Black-Scholes-Merton
•  Continuous-time finance
•  Value at risk
•  Monte-Carlo simulation
•  Expected-tail loss
Are these “advances” i.e., model complexification worth the investment?
24
GXN-1534GXN-1534
Choosing a model
25
GXN-1534GXN-1534
Criteria for deciding degree of model complexity
•  Model performance
•  Ease of communicating actionable model output to quantitatively-informed executives
•  Ignores extraneous information
•  Balances model uncertainty and credibility of priors
•  Minimizes over-fitting risk: Sample size relative to number of parameters
•  Resistant to manipulation as a consequence of incentives
“Because complexity generates uncertainty, not risk, it requires a regulatory response
grounded in simplicity, not complexity.” (Haldane, 2005, p. 19)
What does this mean in practice?
26
GXN-1534GXN-1534
Questioning the entire modeling enterprise
•  Preproducibility is a prerequisite for attempting to reproduce a result: it involves providing
an adequate description of an experiment or analysis for the work to be re-undertaken. It
requires documentation, openness, and communication.
•  Quantifauxcation is to assign a meaningless number, then pretend that since it’s
quantitative, it’s meaningful. Usually involves some combination of data, pure invention, ad-
hoc models, inappropriate statistics, and logical lacunae.
•  Cost of most policy cost-benefit analyses is high: lost rationality
•  Rates vs. probabilities
–  Randomness created by taking a random sample, assigning subjects at random, etc.
–  Probability model invented (assumed) for data that world generates– inferences are
only as good as the assumptions
–  Aleatory: Coin toss, die roll, under some circumstances, behave “as if” random
–  Epistemic: Stuff we don’t know
–  Trials are random, have same chance of success and have known dependence– can
quantify estimate uncertainty.
–  Ignorance does not equal randomness
–  Tendency to treat haphazard as random
–  Probability as metaphor
(From Stark 2015)
27
GXN-1534GXN-1534
Model risk
•  Mis-specify data-generating process: Fail to estimate the “true” distribution due to the
fact that the underlying process differs materially from the model’s process assumptions.
–  Second-order assessment of misspecification also important.
–  What is range of wrong estimates?
–  Are out-of-sample tests showing under-estimates in times of stress?
•  Mis-estimate parameters: While the assumed data-generating process may be
reasonable, still fail to estimate “true” distribution based on parameterization problems.
–  Are there enough data points to credibly estimate parameters?
–  Can model structure be used to infer tail events without requisite data?
28
GXN-1534GXN-1534
Risk versus uncertainty
•  Frank Knight (Knight, 1921): "Uncertainty must be taken in a sense radically distinct from
the familiar notion of risk, from which it has never been properly separated.... The
essential fact is that 'risk' means in some cases a quantity susceptible of measurement,
while at other times it is something distinctly not of this character; and there are far-
reaching and crucial differences in the bearings of the phenomena depending on which
of the two is really present and operating.... It will appear that a measurable uncertainty,
or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one
that it is not in effect an uncertainty at all.”
•  Risk is known unknowns i.e., data generating process, inter-relationships arising from
model structure, parameters imply known distributions
•  Knightian uncertainty is unknown unknowns i.e., data generating process, model
structure and parameters are unknown
29
GXN-1534GXN-1534
Types of ignorance
Hansen and Sargent (2015)
		
Xt+1
=κ Xt
+ βUt
+αWt+1
Xt
≡ Observable	state	variable	at	time	t
Ut
≡ Control	variable
Wt+1
≡ Random	shock	to	process
1.  “Bayesian decision maker” does not know β, but trusts prior prob. distribution
2.  “Robust Bayesian decision maker” does not trust prior distribution for response
coefficient β; but uses operators to twist prior distributions to generate conservative est.
3. “Robust decision maker” uses a multiplier or constraint preferences to express doubt
regarding probability distribution of W conditional on X and U.
4. “Robust decision maker” asserts ignorance as in 3 by adjusting an entropy penalty to
Make model robust to particular alternative probability models.
“The trouble with most folks isn’t so much their ignorance, as knowing so
many things that ain’t so.” Josh Billings related by Friedman (1965)
30
GXN-1534GXN-1534
Evaluating the adequacy of a theory/model
1.  Accuracy [out-of-sample confirmation of estimated probability distribution and
contributions of underlying components to that distribution]
2.  Consistency (both internal and external) [multi-asset-class, assets & liabilities]
3.  Broadness in scope [granularity and comprehensiveness]
4.  Simplicity [complex enough to capture dynamics, but simple enough to be diagnosed
and communicated to a quantitatively-informed business head]
5.  Fruitfulness [output substantively contributes to impactful decisions]
Depending on the theory under evaluation, criteria may contradict each other so a relative
weighting may be needed i.e., given a particular circumstance, some criteria are more
important than others. Kuhn (1977)
In portfolio risk analysis, we typically add Timeliness to the evaluation process– a
successful theory/model/system that cannot provide timely output is useless.
31
GXN-1534
Communicating complexity
32
GXN-1534
Understanding the decision maker: Elephant and the rider
•  Elephant: Automatic processes
•  Rider: Controlled processes
•  Modeling decision making
–  Humean model: Reason is a servant
–  Platonic model: Reason could and should rule
–  Jeffersonian model: Head and heart are co-emperors
•  “Seeing that” vs. “Reasoning why”
•  Rationalist delusion: Maintaining healthy skepticism of reason– smarter people
rationalize better
People who devote their lives to studying something often come to believe that the object
of their fascination is the key to understanding everything. Location 58
Conscious reasoning functions like a press secretary who automatically justifies any
position taken by the president. p. 106
And as reasoning is not the source, whence either disputant derives his tenets; it is in vain
to expect, that any logic, which speaks not to the affections, will ever engage him to
embrace sounder principles. – David Hume
Haidt (2012)
33
GXN-1534
How individuals make decisions
Agree on values Disagree on values
Agree on facts Computational decision Negotiate
Disagree on facts Experiment Paralysis or chaos
34
From Koomey (2001) figure 19.1 p. 88.
GXN-1534
How to communicate in a cognitively compelling way?
•  Research audience…
–  Culture (values and biases)
–  Education (general and specific to communicated analytics)
–  Incentives (degree of departure from objectivity)
•  Distill message...
–  Empirical vs. rational/theoretical (where is the model in this iterative evolution?)
–  Transparent assumptions
–  Frank assessment of model uncertainty
–  Link to audience’s narrative/values
•  Use intuitive framing and strong visuals…
–  Relevant metaphors
–  Examples from different, but similar domains (e.g., health/medical)
–  Design compelling graphs and figures
35
GXN-1534
Working memory and compelling communication
•  Working memory
–  Operates over a few seconds
–  Temporary storage
–  Manipulates attention
–  Focuses attention
–  Resists distractions
–  Guides decision-making
•  Can only process 5 to 9 “chunks” of information within working memory at any given
moment in time (Miller, 1955)
•  Deviating from expectations typically causes the listener to disengage
•  Working memory dis-fluently “chunks” instead of always focusing on what matters
•  Working memory “calls” long-term memory to assist in processing; if nothing is there,
cognitive flow is broken– result is likely disengagement
Education is essential to build up long-term memory
36
GXN-1534
Building a narrative
1.  Identify the [the set of] focal issue(s)
2.  Determine the key micro or local forces impacting the focal issue(s)
3.  Identify the key macro or global forces impacting the focal issue(s)
4.  Rank by importance and uncertainty– distinguish parameterization of probability
distributions (known unknowns) from model uncertainty (unknown unknowns)
5.  Select scenario logic in terms of the parameters (and maybe models) to adjust to show
range of possible outputs
6.  Flesh out the scenarios of most importance and highest likelihood– drill into details of
micro/macro forces and nature of parameterized probabilities and uncertainty
7.  Determine implications
8.  Select leading indicators and signposts to monitor evolution of scenario in light of
decisions
37
From Koomey (2001)
GXN-1534
Communicating Uncertainty
•  Explain signal and noise in specific terms
•  Communicate how model disentangles signal and noise
•  Identify and root out data biases
•  Educate on error bars and confidence intervals
•  Beware illusory precision– clarify how much precision is possible given data/model
Sampling error does not necessarily equal “uncertainty” in terms of implications of model
output
38
GXN-1534GXN-1534
Cognitive biases
•  False dichotomy: Presenting two choices such that it seems they are the only possibilities.
–  Simple vs. complex model
–  Use no models vs. use only one model
•  Perfect as the enemy of the good (or good enough)
•  Red herrings and missing forest for the trees
•  Biases
–  Affect heuristic: Analyst or executive has “fallen in love with” a particular output so that
they minimize model problems and exaggerate model strengths.
–  Groupthink
–  Saliency bias: Overly influence by analogous, past success
–  Confirmation bias
–  Availability bias
–  Anchoring bias
–  Halo effect: Impression of model author, analyst or even model influences interpretation
–  Sunk-cost fallacy: A particular model output has driven strategy/investment
–  Overconfidence
–  Disaster neglect
–  Loss aversion
	
Traps arising from logical fallacies and cognitive biases
39
GXN-1534GXN-1534
Key points to remember for compelling communication
•  Frame within a narrative
•  Avoid “quantifauxcation”
•  Contextualize (across time and across cohorts)
•  Address biases: Highlight data selection concerns and explain assumptions & process
•  Use transparency in model estimation process to spark questions and debate
•  Compare output from multiple models (when possible)
•  Visualize data– encourage interactive diagnostics and drill-down
•  Emphasize actionable insight
•  Educate
–  Explain key components of analytical process
–  Teach how to understand confidence intervals (noise vs. signal)
Build on understanding: Descriptive, prescriptive and cognitive
Move from analysis (breaking into components) to synthesis (re-assembling with insight)
40
GXN-1534
Principles of risk-data visualization
•  Match output to use cases
–  Concentration risk assessment
–  Risk appetite assessment (stress testing)
–  Position-level limits/allocation
•  Prepare for multiple dimensions (e.g., region, sector, asset class, customer type, size)
•  Incorporate drill-down capability
•  Contextualize output (e.g., benchmarks, time series, scenario-based)
•  Use robust statistics (e.g., median, inter-quartile, mean absolute deviation)
•  Use techniques to address data difficulties (e.g., Winsorization, shrinkage)
•  Target near-instantaneous rendering of decision-support output
Risk data tend to be defined by outliers
41
GXN-1534
Data are all figurative for illustrative purposes
Multiple attributes represented in two dimensions
Expected Returns vs. Volatility by Exposure Size – Sharpe Ratio as Color
High Sharpe Ratios,
but small positions
OK Sharpe Ratios,
and larger position
42
GXN-1534
Data are all figurative for illustrative purposes
So which is richer, from a data insight perspective?
43
GXN-1534
Improve visualization independent of model complexity
•  Google study (Tuch, et al., 2012) found for websites:
–  Visually complex websites are less appealing
–  Prototypical websites (for a given category) are more appealing
–  Simpler design is rated higher
•  What makes analytical output compelling and credible?
–  Prototypicality: Basic mental image one’s brain creates to categorize everything with
which you interact.
–  Cognitive fluency: One’s brain prefers output that is easier to process.
–  Mere exposure effect: Familiarity arising from repeated exposure.
–  Metric balancing: Too many metrics equals no understanding.
44
GXN-1534GXN-1534
Recommendations
45
GXN-1534GXN-1534
Be transparent as to epistemic nature of model output
•  What the executive doesn’t know, but is knowable: Model output is available and useful
e.g., credible metrics identify risk (in the technical sense.)
•  What the executive or the analyst don’t know yet, but is knowable: Proof-of-concept
model is available; however, more investment (e.g., data, analysts, systems, tools) is
needed.
•  What is knowable with uncertainty: Model output is available and potentially useful;
however, questions remain as to whether the model itself is specified correctly e.g.,
metrics reflect Knightian uncertainty. (Bayesian methods may be helpful.)
•  What is unknowable: Model output is not available.
•  What one chooses not to know: Incentives overpower model output.
46
GXN-1534GXN-1534
Contextualization: Describe regimes
•  Business as usual (BAU) i.e., sustainable growth
•  Cyclical (typical up and down growth– but same process and similar trend)
•  Structural (move to a different growth path driven by a different process)
•  Providing context is critical:
–  Benchmark to competitors (cohorts)
–  Benchmark to optimal, feasible outcome
–  Show time series
–  Drill into components on a consistent basis
47
GXN-1534GXN-1534
Compelling communication requires multiple interactions
•  Educate as to model’s usefulness as a function of complexity
•  Frame key performance indicators (KPIs)
•  Prototype, socialize, productionize
•  Avoid big-bang projects– include executives in discovery/iteration process
48
GXN-1534GXN-1534
Manage vagueness to communicate more compellingly
“Vagueness blurring and imprecision effectively provide a protective shell to guard [a]
statement against a charge of falsity.” (Rescher, p. 222)
•  Distinguish overarching “truth claim” from (possibly inaccessible) “true details.”
•  Avoid recommendation-rejection skepticism by…
–  Combating cognitive myopia with context
–  Clarifying certainty with respect to aspects of output
–  Separating critical truth-claim components from color-commentary details
•  Paradoxically, judicious omission of details (sometime unavoidable) may produce a
clearer, more compelling message. Finding the “goldilocks” balance is not always
straightforward.
•  Examples
–  Truth claim: I grew up in San Francisco. This vague statement masks the true detail
that I grew up in Danville– loosely part of metropolitan San Francisco.
–  Truth claim: Portfolio is likely to lose 10% or more of its value in the next structural
recession. This vague statement masks details that a simulation algorithm was used
to determine the frequency of losses beyond 10% and the modeler assumes a
structural recession occurs 3% (1-in-33 years) of the time, which marks the amount
of loss– in this case, 10% of starting portfolio value.
49
GXN-1534
Moving from discovery to action
Persuade decision makers regarding…
•  Credibility
•  Likelihood
•  Materiality
•  Addressability
Chronic model weaknesses
•  No feedback loops
•  No thresholds
•  Inadequate spill-over effects
50
GXN-1534GXN-1534
References 1
•  Aikman, David, Piergiorgio Alessandri, Bruno Eklund, Prasanna Gai, Sujit Kapadia, Elizabeth
Martin, Nada Mora, Gabriel Sterne and Matthew Willison, 2009, “Funding liquidity risk in a
quantitative model of systemic stability,” Working Paper 372, Bank of England.
•  Arrow, Kenneth J. and Gerard Debreu, 1954, "Existence of an equilibrium for a competitive
economy,” Econometrica 22 (3), pp. 265–290.
•  Bohn, Jeffrey and Roger Stein 2009, Active Portfolio Management in Practice, Wiley.
•  Diaconis, Persi, 2003, “The problem of thinking too much,” Bulletin American Academy of
Science, Spring, pp. 26-38.
•  Fender, Ingo and John Kiff, 2004, “CDO rating methodology: Some thoughts on model risk and
its implications,” Monetary and Economic Department, BIS.
•  Feynman, Richard P., 1974, “Cargo cult science,” Engineering and Science, June, pp. 10-13.
•  Folinsbee, Kaila E; et al. 2007, "Quantitative approaches to phylogenetics; §5.2: Fount of
stability and confusion: A synopsis of parsimony in systematics,” In Winfried Henke, ed.
Handbook of Paleoanthropology: Primate evolution and human origins: Volume 2, Springer, p.
168.
•  Gordy, Michael B., 2003, “A risk-factor model foundation for ratings-based bank capital rules,”
Journal of Financial Intermediation, 12, pp. 199-232.
•  Gray, Dale and Samuel Malone, 2008, Macrofinancial Risk Analysis, Wiley.
•  Haidt, Jonathan, 2012, The righteous mind: Why good people are divided by politics and
religion, Pantheon Books, New York, NY.
•  Haldane, Andrew G., 2012, “The dog and the frisbee,” Bank of England speech.
•  Hamilton, James, 1994, Time Series Analysis, Princeton University Press.
•  Hansen, Lars P. and Thomas J. Sargent, 2015, “Four types of ignorance,” Journal of Monetary
Economics, 69, pp. 97-113.
51
GXN-1534GXN-1534
References 2
•  Hansen, Lars P. and Thomas J. Sargent, 2007, “Recursive robust estimation and control
without commitment,” Journal of Economic Theory, 136, pp. 1-27.
•  Jablonka, Eva and Marion J. Lamb, 2014, Evolution in four dimensions: genetic, epigenetic,
behavioral, and symbolic variation in the history of life, MIT Press, Cambridge, MA.
•  Kalirai, Harvir and Martin Scheicher, 2002, “Macroeconomic Stress Testing: Preliminary Evidence for
Austria,” Financial Stability Report 3, Oesterreichische Nationalbank, 58-74.
•  Knight, Frank H., 1921, Risk, Uncertainty, and Profit, Hart, Schaffner & Marx; Houghton Mifflin
Company: Boston, MA.
•  Koomey, Jonathan, 2001, Turning numbers into knowledge: Mastering the art of problem solving,
Analytics Press, Oakland, CA.
•  Kuhn, Thomas S., 1977, The essential tension: Selected studies in scientific tradition change,
University of Chicago Press, Chicago.
•  Lewis, Michael, 2010, The big short: Inside the doomsday machine, W.W. Norton & Company: New
York, NY.
•  Lucas, Robert, 1977, “Understanding business cycles,” in K. Brunner and A.H. Metzler, eds.,
Stabilization of the domestic and international economy, Carnegie-Rochester Conference Series on
Public Policy, 7729.
•  Popper, Karl, 1972, Objective knowledge: An evolutionary approach, Clarendon Press, Oxford.
•  Rescher, Nicholas, 2004, “Leibniz quantitative epistemology,” Studia Leibnitiana, 36(2), pp. 210-231.
•  Simon, Herbert A., 1955, “A behavioral model of rational choice,” Quarterly Journal of Economics,
69(1), pp. 99-118.
•  Stark, Philip B., 2015, “Pay no attention to the model behind the curtain,” Working paper, UC
Berkeley.
•  Vasicek, Oldrich A., 1997, “The distribution of loan portfolio value,” Working paper, KMV.
•  Wilson, Edward O., 1999, “Consilience: The unity of knowledge,” First Vintage Books, New York, NY.
52
GXN-1534	
Disclaimers and Important Risk Information
State Street Global Exchange℠ is a trademark of State Street Corporation (incorporated in Massachusetts) and is registered or has registrations pending in
multiple jurisdictions. State Street Associates® is a research partnership between State Street Global Exchange and academia under which this document is
produced. State Street Associates is a registered trademark of State Street Corporation.
Use of report
This document and information herein (together, the “Content”) is for informational, illustrative and/or marketing purposes only and it does not constitute investment
research or investment, legal, or tax advice. The Content provided is not, nor should be construed as, any offer or solicitation to buy or sell any product, service, or
securities or any financial instrument, and it does not constitute any binding contractual arrangement or commitment for State Street Corporation and its
subsidiaries and affiliates (“State Street”) of any kind. The Content provided does not purport to be comprehensive nor intended to replace the exercise of a
client’s own careful independent review regarding any corresponding investment or other financial decision.
Distribution
The Content provided is not intended for retail clients, nor is intended to be relied upon by any person or entity, and is not intended for distribution to or use by any
person or entity in any jurisdiction where such distribution or use would be contrary to applicable law or regulation. No permission is granted to reprint, sell, copy,
distribute, or modify the Content in any form or by any means without the prior written consent of State Street.
Other Important Disclosures
The Content provided has been prepared and obtained from sources believed to be reliable at the time of preparation, however it is provided “as-is” and State
Street makes no guarantee, representation, or warranty of any kind including, without limitation, as to its accuracy, suitability, timeliness, merchantability, fitness for
a particular purpose, non-infringement of third-party rights, or otherwise. Views and opinions expressed herein are those of the author(s) and are subject to
change without notice based on market and other conditions and in any event may not reflect the views of State Street. State Street disclaims all liability, whether
arising in contract, tort or otherwise, for any claims, losses, liabilities, damages (including direct, indirect, special or consequential), expenses or costs arising from
or connected with the Content.
The Content provided may contain certain statements that could be deemed forward-looking statements; any such statements or forecasted information are not
guarantees or reliable indicators for future performance and actual results or developments may differ materially from those depicted or projected. Past
performance is no guarantee of future results.
Prospects, clients or counterparties should be aware of the risks of participating in trading foreign exchange, equities, fixed income or derivative instruments or in
investments in non-liquid or emerging markets. Prospects, clients or counterparties should be aware that products and services outlined may put their principal
and capital at risk and that diversification does not ensure a profit or guarantee against loss.
Australia: This communication is made available in Australia by State Street Bank and Trust Company ABN 70 062 819 630, AFSL 239679 and is intended only
for wholesale clients, as defined in the Corporations Act 2001.
	
53
GXN-1534	
Brazil: The products in this marketing material have not been and will not be registered with the Comissão de Valores Mobiliários, the Brazilian Securities
and Exchange Commission ("CVM"), and any offer of such products is not directed to the general public within the Federative Republic of Brazil ("Brazil").
The information contained in this marketing material is not provided for the purpose of soliciting investments from investors residing in Brazil and no
information in this marketing material should be construed as a public offering or unauthorised distribution of the products within Brazil, pursuant to
applicable Brazilian law and regulations. To our knowledge, no analyst is related to an individual who works at the issuer of the securities subject to the
content of this website; no analyst or their spouse holds, directly or indirectly, securities subject to the reports on this website; no analyst or their spouse is,
directly or indirectly, involved in the acquisition, sale or intermediation of securities subject to the reports on this website; no analyst or their spouse has,
directly or indirectly, a financial interest in relation to the subject matter of the reports on this website; and the relevant analyst's compensation is not,
directly or indirectly, influenced by the revenues arising from the business and financial transactions carried out by the entity to which is associated or
otherwise related.
Canada: The products and services outlined in this document are generally offered in Canada by State Street Bank and Trust Company and/or by State
Street Global Markets Canada Inc.
Hong Kong: This communication is made available in Hong Kong by State Street Bank and Trust Company, which accepts responsibility for its contents,
and is intended for distribution to professional investors only (as defined in the Securities and Futures Ordinance).
Indonesia: This communication is made available in Indonesia by State Street Bank and Trust Company and its affiliates. Neither this communication nor
any copy hereof may be distributed in Indonesia or to any Indonesian citizens wherever they are domiciled or to Indonesian residents except in compliance
with applicable Indonesian capital market laws and regulations. This communication is not an offer of securities in Indonesia. Any securities referred to in
this communication have not been registered with the Capital Market and Financial Institutions Supervisory Agency (BAPEPAM-LK) pursuant to relevant
capital market laws and regulations, and may not be offered or sold within the territory of the Republic of Indonesia or to Indonesian citizens through a
public offering or in circumstances which constitute an offer within the meaning of the Indonesian capital market law and regulations.
Israel: This communication is made available in Israel by State Street Global Markets International Limited, which is not licensed under Israel’s Regulation
of Investment Advice, Investment Marketing and Portfolio Management Law, 1995. This communication may only be distributed to or used by investors in
Israel which are “eligible clients” as listed in the First Schedule to Israel’s Regulation of Investment Advice, Investment Marketing and Portfolio
Management Law 1995.
Japan: This communication is made available in Japan by State Street Global Markets (Japan) which is regulated by the Financial Services Agency of
Japan as a financial instruments firm.
Malaysia: This communication is made available in Malaysia by State Street Global Markets International Limited (“SSGMIL”) which is authorised and
regulated by the United Kingdom’s Financial Conduct Authority. SSGMIL is not licensed within or doing business within Malaysia and the activities that are
being discussed are carried out off-shore. The written materials do not constitute, and should not be construed as constituting: 1) an offer or invitation to
subscribe for or purchase securities or futures in Malaysia or the making available of securities or futures for purchase or subscription in Malaysia; 2) the
provision of investment advice concerning securities or futures; or 3) an undertaking by SSGMIL to manage the portfolio of securities or futures contracts
on behalf of other persons.
Mexico: This communication is distributed by State Street Bank and Trust Company and its affiliates from outside Mexico. State Street is not authorized to
act as an investment advisor in Mexico or as a regulated entity under Mexican law. Any products and services outlined herein will be provided from outside
Mexico exclusively.
New Zealand: This communication is made available in New Zealand by State Street Bank and Trust Company, which accepts responsibility for its
contents, and is intended for distribution only to “wholesale clients” as defined in the Financial Advisers Act 2008 (New Zealand). This communication is not
intended to constitute “financial advice” (as that term is defined in the Financial Advisers Act), or “advice or assistance” (in terms of section 37(5) of the
Securities Markets Act 1988 (New Zealand)), to any person.
	
	
54
GXN-1534	
Oman: This communication is made available in Oman by State Street Bank and Trust Company and its affiliates. The information contained in this communication
is for information purposes and does not constitute an offer for the sale of foreign securities in Oman or an invitation to an offer for the sale of foreign securities.
State Street is neither a bank or financial services provider registered to undertake business in Oman and is neither regulated by the Central Bank of Oman nor the
Capital Market Authority. This document is confidential and is intended solely for the information of the person to whom it has been delivered. No representation or
warranty is given as to the achievement or reasonableness of any research material contained in this communication. Nothing contained in this communication
report is intended to constitute Omani investment, legal, tax, accounting, investment or other professional advice.
Qatar: This communication is made available in Qatar by State Street Bank and Trust Company and its affiliates. The information in this communication has not
been reviewed or approved by the Qatar Central Bank, the Qatar Financial Markets Authority or the Qatar Financial Centre Regulatory Authority, or any other
relevant Qatari regulatory body.
Singapore: This communication is made available in Singapore by State Street Bank and Trust Company, Singapore Branch (“SSBTS”), which holds a wholesale
bank license by the Monetary Authority of Singapore. In Singapore, this communication is only distributed to accredited, institutional investors as defined in the
Singapore Financial Advisers Act (“FAA”). Note that SSBTS is exempt from Sections 27 and 36 of the FAA. When this communication is distributed to overseas
investors as defined in the FAA, note that SSBTS is exempt from Sections 26, 27, 29 and 36 of the FAA. State Street Bank and Trust Company Limited, Singapore
Branch’s Unique Entity Number is T01FC6134G.
South Africa: The products and services outlined in this document are made available in South Africa through either State Street Global Markets International
Limited or State Street Bank and Trust Company, both of which are authorized in South Africa under the Financial Advisory and Intermediary Services Act, 2002 as
a Category I Financial Services Provider; FSP No. 42823 and 42671 respectively.
South Korea: This communication is made available in South Korea by State Street Bank and Trust Company and its affiliates, which accept responsibility for its
contents, and is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within
South Korea, and any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-South Korea securities.
Taiwan: This communication is made available in Taiwan by State Street Bank and Trust Company and its affiliates, which accept responsibility for its contents, and
is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within Taiwan, and
any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-Taiwan securities.
Turkey: This communication is made available in Turkey by State Street Bank and Trust Company and its affiliates. The information included herein is not
investment advice. Investment advisory services are provided by portfolio management companies, brokers and banks without deposit collection licenses within the
scope of the investment advisory agreements to be executed with clients. Any opinions and statements included herein are based on the personal opinions of the
commentators and authors. These opinions may not be suitable to your financial status and your risk and return preferences. Therefore, an investment decision
based solely on the information herein may not be appropriate to your expectations.
United Arab Emirates: This communication is made available in United Arab Emirates by State Street Bank and Trust Company and its affiliates. This
communication does not, and is not intended to, constitute an offer of securities anywhere in the United Arab Emirates and accordingly should not be construed as
such. Nor does the addressing of this research publication to you constitute, or is intended to constitute, the carrying on or engagement in banking, financial and/or
investment consultation business in the United Arab Emirates under the rules and regulations made by the Central Bank of the United Arab Emirates, the Emirates
Securities and Commodities Authority or the United Arab Emirates Ministry of Economy. Any public offer of securities in the United Arab Emirates, if made, will be
made pursuant to one or more separate documents and only in accordance with the applicable laws and regulations. Nothing contained in this communication is
intended to endorse or recommend a particular course of action or to constitute investment, legal, tax, accounting or other professional advice. Prospective
investors should consult with an appropriate professional for specific advice rendered on the basis of their situation. Further, the information contained within this
communication is not intended to lead to the conclusion of any contract of whatsoever nature within the territory of the United Arab Emirates. This communication
has been forwarded to you solely for your information, and may not be reproduced or passed on, directly or indirectly, to any other person or published, in whole or
in part, for any purpose. This communication is addressed only to persons who are professional, institutional or otherwise sophisticated investors.
55
GXN-1534	
United Kingdom and European Union: The products and services outlined herein are only offered to professional clients or eligible counterparties through State
Street Bank and Trust Company, London Branch, authorised and regulated by Federal Reserve Board, authorised and subject to limited regulation by the Prudential
Regulation Authority and subject to regulation by the Financial Conduct Authority and/or State Street Global Markets International Limited, authorised and regulated
by the Financial Conduct Authority Details about the extent of our regulation by the Financial Conduct Authority and Prudential Regulation Authority are available from
us on request. Please note that certain foreign exchange business (spot and certain forward transactions) are not regulated by the Financial Conduct Authority.
United States and Latin America: The products and services outlined in this document are generally offered in the United States and in Latin America by State
Street Bank and Trust Company and/or by State Street Global Markets, LLC.
Please contact your sales representative for further information.
© 2016, State Street Corporation, All rights reserved.
56

More Related Content

Viewers also liked

Managing "Big Data" Application Complexity with CloudGraph
Managing "Big Data" Application Complexity with CloudGraphManaging "Big Data" Application Complexity with CloudGraph
Managing "Big Data" Application Complexity with CloudGraph
Scott Cinnamond
 
Complex User Interfaces Don't Need to Be...Complex
Complex User Interfaces Don't Need to Be...ComplexComplex User Interfaces Don't Need to Be...Complex
Complex User Interfaces Don't Need to Be...Complex
GfK User Centric
 
Effort estimation for web applications
Effort estimation for web applicationsEffort estimation for web applications
Effort estimation for web applications
Nagaraja Gundappa
 
Algorithm Design and Complexity - Course 8
Algorithm Design and Complexity - Course 8Algorithm Design and Complexity - Course 8
Algorithm Design and Complexity - Course 8
Traian Rebedea
 
The Future of Applications: Three Strategies for the High-velocity, Software-...
The Future of Applications: Three Strategies for the High-velocity, Software-...The Future of Applications: Three Strategies for the High-velocity, Software-...
The Future of Applications: Three Strategies for the High-velocity, Software-...
Accenture Technology
 
Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...
Chris Richardson
 

Viewers also liked (6)

Managing "Big Data" Application Complexity with CloudGraph
Managing "Big Data" Application Complexity with CloudGraphManaging "Big Data" Application Complexity with CloudGraph
Managing "Big Data" Application Complexity with CloudGraph
 
Complex User Interfaces Don't Need to Be...Complex
Complex User Interfaces Don't Need to Be...ComplexComplex User Interfaces Don't Need to Be...Complex
Complex User Interfaces Don't Need to Be...Complex
 
Effort estimation for web applications
Effort estimation for web applicationsEffort estimation for web applications
Effort estimation for web applications
 
Algorithm Design and Complexity - Course 8
Algorithm Design and Complexity - Course 8Algorithm Design and Complexity - Course 8
Algorithm Design and Complexity - Course 8
 
The Future of Applications: Three Strategies for the High-velocity, Software-...
The Future of Applications: Three Strategies for the High-velocity, Software-...The Future of Applications: Three Strategies for the High-velocity, Software-...
The Future of Applications: Three Strategies for the High-velocity, Software-...
 
Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...Developing applications with a microservice architecture (SVforum, microservi...
Developing applications with a microservice architecture (SVforum, microservi...
 

Similar to Balancing model performance and complexity in real-world analytics applications

Ch9_slides.ppt chemistry lectures for UG
Ch9_slides.ppt chemistry lectures for UGCh9_slides.ppt chemistry lectures for UG
Ch9_slides.ppt chemistry lectures for UG
DrOjoBolanleGlory
 
Ch9_slides.ppt
Ch9_slides.pptCh9_slides.ppt
Ch9_slides.ppt
QUNHHUNHTRNTRC
 
Ch9 slides
Ch9 slidesCh9 slides
Ch9 slides
fentaw leykun
 
Three methodological issues for system dynamics practice
Three methodological issues for system dynamics practiceThree methodological issues for system dynamics practice
Three methodological issues for system dynamics practice
Andreas Größler
 
MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...
MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...
MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...
Mehdi Merai Ph.D.(c)
 
Managing Complexity
Managing ComplexityManaging Complexity
Managing Complexity
Tathagat Varma
 
Abstract.doc.doc
Abstract.doc.docAbstract.doc.doc
Abstract.doc.doc
butest
 
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
Analytics India Magazine
 
Decision Making Under Uncertainty
Decision Making Under UncertaintyDecision Making Under Uncertainty
Decision Making Under Uncertainty
Bob Prieto
 
Ieaust -aug15_couch-complexity_-_slides_fp_2w
Ieaust  -aug15_couch-complexity_-_slides_fp_2wIeaust  -aug15_couch-complexity_-_slides_fp_2w
Ieaust -aug15_couch-complexity_-_slides_fp_2w
Graeme Couch
 
Uncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial IntelligenceUncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial Intelligence
Experfy
 
Stephen cresswell risk are we missing a trick - 25th june
Stephen cresswell   risk are we missing a trick - 25th juneStephen cresswell   risk are we missing a trick - 25th june
Stephen cresswell risk are we missing a trick - 25th june
Association for Project Management
 
Follow the Ants: The Knowledge Economy & Big Data Management
Follow the Ants: The Knowledge Economy & Big Data ManagementFollow the Ants: The Knowledge Economy & Big Data Management
Follow the Ants: The Knowledge Economy & Big Data Management
Zola Dube
 
4_5_Model Interpretation and diagnostics part 4.pdf
4_5_Model Interpretation and diagnostics part 4.pdf4_5_Model Interpretation and diagnostics part 4.pdf
4_5_Model Interpretation and diagnostics part 4.pdf
Leonardo Auslender
 
OC 6440 scenario planning
OC 6440 scenario planningOC 6440 scenario planning
OC 6440 scenario planning
lindseygibsonphd
 
How To Write Literature Essays. Scholarship essay: How to write literary essay
How To Write Literature Essays. Scholarship essay: How to write literary essayHow To Write Literature Essays. Scholarship essay: How to write literary essay
How To Write Literature Essays. Scholarship essay: How to write literary essay
bdg8266a
 
Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...
Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...
Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...
Michael Rosenblum
 
Re-Learning Strategy with Big Data
Re-Learning Strategy with Big DataRe-Learning Strategy with Big Data
Re-Learning Strategy with Big Data
Malcolm Ryder
 
Statistical Information Retrieval Modelling: from the Probability Ranking Pr...
Statistical Information Retrieval Modelling:  from the Probability Ranking Pr...Statistical Information Retrieval Modelling:  from the Probability Ranking Pr...
Statistical Information Retrieval Modelling: from the Probability Ranking Pr...
Jun Wang
 
Bcu msc cg week 5 rm framework
Bcu msc cg week 5 rm frameworkBcu msc cg week 5 rm framework
Bcu msc cg week 5 rm framework
Stephen Ong
 

Similar to Balancing model performance and complexity in real-world analytics applications (20)

Ch9_slides.ppt chemistry lectures for UG
Ch9_slides.ppt chemistry lectures for UGCh9_slides.ppt chemistry lectures for UG
Ch9_slides.ppt chemistry lectures for UG
 
Ch9_slides.ppt
Ch9_slides.pptCh9_slides.ppt
Ch9_slides.ppt
 
Ch9 slides
Ch9 slidesCh9 slides
Ch9 slides
 
Three methodological issues for system dynamics practice
Three methodological issues for system dynamics practiceThree methodological issues for system dynamics practice
Three methodological issues for system dynamics practice
 
MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...
MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...
MLX 2018 - Marcos López de Prado, Lawrence Berkeley National Laboratory Comp...
 
Managing Complexity
Managing ComplexityManaging Complexity
Managing Complexity
 
Abstract.doc.doc
Abstract.doc.docAbstract.doc.doc
Abstract.doc.doc
 
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
Deciphering AI - Unlocking the Black Box of AIML with State-of-the-Art Techno...
 
Decision Making Under Uncertainty
Decision Making Under UncertaintyDecision Making Under Uncertainty
Decision Making Under Uncertainty
 
Ieaust -aug15_couch-complexity_-_slides_fp_2w
Ieaust  -aug15_couch-complexity_-_slides_fp_2wIeaust  -aug15_couch-complexity_-_slides_fp_2w
Ieaust -aug15_couch-complexity_-_slides_fp_2w
 
Uncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial IntelligenceUncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial Intelligence
 
Stephen cresswell risk are we missing a trick - 25th june
Stephen cresswell   risk are we missing a trick - 25th juneStephen cresswell   risk are we missing a trick - 25th june
Stephen cresswell risk are we missing a trick - 25th june
 
Follow the Ants: The Knowledge Economy & Big Data Management
Follow the Ants: The Knowledge Economy & Big Data ManagementFollow the Ants: The Knowledge Economy & Big Data Management
Follow the Ants: The Knowledge Economy & Big Data Management
 
4_5_Model Interpretation and diagnostics part 4.pdf
4_5_Model Interpretation and diagnostics part 4.pdf4_5_Model Interpretation and diagnostics part 4.pdf
4_5_Model Interpretation and diagnostics part 4.pdf
 
OC 6440 scenario planning
OC 6440 scenario planningOC 6440 scenario planning
OC 6440 scenario planning
 
How To Write Literature Essays. Scholarship essay: How to write literary essay
How To Write Literature Essays. Scholarship essay: How to write literary essayHow To Write Literature Essays. Scholarship essay: How to write literary essay
How To Write Literature Essays. Scholarship essay: How to write literary essay
 
Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...
Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...
Managing the Earthquake: Surviving Major Database Architecture Changes (rev.2...
 
Re-Learning Strategy with Big Data
Re-Learning Strategy with Big DataRe-Learning Strategy with Big Data
Re-Learning Strategy with Big Data
 
Statistical Information Retrieval Modelling: from the Probability Ranking Pr...
Statistical Information Retrieval Modelling:  from the Probability Ranking Pr...Statistical Information Retrieval Modelling:  from the Probability Ranking Pr...
Statistical Information Retrieval Modelling: from the Probability Ranking Pr...
 
Bcu msc cg week 5 rm framework
Bcu msc cg week 5 rm frameworkBcu msc cg week 5 rm framework
Bcu msc cg week 5 rm framework
 

More from State Street

State Street LIVE: Munich
State Street LIVE: MunichState Street LIVE: Munich
State Street LIVE: Munich
State Street
 
State Street LIVE: Research Retreat
State Street LIVE: Research RetreatState Street LIVE: Research Retreat
State Street LIVE: Research Retreat
State Street
 
Massachusetts Conference for Women
Massachusetts Conference for WomenMassachusetts Conference for Women
Massachusetts Conference for Women
State Street
 
European Client Conference
European Client ConferenceEuropean Client Conference
European Client Conference
State Street
 
State Street LIVE
State Street LIVEState Street LIVE
State Street LIVE
State Street
 
European Client Conference
European Client ConferenceEuropean Client Conference
European Client Conference
State Street
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
State Street
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
State Street
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
State Street
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
State Street
 
Brexit Explainer Series
Brexit Explainer SeriesBrexit Explainer Series
Brexit Explainer Series
State Street
 
State Street Global Research Retreats
State Street Global Research RetreatsState Street Global Research Retreats
State Street Global Research Retreats
State Street
 
Ron O'Hanley's Open Letter to Congress
Ron O'Hanley's Open Letter to CongressRon O'Hanley's Open Letter to Congress
Ron O'Hanley's Open Letter to Congress
State Street
 
Asset Owner Client Quote
Asset Owner Client QuoteAsset Owner Client Quote
Asset Owner Client Quote
State Street
 
Alternatives Experts Wanted
Alternatives Experts WantedAlternatives Experts Wanted
Alternatives Experts Wanted
State Street
 
Search for Transparency
Search for TransparencySearch for Transparency
Search for Transparency
State Street
 
Acceleration into Alternatives
Acceleration into AlternativesAcceleration into Alternatives
Acceleration into Alternatives
State Street
 
PWC Quote
PWC QuotePWC Quote
PWC Quote
State Street
 
Ilmarinen Quote
Ilmarinen QuoteIlmarinen Quote
Ilmarinen Quote
State Street
 
Asset Owners Newsletter (Italian)
Asset Owners Newsletter (Italian)Asset Owners Newsletter (Italian)
Asset Owners Newsletter (Italian)
State Street
 

More from State Street (20)

State Street LIVE: Munich
State Street LIVE: MunichState Street LIVE: Munich
State Street LIVE: Munich
 
State Street LIVE: Research Retreat
State Street LIVE: Research RetreatState Street LIVE: Research Retreat
State Street LIVE: Research Retreat
 
Massachusetts Conference for Women
Massachusetts Conference for WomenMassachusetts Conference for Women
Massachusetts Conference for Women
 
European Client Conference
European Client ConferenceEuropean Client Conference
European Client Conference
 
State Street LIVE
State Street LIVEState Street LIVE
State Street LIVE
 
European Client Conference
European Client ConferenceEuropean Client Conference
European Client Conference
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
 
Hannah Grove at FT Live
Hannah Grove at FT LiveHannah Grove at FT Live
Hannah Grove at FT Live
 
Brexit Explainer Series
Brexit Explainer SeriesBrexit Explainer Series
Brexit Explainer Series
 
State Street Global Research Retreats
State Street Global Research RetreatsState Street Global Research Retreats
State Street Global Research Retreats
 
Ron O'Hanley's Open Letter to Congress
Ron O'Hanley's Open Letter to CongressRon O'Hanley's Open Letter to Congress
Ron O'Hanley's Open Letter to Congress
 
Asset Owner Client Quote
Asset Owner Client QuoteAsset Owner Client Quote
Asset Owner Client Quote
 
Alternatives Experts Wanted
Alternatives Experts WantedAlternatives Experts Wanted
Alternatives Experts Wanted
 
Search for Transparency
Search for TransparencySearch for Transparency
Search for Transparency
 
Acceleration into Alternatives
Acceleration into AlternativesAcceleration into Alternatives
Acceleration into Alternatives
 
PWC Quote
PWC QuotePWC Quote
PWC Quote
 
Ilmarinen Quote
Ilmarinen QuoteIlmarinen Quote
Ilmarinen Quote
 
Asset Owners Newsletter (Italian)
Asset Owners Newsletter (Italian)Asset Owners Newsletter (Italian)
Asset Owners Newsletter (Italian)
 

Recently uploaded

一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
z6osjkqvd
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Social Samosa
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
bmucuha
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
Monthly Management report for the Month of May 2024
Monthly Management report for the Month of May 2024Monthly Management report for the Month of May 2024
Monthly Management report for the Month of May 2024
facilitymanager11
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
writing report business partner b1+ .pdf
writing report business partner b1+ .pdfwriting report business partner b1+ .pdf
writing report business partner b1+ .pdf
VyNguyen709676
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
nyfuhyz
 
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Aggregage
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
Márton Kodok
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
hyfjgavov
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
Social Samosa
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
v7oacc3l
 
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
taqyea
 
Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
jitskeb
 
University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma TranscriptUniversity of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
soxrziqu
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
ihavuls
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
AndrzejJarynowski
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
nuttdpt
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Timothy Spann
 

Recently uploaded (20)

一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
Monthly Management report for the Month of May 2024
Monthly Management report for the Month of May 2024Monthly Management report for the Month of May 2024
Monthly Management report for the Month of May 2024
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
writing report business partner b1+ .pdf
writing report business partner b1+ .pdfwriting report business partner b1+ .pdf
writing report business partner b1+ .pdf
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
 
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
 
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
 
Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
 
University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma TranscriptUniversity of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
 

Balancing model performance and complexity in real-world analytics applications

  • 1. GXN-1534 Balancing model performance and complexity in real-world analytics applications Dr. Jeffrey R. Bohn Chief Science Officer Head of GX Labs jbohn@statestreet.com Limited Access
  • 2. GXN-1534 Outline •  Introduction •  CDO “case study”: Balancing complexity and performance •  Defining complexity •  Driving complexity •  Simplicity and complexity •  Model performance •  Choosing a model •  Communicating complexity •  Recommendations 2
  • 3. GXN-1534GXN-1534 The wisdom of Archilochus (Greek lyric poet from Paros, lived 680 BCE to 645 BCE) The fox knows many things, but the hedgehog knows one big thing. 3
  • 6. GXN-1534 Big Short: Collateralized debt obligations (CDOs) •  Simple in concept: Portfolio of many assets plus cash waterfall changes risk profile •  Complex in structure: Rules in cash waterfall; over-collateralization and interest- coverage triggers •  Simple risk ratings: Diversity score (binomial expansion) or Gaussian copula plus “shading” based on a committee •  Complex market structure: Borrowers, appraisers, mortgage brokers, commercial banks, mortgage servicers, investment bankers, fund managers, rating agencies, institutional investors, regulators (Fed, NAIC, SEC) lawyers and many consultants •  Simplicity in incentives: Money for nothing– or-- at the least-- mis-representation of actual risk relative to fees and spreads Communication regarding CDO risk to senior executives at financial institutions was mostly not cognitively compelling. “Real risk was not volatility; real risk was stupid investment decisions.” Lewis (2010) 6
  • 7. GXN-1534 Failure of a simple CDO model •  Materially underestimated risk of mezzanine and senior tranches •  Often over-simplified waterfall rules– e.g., equity prioritized •  Unrealistic assumptions with respect to default rates •  Herd mentality on CDO ratings •  Correlation dynamics not well represented– i.e., correlation went toward 1 in crisis. 7
  • 8. GXN-1534 Portfolio Simulation: A more complex, but better model Sampling across the Latent Factors to simulation various risk conditions N securities Portfolio Valuation Model … Random draws Single Position K latent factors Factor draws Factor realizationsFactor Betas idiosyncratic value of ith security K = 30 factors N ≈ 10k ~ 500k positions/idiosyncratic draws J ≈ 1m ~ 10m iterations How much computing power needed? J iterations Distribution Repeat J times value of portfolio for one iteration equity fixed income other Repeat N times Source: State Street Global Exchange℠ Limited Access Figure provided for illustrative purposes only. 8
  • 9. GXN-1534 Multi-asset-class portfolio-risk modeling Use-objective focuses on incorporating risk strategy into portfolio management probability Start of TailExpected Tail Loss stress losses Portfolio value Expected Value Risk-appetite assessment •  Model guides executives as to whether allocation is prudent from a risk perspective Sub-portfolio/manager/hedging evaluation •  Model assists in evaluating how well components of portfolio contribute to overall risk/return Scenario analysis •  Model provide input into strategic discussions on portfolio construction/management Figure provided for illustrative purposes. 9
  • 10. GXN-1534 Interpreting simulated scenarios Reverse Stress Testing Values at horizon Search through factor space Inflation S&P Oil Price GDP Rates Macro scenarios What factor realizations generate a portfolio value in the specified confidence interval of portfolio distribution? Source: State Street Global Exchange℠ Limited Access Figure provided for illustrative purposes. 10
  • 12. GXN-1534GXN-1534 Defining complexity •  Epistemic: Hard to understand conceptually •  Computational: Algorithm hard to understand and/or implement •  Dynamic: System changes over time– sometimes as a function of itself or its users •  Human minds are memory and predicting mechanisms: Does the complexity arise from quantity of “insight chunks” that need to be remembered or from the quantity of steps/inter-relationships in the predictive model? •  Simple heuristics may arise from highly complex models/systems. Complexity may arise from incorporating nuances into interpretations/decisions based on a given heuristic. 12
  • 13. GXN-1534GXN-1534 Consilience: A frequent driver toward complexity A "jumping together” of knowledge by the linking of facts and fact-based theory across disciplines to create a common groundwork of explana=on. Wilson (1999) p. 8 13
  • 14. GXN-1534 What makes decision-support analytics complex? •  Multi-variate optimization problem– often with controls and results playing out over different time horizons. •  Uncertainty– “risk” defined by known distributions and “Knightian uncertainty” defined by unknown models/data-generating processes •  Interconnectedness– both explicit and implicit. •  Hierarchies of relationships and relevant assumptions •  Separation of point predictions/estimates and distribution estimates 14
  • 15. GXN-1534GXN-1534 Drivers of model complexity •  Nonstationarity: –  Macro-economic regime changes –  Firms changing leverage –  Governments changing regulation •  Heterogeneous return distributions by asset class –  Non-normal –  Skewed –  Fat tails –  Liquidity •  Capturing obscure, but material risks –  Correlated exposure to a latent factor (e.g., US housing market) –  Sectorally diversified investment-grade bond portfolio may constitute material concentration risk –  Wrong-way, counterparty risk to an over-hedged energy company •  Human reaction to changing - Bank management reaction to stress environment by changing underwriting standards - Not selling in a down market and distressed-selling in an up market not recognized - Inability or unwillingness to understand a model and its implications may lead to value- destroying behavior - Traders gaming particularly sensitive aspects of a risk model 15
  • 16. GXN-1534 Shift in risk modeling favors more model complexity Normal Distribution Assumptions Linear estimation is good enough Current Mainstream Paradigm (examples) Focus on 2nd moment Linear Regressions Bias toward tractable “closed form” solutions Examples FastData™ BigComputation™ Machine Learning Empirical orientation Enablers Skewed, Fat-tailed distributions Focus on Skewness and Tail, generated by simulations Deep Learning Need to recognize non-linearities Empirical approach with recognition of non-linearity Examples New Approaches Source: State Street Global Exchange℠ 16
  • 18. GXN-1534GXN-1534 Simple/Complex Enough Albert Einstein displayed the following aphorism in his office: “Things that are difficult to do are being done from the wrong centers and are not worth doing.” -- Diaconis (2003) Albert Einstein may have said something like “[Theories], [Education] or [Things] should be as simple as possible, but no simpler.” 18
  • 19. GXN-1534GXN-1534 Simplicity may not always be the primary or best criterion •  Also attributed to Einstein: "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction.” •  Simplicity should be one goal, not a hard criterion. •  Usefulness should also be a goal and should be a hard criterion: Thus, simple enough, but no simpler. This thought can lead to… •  Occam’s Razor: “Among competing hypotheses, the one with the fewest assumptions should be selected.” –  William of Ockham’s actual quote (I think): “Numquam ponenda est pluralitas sine necessitate [Plurality must never be posited without necessity]” –  Bertrand Russell’s version: "Whenever possible, substitute constructions out of known entities for inferences to unknown entities.” •  Simplicity in concept may belie complexity in reality– e.g., biological evolution, construct a portfolio that finds highest return/risk •  Not likely to be a global principle– important that model “suitably” explains/predicts data –  “Suitably” implies matching model with use objective –  Over time, more complex models may explain data better and open new vistas e.g., atomic theory, plate tectonics, Black-Scholes, light theory (Newton’s “simpler” particle’s versus Huygen’s waves)– bottom-up, factor-based portfolio simulations? 19
  • 21. GXN-1534 Hume’s problem: When will a model perform? •  When is it reasonable to think the future will be like the past? •  Rephrase– what part of a model’s/system’s/algorithm’s structure will operate in the future in the same way it has in the past? –  Means are difficult to estimate and historical observation may be a poor guide –  Volatility may be easier (than means) to estimate, but still may change so much that historical observation is likely to be a poor guide. –  Underlying correlation structure may change-- but not as much as volatility –  Co-skew? –  Co-kurtosis? How much can/should non-linear-process-inducing feedback loops and tipping points be included in a model and can historical observation help? How does model complexification affect strategies for communicating to quantitatively- informed business executives? 21
  • 22. GXN-1534GXN-1534 How do we know a portfolio-risk system/model works? “Science is no inexorable march to truth, mediated by the collection of objective information and the destruction of ancient superstition. Scientists, as ordinary human beings, unconsciously reflect in their theories the social and political constraints of their times.” -- Stephen Jay Gould Karl Popper’s process applied to portfolio-risk analysis: •  Problem situation: “Risk-appetite consistent allocation” •  Tentative theories: “Hypothesize scenarios” •  Error elimination: “Stress the stress test i.e., simulate” However, falsifying every scenario is not possible; further, Gould argues to beware theory-laden analyses. 22
  • 23. GXN-1534 Complex models perform better: Evolution •  Darwin (1859): Natural selection is a key mechanism of evolution based on the differential survival and reproduction of individuals due to phenotype differences. •  Romanes (1895): Neo-darwinism refers to germ-plasm theory advocated by Wallace and Weissmann. •  Huxley (1942): Modern synthesis drew together multiple fields of biology marrying natural selection, genetics, natural population analysis, systematics, etc. •  Today’s models of evolution are much more complex: –  Genetic networks composed of tens to hundreds of genes interact –  “Regulatory” genes behave conditional on the environment –  Some hereditary variations are nonrandom in origin –  Some acquired information is inherited (epigenetic inheritance) –  Evolutionary change can result from instruction as well as selection Jablonka and Lamb (2014) 23
  • 24. GXN-1534 Complex models perform better: Risk analytics •  Capital-asset pricing model •  Markowitzian portfolio theory •  Arrow-Debreu •  Black-Scholes-Merton •  Continuous-time finance •  Value at risk •  Monte-Carlo simulation •  Expected-tail loss Are these “advances” i.e., model complexification worth the investment? 24
  • 26. GXN-1534GXN-1534 Criteria for deciding degree of model complexity •  Model performance •  Ease of communicating actionable model output to quantitatively-informed executives •  Ignores extraneous information •  Balances model uncertainty and credibility of priors •  Minimizes over-fitting risk: Sample size relative to number of parameters •  Resistant to manipulation as a consequence of incentives “Because complexity generates uncertainty, not risk, it requires a regulatory response grounded in simplicity, not complexity.” (Haldane, 2005, p. 19) What does this mean in practice? 26
  • 27. GXN-1534GXN-1534 Questioning the entire modeling enterprise •  Preproducibility is a prerequisite for attempting to reproduce a result: it involves providing an adequate description of an experiment or analysis for the work to be re-undertaken. It requires documentation, openness, and communication. •  Quantifauxcation is to assign a meaningless number, then pretend that since it’s quantitative, it’s meaningful. Usually involves some combination of data, pure invention, ad- hoc models, inappropriate statistics, and logical lacunae. •  Cost of most policy cost-benefit analyses is high: lost rationality •  Rates vs. probabilities –  Randomness created by taking a random sample, assigning subjects at random, etc. –  Probability model invented (assumed) for data that world generates– inferences are only as good as the assumptions –  Aleatory: Coin toss, die roll, under some circumstances, behave “as if” random –  Epistemic: Stuff we don’t know –  Trials are random, have same chance of success and have known dependence– can quantify estimate uncertainty. –  Ignorance does not equal randomness –  Tendency to treat haphazard as random –  Probability as metaphor (From Stark 2015) 27
  • 28. GXN-1534GXN-1534 Model risk •  Mis-specify data-generating process: Fail to estimate the “true” distribution due to the fact that the underlying process differs materially from the model’s process assumptions. –  Second-order assessment of misspecification also important. –  What is range of wrong estimates? –  Are out-of-sample tests showing under-estimates in times of stress? •  Mis-estimate parameters: While the assumed data-generating process may be reasonable, still fail to estimate “true” distribution based on parameterization problems. –  Are there enough data points to credibly estimate parameters? –  Can model structure be used to infer tail events without requisite data? 28
  • 29. GXN-1534GXN-1534 Risk versus uncertainty •  Frank Knight (Knight, 1921): "Uncertainty must be taken in a sense radically distinct from the familiar notion of risk, from which it has never been properly separated.... The essential fact is that 'risk' means in some cases a quantity susceptible of measurement, while at other times it is something distinctly not of this character; and there are far- reaching and crucial differences in the bearings of the phenomena depending on which of the two is really present and operating.... It will appear that a measurable uncertainty, or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all.” •  Risk is known unknowns i.e., data generating process, inter-relationships arising from model structure, parameters imply known distributions •  Knightian uncertainty is unknown unknowns i.e., data generating process, model structure and parameters are unknown 29
  • 30. GXN-1534GXN-1534 Types of ignorance Hansen and Sargent (2015) Xt+1 =κ Xt + βUt +αWt+1 Xt ≡ Observable state variable at time t Ut ≡ Control variable Wt+1 ≡ Random shock to process 1.  “Bayesian decision maker” does not know β, but trusts prior prob. distribution 2.  “Robust Bayesian decision maker” does not trust prior distribution for response coefficient β; but uses operators to twist prior distributions to generate conservative est. 3. “Robust decision maker” uses a multiplier or constraint preferences to express doubt regarding probability distribution of W conditional on X and U. 4. “Robust decision maker” asserts ignorance as in 3 by adjusting an entropy penalty to Make model robust to particular alternative probability models. “The trouble with most folks isn’t so much their ignorance, as knowing so many things that ain’t so.” Josh Billings related by Friedman (1965) 30
  • 31. GXN-1534GXN-1534 Evaluating the adequacy of a theory/model 1.  Accuracy [out-of-sample confirmation of estimated probability distribution and contributions of underlying components to that distribution] 2.  Consistency (both internal and external) [multi-asset-class, assets & liabilities] 3.  Broadness in scope [granularity and comprehensiveness] 4.  Simplicity [complex enough to capture dynamics, but simple enough to be diagnosed and communicated to a quantitatively-informed business head] 5.  Fruitfulness [output substantively contributes to impactful decisions] Depending on the theory under evaluation, criteria may contradict each other so a relative weighting may be needed i.e., given a particular circumstance, some criteria are more important than others. Kuhn (1977) In portfolio risk analysis, we typically add Timeliness to the evaluation process– a successful theory/model/system that cannot provide timely output is useless. 31
  • 33. GXN-1534 Understanding the decision maker: Elephant and the rider •  Elephant: Automatic processes •  Rider: Controlled processes •  Modeling decision making –  Humean model: Reason is a servant –  Platonic model: Reason could and should rule –  Jeffersonian model: Head and heart are co-emperors •  “Seeing that” vs. “Reasoning why” •  Rationalist delusion: Maintaining healthy skepticism of reason– smarter people rationalize better People who devote their lives to studying something often come to believe that the object of their fascination is the key to understanding everything. Location 58 Conscious reasoning functions like a press secretary who automatically justifies any position taken by the president. p. 106 And as reasoning is not the source, whence either disputant derives his tenets; it is in vain to expect, that any logic, which speaks not to the affections, will ever engage him to embrace sounder principles. – David Hume Haidt (2012) 33
  • 34. GXN-1534 How individuals make decisions Agree on values Disagree on values Agree on facts Computational decision Negotiate Disagree on facts Experiment Paralysis or chaos 34 From Koomey (2001) figure 19.1 p. 88.
  • 35. GXN-1534 How to communicate in a cognitively compelling way? •  Research audience… –  Culture (values and biases) –  Education (general and specific to communicated analytics) –  Incentives (degree of departure from objectivity) •  Distill message... –  Empirical vs. rational/theoretical (where is the model in this iterative evolution?) –  Transparent assumptions –  Frank assessment of model uncertainty –  Link to audience’s narrative/values •  Use intuitive framing and strong visuals… –  Relevant metaphors –  Examples from different, but similar domains (e.g., health/medical) –  Design compelling graphs and figures 35
  • 36. GXN-1534 Working memory and compelling communication •  Working memory –  Operates over a few seconds –  Temporary storage –  Manipulates attention –  Focuses attention –  Resists distractions –  Guides decision-making •  Can only process 5 to 9 “chunks” of information within working memory at any given moment in time (Miller, 1955) •  Deviating from expectations typically causes the listener to disengage •  Working memory dis-fluently “chunks” instead of always focusing on what matters •  Working memory “calls” long-term memory to assist in processing; if nothing is there, cognitive flow is broken– result is likely disengagement Education is essential to build up long-term memory 36
  • 37. GXN-1534 Building a narrative 1.  Identify the [the set of] focal issue(s) 2.  Determine the key micro or local forces impacting the focal issue(s) 3.  Identify the key macro or global forces impacting the focal issue(s) 4.  Rank by importance and uncertainty– distinguish parameterization of probability distributions (known unknowns) from model uncertainty (unknown unknowns) 5.  Select scenario logic in terms of the parameters (and maybe models) to adjust to show range of possible outputs 6.  Flesh out the scenarios of most importance and highest likelihood– drill into details of micro/macro forces and nature of parameterized probabilities and uncertainty 7.  Determine implications 8.  Select leading indicators and signposts to monitor evolution of scenario in light of decisions 37 From Koomey (2001)
  • 38. GXN-1534 Communicating Uncertainty •  Explain signal and noise in specific terms •  Communicate how model disentangles signal and noise •  Identify and root out data biases •  Educate on error bars and confidence intervals •  Beware illusory precision– clarify how much precision is possible given data/model Sampling error does not necessarily equal “uncertainty” in terms of implications of model output 38
  • 39. GXN-1534GXN-1534 Cognitive biases •  False dichotomy: Presenting two choices such that it seems they are the only possibilities. –  Simple vs. complex model –  Use no models vs. use only one model •  Perfect as the enemy of the good (or good enough) •  Red herrings and missing forest for the trees •  Biases –  Affect heuristic: Analyst or executive has “fallen in love with” a particular output so that they minimize model problems and exaggerate model strengths. –  Groupthink –  Saliency bias: Overly influence by analogous, past success –  Confirmation bias –  Availability bias –  Anchoring bias –  Halo effect: Impression of model author, analyst or even model influences interpretation –  Sunk-cost fallacy: A particular model output has driven strategy/investment –  Overconfidence –  Disaster neglect –  Loss aversion Traps arising from logical fallacies and cognitive biases 39
  • 40. GXN-1534GXN-1534 Key points to remember for compelling communication •  Frame within a narrative •  Avoid “quantifauxcation” •  Contextualize (across time and across cohorts) •  Address biases: Highlight data selection concerns and explain assumptions & process •  Use transparency in model estimation process to spark questions and debate •  Compare output from multiple models (when possible) •  Visualize data– encourage interactive diagnostics and drill-down •  Emphasize actionable insight •  Educate –  Explain key components of analytical process –  Teach how to understand confidence intervals (noise vs. signal) Build on understanding: Descriptive, prescriptive and cognitive Move from analysis (breaking into components) to synthesis (re-assembling with insight) 40
  • 41. GXN-1534 Principles of risk-data visualization •  Match output to use cases –  Concentration risk assessment –  Risk appetite assessment (stress testing) –  Position-level limits/allocation •  Prepare for multiple dimensions (e.g., region, sector, asset class, customer type, size) •  Incorporate drill-down capability •  Contextualize output (e.g., benchmarks, time series, scenario-based) •  Use robust statistics (e.g., median, inter-quartile, mean absolute deviation) •  Use techniques to address data difficulties (e.g., Winsorization, shrinkage) •  Target near-instantaneous rendering of decision-support output Risk data tend to be defined by outliers 41
  • 42. GXN-1534 Data are all figurative for illustrative purposes Multiple attributes represented in two dimensions Expected Returns vs. Volatility by Exposure Size – Sharpe Ratio as Color High Sharpe Ratios, but small positions OK Sharpe Ratios, and larger position 42
  • 43. GXN-1534 Data are all figurative for illustrative purposes So which is richer, from a data insight perspective? 43
  • 44. GXN-1534 Improve visualization independent of model complexity •  Google study (Tuch, et al., 2012) found for websites: –  Visually complex websites are less appealing –  Prototypical websites (for a given category) are more appealing –  Simpler design is rated higher •  What makes analytical output compelling and credible? –  Prototypicality: Basic mental image one’s brain creates to categorize everything with which you interact. –  Cognitive fluency: One’s brain prefers output that is easier to process. –  Mere exposure effect: Familiarity arising from repeated exposure. –  Metric balancing: Too many metrics equals no understanding. 44
  • 46. GXN-1534GXN-1534 Be transparent as to epistemic nature of model output •  What the executive doesn’t know, but is knowable: Model output is available and useful e.g., credible metrics identify risk (in the technical sense.) •  What the executive or the analyst don’t know yet, but is knowable: Proof-of-concept model is available; however, more investment (e.g., data, analysts, systems, tools) is needed. •  What is knowable with uncertainty: Model output is available and potentially useful; however, questions remain as to whether the model itself is specified correctly e.g., metrics reflect Knightian uncertainty. (Bayesian methods may be helpful.) •  What is unknowable: Model output is not available. •  What one chooses not to know: Incentives overpower model output. 46
  • 47. GXN-1534GXN-1534 Contextualization: Describe regimes •  Business as usual (BAU) i.e., sustainable growth •  Cyclical (typical up and down growth– but same process and similar trend) •  Structural (move to a different growth path driven by a different process) •  Providing context is critical: –  Benchmark to competitors (cohorts) –  Benchmark to optimal, feasible outcome –  Show time series –  Drill into components on a consistent basis 47
  • 48. GXN-1534GXN-1534 Compelling communication requires multiple interactions •  Educate as to model’s usefulness as a function of complexity •  Frame key performance indicators (KPIs) •  Prototype, socialize, productionize •  Avoid big-bang projects– include executives in discovery/iteration process 48
  • 49. GXN-1534GXN-1534 Manage vagueness to communicate more compellingly “Vagueness blurring and imprecision effectively provide a protective shell to guard [a] statement against a charge of falsity.” (Rescher, p. 222) •  Distinguish overarching “truth claim” from (possibly inaccessible) “true details.” •  Avoid recommendation-rejection skepticism by… –  Combating cognitive myopia with context –  Clarifying certainty with respect to aspects of output –  Separating critical truth-claim components from color-commentary details •  Paradoxically, judicious omission of details (sometime unavoidable) may produce a clearer, more compelling message. Finding the “goldilocks” balance is not always straightforward. •  Examples –  Truth claim: I grew up in San Francisco. This vague statement masks the true detail that I grew up in Danville– loosely part of metropolitan San Francisco. –  Truth claim: Portfolio is likely to lose 10% or more of its value in the next structural recession. This vague statement masks details that a simulation algorithm was used to determine the frequency of losses beyond 10% and the modeler assumes a structural recession occurs 3% (1-in-33 years) of the time, which marks the amount of loss– in this case, 10% of starting portfolio value. 49
  • 50. GXN-1534 Moving from discovery to action Persuade decision makers regarding… •  Credibility •  Likelihood •  Materiality •  Addressability Chronic model weaknesses •  No feedback loops •  No thresholds •  Inadequate spill-over effects 50
  • 51. GXN-1534GXN-1534 References 1 •  Aikman, David, Piergiorgio Alessandri, Bruno Eklund, Prasanna Gai, Sujit Kapadia, Elizabeth Martin, Nada Mora, Gabriel Sterne and Matthew Willison, 2009, “Funding liquidity risk in a quantitative model of systemic stability,” Working Paper 372, Bank of England. •  Arrow, Kenneth J. and Gerard Debreu, 1954, "Existence of an equilibrium for a competitive economy,” Econometrica 22 (3), pp. 265–290. •  Bohn, Jeffrey and Roger Stein 2009, Active Portfolio Management in Practice, Wiley. •  Diaconis, Persi, 2003, “The problem of thinking too much,” Bulletin American Academy of Science, Spring, pp. 26-38. •  Fender, Ingo and John Kiff, 2004, “CDO rating methodology: Some thoughts on model risk and its implications,” Monetary and Economic Department, BIS. •  Feynman, Richard P., 1974, “Cargo cult science,” Engineering and Science, June, pp. 10-13. •  Folinsbee, Kaila E; et al. 2007, "Quantitative approaches to phylogenetics; §5.2: Fount of stability and confusion: A synopsis of parsimony in systematics,” In Winfried Henke, ed. Handbook of Paleoanthropology: Primate evolution and human origins: Volume 2, Springer, p. 168. •  Gordy, Michael B., 2003, “A risk-factor model foundation for ratings-based bank capital rules,” Journal of Financial Intermediation, 12, pp. 199-232. •  Gray, Dale and Samuel Malone, 2008, Macrofinancial Risk Analysis, Wiley. •  Haidt, Jonathan, 2012, The righteous mind: Why good people are divided by politics and religion, Pantheon Books, New York, NY. •  Haldane, Andrew G., 2012, “The dog and the frisbee,” Bank of England speech. •  Hamilton, James, 1994, Time Series Analysis, Princeton University Press. •  Hansen, Lars P. and Thomas J. Sargent, 2015, “Four types of ignorance,” Journal of Monetary Economics, 69, pp. 97-113. 51
  • 52. GXN-1534GXN-1534 References 2 •  Hansen, Lars P. and Thomas J. Sargent, 2007, “Recursive robust estimation and control without commitment,” Journal of Economic Theory, 136, pp. 1-27. •  Jablonka, Eva and Marion J. Lamb, 2014, Evolution in four dimensions: genetic, epigenetic, behavioral, and symbolic variation in the history of life, MIT Press, Cambridge, MA. •  Kalirai, Harvir and Martin Scheicher, 2002, “Macroeconomic Stress Testing: Preliminary Evidence for Austria,” Financial Stability Report 3, Oesterreichische Nationalbank, 58-74. •  Knight, Frank H., 1921, Risk, Uncertainty, and Profit, Hart, Schaffner & Marx; Houghton Mifflin Company: Boston, MA. •  Koomey, Jonathan, 2001, Turning numbers into knowledge: Mastering the art of problem solving, Analytics Press, Oakland, CA. •  Kuhn, Thomas S., 1977, The essential tension: Selected studies in scientific tradition change, University of Chicago Press, Chicago. •  Lewis, Michael, 2010, The big short: Inside the doomsday machine, W.W. Norton & Company: New York, NY. •  Lucas, Robert, 1977, “Understanding business cycles,” in K. Brunner and A.H. Metzler, eds., Stabilization of the domestic and international economy, Carnegie-Rochester Conference Series on Public Policy, 7729. •  Popper, Karl, 1972, Objective knowledge: An evolutionary approach, Clarendon Press, Oxford. •  Rescher, Nicholas, 2004, “Leibniz quantitative epistemology,” Studia Leibnitiana, 36(2), pp. 210-231. •  Simon, Herbert A., 1955, “A behavioral model of rational choice,” Quarterly Journal of Economics, 69(1), pp. 99-118. •  Stark, Philip B., 2015, “Pay no attention to the model behind the curtain,” Working paper, UC Berkeley. •  Vasicek, Oldrich A., 1997, “The distribution of loan portfolio value,” Working paper, KMV. •  Wilson, Edward O., 1999, “Consilience: The unity of knowledge,” First Vintage Books, New York, NY. 52
  • 53. GXN-1534 Disclaimers and Important Risk Information State Street Global Exchange℠ is a trademark of State Street Corporation (incorporated in Massachusetts) and is registered or has registrations pending in multiple jurisdictions. State Street Associates® is a research partnership between State Street Global Exchange and academia under which this document is produced. State Street Associates is a registered trademark of State Street Corporation. Use of report This document and information herein (together, the “Content”) is for informational, illustrative and/or marketing purposes only and it does not constitute investment research or investment, legal, or tax advice. The Content provided is not, nor should be construed as, any offer or solicitation to buy or sell any product, service, or securities or any financial instrument, and it does not constitute any binding contractual arrangement or commitment for State Street Corporation and its subsidiaries and affiliates (“State Street”) of any kind. The Content provided does not purport to be comprehensive nor intended to replace the exercise of a client’s own careful independent review regarding any corresponding investment or other financial decision. Distribution The Content provided is not intended for retail clients, nor is intended to be relied upon by any person or entity, and is not intended for distribution to or use by any person or entity in any jurisdiction where such distribution or use would be contrary to applicable law or regulation. No permission is granted to reprint, sell, copy, distribute, or modify the Content in any form or by any means without the prior written consent of State Street. Other Important Disclosures The Content provided has been prepared and obtained from sources believed to be reliable at the time of preparation, however it is provided “as-is” and State Street makes no guarantee, representation, or warranty of any kind including, without limitation, as to its accuracy, suitability, timeliness, merchantability, fitness for a particular purpose, non-infringement of third-party rights, or otherwise. Views and opinions expressed herein are those of the author(s) and are subject to change without notice based on market and other conditions and in any event may not reflect the views of State Street. State Street disclaims all liability, whether arising in contract, tort or otherwise, for any claims, losses, liabilities, damages (including direct, indirect, special or consequential), expenses or costs arising from or connected with the Content. The Content provided may contain certain statements that could be deemed forward-looking statements; any such statements or forecasted information are not guarantees or reliable indicators for future performance and actual results or developments may differ materially from those depicted or projected. Past performance is no guarantee of future results. Prospects, clients or counterparties should be aware of the risks of participating in trading foreign exchange, equities, fixed income or derivative instruments or in investments in non-liquid or emerging markets. Prospects, clients or counterparties should be aware that products and services outlined may put their principal and capital at risk and that diversification does not ensure a profit or guarantee against loss. Australia: This communication is made available in Australia by State Street Bank and Trust Company ABN 70 062 819 630, AFSL 239679 and is intended only for wholesale clients, as defined in the Corporations Act 2001. 53
  • 54. GXN-1534 Brazil: The products in this marketing material have not been and will not be registered with the Comissão de Valores Mobiliários, the Brazilian Securities and Exchange Commission ("CVM"), and any offer of such products is not directed to the general public within the Federative Republic of Brazil ("Brazil"). The information contained in this marketing material is not provided for the purpose of soliciting investments from investors residing in Brazil and no information in this marketing material should be construed as a public offering or unauthorised distribution of the products within Brazil, pursuant to applicable Brazilian law and regulations. To our knowledge, no analyst is related to an individual who works at the issuer of the securities subject to the content of this website; no analyst or their spouse holds, directly or indirectly, securities subject to the reports on this website; no analyst or their spouse is, directly or indirectly, involved in the acquisition, sale or intermediation of securities subject to the reports on this website; no analyst or their spouse has, directly or indirectly, a financial interest in relation to the subject matter of the reports on this website; and the relevant analyst's compensation is not, directly or indirectly, influenced by the revenues arising from the business and financial transactions carried out by the entity to which is associated or otherwise related. Canada: The products and services outlined in this document are generally offered in Canada by State Street Bank and Trust Company and/or by State Street Global Markets Canada Inc. Hong Kong: This communication is made available in Hong Kong by State Street Bank and Trust Company, which accepts responsibility for its contents, and is intended for distribution to professional investors only (as defined in the Securities and Futures Ordinance). Indonesia: This communication is made available in Indonesia by State Street Bank and Trust Company and its affiliates. Neither this communication nor any copy hereof may be distributed in Indonesia or to any Indonesian citizens wherever they are domiciled or to Indonesian residents except in compliance with applicable Indonesian capital market laws and regulations. This communication is not an offer of securities in Indonesia. Any securities referred to in this communication have not been registered with the Capital Market and Financial Institutions Supervisory Agency (BAPEPAM-LK) pursuant to relevant capital market laws and regulations, and may not be offered or sold within the territory of the Republic of Indonesia or to Indonesian citizens through a public offering or in circumstances which constitute an offer within the meaning of the Indonesian capital market law and regulations. Israel: This communication is made available in Israel by State Street Global Markets International Limited, which is not licensed under Israel’s Regulation of Investment Advice, Investment Marketing and Portfolio Management Law, 1995. This communication may only be distributed to or used by investors in Israel which are “eligible clients” as listed in the First Schedule to Israel’s Regulation of Investment Advice, Investment Marketing and Portfolio Management Law 1995. Japan: This communication is made available in Japan by State Street Global Markets (Japan) which is regulated by the Financial Services Agency of Japan as a financial instruments firm. Malaysia: This communication is made available in Malaysia by State Street Global Markets International Limited (“SSGMIL”) which is authorised and regulated by the United Kingdom’s Financial Conduct Authority. SSGMIL is not licensed within or doing business within Malaysia and the activities that are being discussed are carried out off-shore. The written materials do not constitute, and should not be construed as constituting: 1) an offer or invitation to subscribe for or purchase securities or futures in Malaysia or the making available of securities or futures for purchase or subscription in Malaysia; 2) the provision of investment advice concerning securities or futures; or 3) an undertaking by SSGMIL to manage the portfolio of securities or futures contracts on behalf of other persons. Mexico: This communication is distributed by State Street Bank and Trust Company and its affiliates from outside Mexico. State Street is not authorized to act as an investment advisor in Mexico or as a regulated entity under Mexican law. Any products and services outlined herein will be provided from outside Mexico exclusively. New Zealand: This communication is made available in New Zealand by State Street Bank and Trust Company, which accepts responsibility for its contents, and is intended for distribution only to “wholesale clients” as defined in the Financial Advisers Act 2008 (New Zealand). This communication is not intended to constitute “financial advice” (as that term is defined in the Financial Advisers Act), or “advice or assistance” (in terms of section 37(5) of the Securities Markets Act 1988 (New Zealand)), to any person. 54
  • 55. GXN-1534 Oman: This communication is made available in Oman by State Street Bank and Trust Company and its affiliates. The information contained in this communication is for information purposes and does not constitute an offer for the sale of foreign securities in Oman or an invitation to an offer for the sale of foreign securities. State Street is neither a bank or financial services provider registered to undertake business in Oman and is neither regulated by the Central Bank of Oman nor the Capital Market Authority. This document is confidential and is intended solely for the information of the person to whom it has been delivered. No representation or warranty is given as to the achievement or reasonableness of any research material contained in this communication. Nothing contained in this communication report is intended to constitute Omani investment, legal, tax, accounting, investment or other professional advice. Qatar: This communication is made available in Qatar by State Street Bank and Trust Company and its affiliates. The information in this communication has not been reviewed or approved by the Qatar Central Bank, the Qatar Financial Markets Authority or the Qatar Financial Centre Regulatory Authority, or any other relevant Qatari regulatory body. Singapore: This communication is made available in Singapore by State Street Bank and Trust Company, Singapore Branch (“SSBTS”), which holds a wholesale bank license by the Monetary Authority of Singapore. In Singapore, this communication is only distributed to accredited, institutional investors as defined in the Singapore Financial Advisers Act (“FAA”). Note that SSBTS is exempt from Sections 27 and 36 of the FAA. When this communication is distributed to overseas investors as defined in the FAA, note that SSBTS is exempt from Sections 26, 27, 29 and 36 of the FAA. State Street Bank and Trust Company Limited, Singapore Branch’s Unique Entity Number is T01FC6134G. South Africa: The products and services outlined in this document are made available in South Africa through either State Street Global Markets International Limited or State Street Bank and Trust Company, both of which are authorized in South Africa under the Financial Advisory and Intermediary Services Act, 2002 as a Category I Financial Services Provider; FSP No. 42823 and 42671 respectively. South Korea: This communication is made available in South Korea by State Street Bank and Trust Company and its affiliates, which accept responsibility for its contents, and is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within South Korea, and any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-South Korea securities. Taiwan: This communication is made available in Taiwan by State Street Bank and Trust Company and its affiliates, which accept responsibility for its contents, and is intended for distribution to professional investors only. State Street Bank and Trust Company is not licensed to undertake securities business within Taiwan, and any activities related to the content hereof will be carried out off-shore and only in relation to off-shore non-Taiwan securities. Turkey: This communication is made available in Turkey by State Street Bank and Trust Company and its affiliates. The information included herein is not investment advice. Investment advisory services are provided by portfolio management companies, brokers and banks without deposit collection licenses within the scope of the investment advisory agreements to be executed with clients. Any opinions and statements included herein are based on the personal opinions of the commentators and authors. These opinions may not be suitable to your financial status and your risk and return preferences. Therefore, an investment decision based solely on the information herein may not be appropriate to your expectations. United Arab Emirates: This communication is made available in United Arab Emirates by State Street Bank and Trust Company and its affiliates. This communication does not, and is not intended to, constitute an offer of securities anywhere in the United Arab Emirates and accordingly should not be construed as such. Nor does the addressing of this research publication to you constitute, or is intended to constitute, the carrying on or engagement in banking, financial and/or investment consultation business in the United Arab Emirates under the rules and regulations made by the Central Bank of the United Arab Emirates, the Emirates Securities and Commodities Authority or the United Arab Emirates Ministry of Economy. Any public offer of securities in the United Arab Emirates, if made, will be made pursuant to one or more separate documents and only in accordance with the applicable laws and regulations. Nothing contained in this communication is intended to endorse or recommend a particular course of action or to constitute investment, legal, tax, accounting or other professional advice. Prospective investors should consult with an appropriate professional for specific advice rendered on the basis of their situation. Further, the information contained within this communication is not intended to lead to the conclusion of any contract of whatsoever nature within the territory of the United Arab Emirates. This communication has been forwarded to you solely for your information, and may not be reproduced or passed on, directly or indirectly, to any other person or published, in whole or in part, for any purpose. This communication is addressed only to persons who are professional, institutional or otherwise sophisticated investors. 55
  • 56. GXN-1534 United Kingdom and European Union: The products and services outlined herein are only offered to professional clients or eligible counterparties through State Street Bank and Trust Company, London Branch, authorised and regulated by Federal Reserve Board, authorised and subject to limited regulation by the Prudential Regulation Authority and subject to regulation by the Financial Conduct Authority and/or State Street Global Markets International Limited, authorised and regulated by the Financial Conduct Authority Details about the extent of our regulation by the Financial Conduct Authority and Prudential Regulation Authority are available from us on request. Please note that certain foreign exchange business (spot and certain forward transactions) are not regulated by the Financial Conduct Authority. United States and Latin America: The products and services outlined in this document are generally offered in the United States and in Latin America by State Street Bank and Trust Company and/or by State Street Global Markets, LLC. Please contact your sales representative for further information. © 2016, State Street Corporation, All rights reserved. 56