https://quspeakerseries9.splashthat.com/ Lecture 2: Ben Steiner In this talk we will discuss the challenges of Deep learning and focus on the key aspects of Model Risk Management for Deep Learning and Alpha Strategies

1. QU Summer School - Ben Steiner - Sept 2020
Ben Steiner
BNP Paribas Asset Management
Adjunct Lecturer, Columbia University
The views expressed in this presentation are those of the speaker and not his current employer
Model Risk Management for
Deep Learning and
Investment Strategies

2. QU Summer School - Ben Steiner - Sept 2020
Agenda
1. Introductory concepts
a. What is model risk management?
b. Machine Learning for investment strategies
c. Introduction to Deep Learning
2. Challenges of Deep Learning
a. Non-stationarity
b. Interpretation
c. Learning what we already know
3. Model Risk Management for Deep Learning investment strategies
a. Backtest evaluation
b. Ongoing monitoring
MRM FOR DL
CHALLENGES
INTRODUCTION
3

3. QU Summer School - Ben Steiner - Sept 2020
Agenda
1. Introductory concepts
a. What is model risk management?
b. Machine Learning for investment strategies
c. Introduction to Deep Learning
2. Challenges of Deep Learning
a. Non-stationarity
b. Interpretation
c. Learning what we already know
3. Model Risk Management for Deep Learning investment strategies
a. Backtest evaluation
b. Ongoing monitoring
MRM FOR DL
CHALLENGES
INTRODUCTION
4

4. QU Summer School - Ben Steiner - Sept 2020
Definitions
A Model is a simplification of the real world into mathematical equations to
forecast some future behavior.
Model Risk comes from either incorrect models (fundamental errors) or models
being misapplied (incorrect or inappropriate usage).
Risk Management is the process of identifying, analysing and controlling
uncertainty around objectives.
Model Risk Management is the understanding, analysing and controlling the risk
inherent in using models.
1. Conceptual Soundness
2. Implementation Validation
3. Ongoing Monitoring
MRM FOR DL
CHALLENGES
INTRODUCTION
5

5. QU Summer School - Ben Steiner - Sept 2020
MRM FOR DL
CHALLENGES
INTRODUCTION
6

6. QU Summer School - Ben Steiner - Sept 2020
Definitions
Machine Learning uses algorithms to learn from data without relying on rules-
based programming
Deep Learning maps inputs to outputs using multiple layers of nonlinear
processing units
MRM FOR DL
CHALLENGES
INTRODUCTION
7

7. QU Summer School - Ben Steiner - Sept 2020
Image credit: Shane Conway, Kepos Capital, SQA Fuzzy Day conference 2017
MRM FOR DL
CHALLENGES
INTRODUCTION
8

8. QU Summer School - Ben Steiner - Sept 2020
Is Machine Learning still “a model”? MRM FOR DL
CHALLENGES
INTRODUCTION
9

9. QU Summer School - Ben Steiner - Sept 2020
Input layer: features (or attributes)
Hidden layers: Bias and weights
Output layer: target variables (or responses)
Deep Learning basics
Input layer
Hidden layer 1
Output layer
Hidden layer 2
Married
Single
Age
Income
Employment
Example: Arno Candel, H20.ai
MRM FOR DL
CHALLENGES
INTRODUCTION
10

10. QU Summer School - Ben Steiner - Sept 2020
Neurons activate each other via weighted sums
y1 = f(( x1u1 + x2u2 + x3u3 ) + b1 )
Non-linear activation function tanh: or rectilinear:
Deep Learning basics
Input layer
Hidden layer 1
Hidden layer 2
Output layer
x1
x2
x3
y1u1
u2
u3
MRM FOR DL
CHALLENGES
INTRODUCTION
11
-1
1
0

11. QU Summer School - Ben Steiner - Sept 2020
Neurons activate each other via weighted sums
z3 = f(( y1v1 + y2v2 + y3v3 + y4v4 ) + c3 )
Non-linear activation function tanh: or rectilinear:
Deep Learning basics
Input layer
Hidden layer 1
Hidden layer 2
Output layer
y1
v1
v2
v3
z3
y2
y3
y4
v4
MRM FOR DL
CHALLENGES
INTRODUCTION
12
-1
1
0

12. QU Summer School - Ben Steiner - Sept 2020
Deep Learning basics
Neurons activate each other via weighted sums
p2 = f(( z1w1 + z2w2 + z3w3 ) + d2 )
Non-linear activation function: softmax
Input layer
Hidden layer 1
Hidden layer 2
Output layer
w1
w2 p2
w3
z3
z2
z1
p(up)
p(down)
p1
MRM FOR DL
CHALLENGES
INTRODUCTION
13

13. QU Summer School - Ben Steiner - Sept 2020
Agenda
1. Introductory concepts
a. What is model risk management?
b. Investment strategy: problem statement & objective
c. Introduction to Deep Learning
2. Challenges of Deep Learning
a. Non-stationarity
b. Interpretation
c. Learning what we already know
3. Model Risk Management for Deep Learning investment strategies
a. Backtest evaluation
b. Ongoing monitoring
MRM FOR DL
CHALLENGES
INTRODUCTION
14

14. QU Summer School - Ben Steiner - Sept 2020
“A model may be reasonable, but the
world itself may be unstable. What’s a
good model today may be inappropriate
tomorrow”
Emanuel Derman, 1996, GS research paper on model risk
MRM FOR DL
CHALLENGES
INTRODUCTION
15

15. QU Summer School - Ben Steiner - Sept 2020
Challenge 1: Non-Stationarity
Stationarity (or Nonstationarity) is an
assumption of a data generating model and not
a property of observed data.
A single realization from a stationary stochastic
process can appear indistinguishable from a
nonstationary deterministic process
‘Change’ is in the timeframe of the beholder
16

16. QU Summer School - Ben Steiner - Sept 2020
Challenge 1: Non-Stationarity
One challenge, many names:
● Concept Drift
● Covariate Shift
● Dataset shift
CONCEPT DRIFT
When the statistical properties of the target
variable, which the model is trying to predict,
change over time in unforeseen ways.
The unforeseen substitution of one data source
𝐒1 (with underlying probability distribution 𝚷S1),
with another source 𝐒2 (with distribution 𝚷S2)
MRM FOR DL
CHALLENGES
INTRODUCTION
17
● Nonstationarity
● Source component shift
● Temporal evolution
Not Concept Drift

17. QU Summer School - Ben Steiner - Sept 2020
Different Types of Concept Drift
S2
S1
S2
S1
S2
S1
S2
S1
S2
S1
S2
S1
MRM FOR DL
CHALLENGES
INTRODUCTION
18Images derived from Dariusz Brzeziński, Mining Data Streams with Concept Drift, 2010

18. QU Summer School - Ben Steiner - Sept 2020
MRM FOR DL
CHALLENGES
INTRODUCTION
19

19. QU Summer School - Ben Steiner - Sept 2020
Challenge 2: Interpretation
Why do we need interpretation?
What do we mean by interpretability?
Zachary Lipton,UCSD, 2016 ICML Workshop on Human Interpretability in Machine Learning
● Causality
● Comprehension
● Decomposition
● Algorithmic transparency
● Post-hoc interpretation
MRM FOR DL
CHALLENGES
INTRODUCTION
20

20. QU Summer School - Ben Steiner - Sept 2020
Challenge 2: Look at the weights!
4 input features: target classification::
Network with 2 hidden layers of 3 & 2 neurons
Weight = thickness
http://playground.tensorflow.org
MRM FOR DL
CHALLENGES
INTRODUCTION
22

21. QU Summer School - Ben Steiner - Sept 2020
Good news: can’t be both!
Solution:
Step 1: Traditional multifactor model (with known factors)
Step 2: Deep Learning (with residuals from step 1)
Learning, but
nothing new
Interpretation
challenge
Challenge 3: Learning what we already know MRM FOR DL
CHALLENGES
INTRODUCTION
23

22. QU Summer School - Ben Steiner - Sept 2020
Challenge 3: Multifactor models MRM FOR DL
CHALLENGES
INTRODUCTION
24

23. QU Summer School - Ben Steiner - Sept 2020
Agenda
1. Introductory concepts
a. What is model risk management?
b. Machine Learning for investment strategies
c. Introduction to Deep Learning
2. Challenges of Deep Learning
a. Non-stationarity
b. Interpretation
c. Learning what we already know
3. Model Risk Management for Deep Learning investment strategies
a. Backtest evaluation
b. Ongoing monitoring
MRM FOR DL
CHALLENGES
INTRODUCTION
25

24. QU Summer School - Ben Steiner - Sept 2020
“The road to
hedge fund failure
is littered with
good backtests”
MRM FOR DL
CHALLENGES
INTRODUCTION
26

25. QU Summer School - Ben Steiner - Sept 2020
Conceptual Soundness = Backtest Evaluation
● Return per unit risk per unit capital required
● Alpha Decay
● Temporal P&L
● Strategy Correlation
● Sensitivity Analysis
● Random Markets
MRM FOR DL
CHALLENGES
INTRODUCTION
27

26. QU Summer School - Ben Steiner - Sept 2020
Evaluation 1: Alpha Decay
Alpha term structure
Cost of implementation delay
Dictates execution speed
Too fast = alpha not capturable
Alpha not declining at all raises
suspicion
Declining alpha indicates profit
from trades at t=0.
✓
✗
!?
MRM FOR DL
CHALLENGES
INTRODUCTION
28

27. QU Summer School - Ben Steiner - Sept 2020 29
Evaluation 2: Temporal P&L
Three strategies with same long run risk
adjusted return but different temporal
performance
Strategy decay: cyclical or secular?
1. Secular decay: avoid
2. Cyclical decay: Trend follow
3. Improving performance: yes please!
✓!
✗
✓?

28. QU Summer School - Ben Steiner - Sept 2020
StrategydP&L
✓✓ What don’t we know?
Evaluation 3: Strategy Sensitivity
Correlation with exogenous factors (eg: macro environment)
MRM FOR DL
CHALLENGES
INTRODUCTION
30
StrategyaP&L
Macro variable
StrategycP&L
StrategybP&L
Macro variable Macro variable
✓✗ ✗ ✓ ✗ ✗✓ ✓ ✓ ✓ ✓ ✓
??

29. QU Summer School - Ben Steiner - Sept 2020
Random portfolio weights. No Deep Learning.
Evaluation 4: Random Portfolios
Long run exposure Active trades
“Impossible” Possible but unlikely
MRM FOR DL
CHALLENGES
INTRODUCTION
31

30. QU Summer School - Ben Steiner - Sept 2020
Evaluation 5: Random Returns
Randomize order of returns. Full retraining on noise…
● Break covariance between returns and features
● Break autocorrelation of returns
● Keep original features
MRM FOR DL
CHALLENGES
INTRODUCTION
32

31. QU Summer School - Ben Steiner - Sept 2020
Strategy Monitoring
“You’ve never experienced your worst drawdown”
Is the strategy performing “as intended”
As intended = per backtest
How bad is bad?
Real world consequences
Out-of-sample:
● Distribution
● Small samples (SPC)
MRM FOR DL
CHALLENGES
INTRODUCTION
33

32. QU Summer School - Ben Steiner - Sept 2020
Summary
1. Introductory concepts
a. Model Risk Management is controlling the risk a model is “wrong”
b. Machine Learning is still a model
c. Deep Learning maps features to price movements via non-linear functions
2. Challenges of Deep Learning for Investment Strategies
a. Financial markets experience Concept Drift (video games do not!)
■ What type of Concept Drift is expected?
■ How much data can be collected before the system changes again?
b. Interpretation can be addressed (hint: look at weights!)
c. Avoid learning what we already know (hint: use residuals as the target)
3. Model Risk Management for Deep Learning investment strategies
a. Backtest evaluation to turn strategies on (but its not about high Sharpe Ratios!)
b. Ongoing monitoring to evaluate when to turn a strategy off
MRM FOR DL
CHALLENGES
INTRODUCTION
34

33. QU Summer School - Ben Steiner - Sept 2020
The road to Strategy
failure is not littered
with Deep Learning
backtests
In conclusion, Model Risk Management can help
make sure ...
35

34. QU Summer School - Ben Steiner - Sept 2020
Speaker Biography
Ben Steiner - BNP Paribas Asset Management
Ben Steiner has spent his career in the application of machine learning and
model risk management for investment firms. In his current role, Ben handles
Chief-of-Staff responsibilities in the Global Fixed Income division of BNP Paribas
Asset Management. He focusses on business management and strategic
initiatives that help BNPP AM deliver long term, sustainable returns for our
clients.
Earlier in his career, Ben was a Head of Model Development, Portfolio Manager
& Quant Researcher at investment managers and quantitative hedge funds. This
covered model development and investment strategies across multiple asset
classes ranging from the traditionally illiquid (Private Debt and Real Estate) to
more liquid products (Global Macro; Managed Futures; Equity Long/Short and
Absolute Return Fixed Income).
Ben holds a BA (Hons.) in Economics from the University of Manchester and an
MSc in Mathematical Finance from Imperial College, London. Since 2013, he
has served on the Board of Directors of the Society of Quantitative Analysts
(www.sqa-us.org). More recently, Ben has presented topics in machine learning
and model risk management at various universities and industry events.
Starting in Jan 2020, he teaches quantitative courses at Columbia University.
36