CFA-NY Workshop - Final slides

AI and Machine Learning
for Financial Professionals
2019 Copyright QuantUniversity LLC.
Presented By:
Sri Krishnamurthy, CFA, CAP
sri@quantuniversity.com
www.analyticscertificate.com
08/12/2019
CFA Society of New York
New York

2
Speaker bio
• Advisory and Consultancy for Financial
Analytics
• Prior Experience at MathWorks, Citigroup
and Endeca and 25+ financial services and
energy customers.
• Columnist for the Wilmott Magazine
• Author of forthcoming book
“Financial Modeling: A case study approach”
published by Wiley
• Teaches Analytics in the Babson College MBA
program and at Northeastern University,
Boston
• Reviewer: Journal of Asset Management
Sri Krishnamurthy
Founder and CEO
QuantUniversity

3
About www.QuantUniversity.com
• Boston-based Data Science, Quant
Finance and Machine Learning
training and consulting advisory
• Trained more than 1000 students in
Quantitative methods, Data Science
and Big Data Technologies using
MATLAB, Python and R
• Building a platform for AI
and Machine Learning Enablement
in the Enterprise

AM
• Key trends in AI and machine learning
• Machine Learning in 1 hour
• Case study 1: Lending Club – Prediction
• 5 things you need to know about machine learning
PM
• Case studies
▫ Case study 2: Stock Data - Clustering
▫ Case study 3: Freddie Mac – Classification
▫ Case study 4: Sentiment analysis
▫ Recap: Building a ML application in 10 steps
Agenda

5
www.tinyurl.com/QuSandbox3
Important:
Use Registration Code
CFA2019NY
Slides and Code

AI and Machine Learning in Finance

7
My journey into AI/ML in finance 5 pictures

8
The 4th Industrial revolution is Here!
Source: Christoph Roser at AllAboutLean.com
As per Wikipedia*, “The 4th Industrial Revolution ….. marked by emerging technology breakthroughs in a
number of fields, including robotics, artificial intelligence, nanotechnology, quantum computing, biotechnology,
the Internet of Things, the Industrial Internet of Things (IIoT), decentralized consensus, fifth-generation wireless
technologies (5G), additive manufacturing/3D printing and fully autonomous vehicles.”
* https://en.wikipedia.org/wiki/Fourth_Industrial_Revolution

9
Your challenge is to design an artificial intelligence and machine learning (AI/ML)
framework capable of flying a drone through several professional drone racing
courses without human intervention or navigational pre-programming.
AI is no longer science fiction!
Source: https://www.lockheedmartin.com/en-us/news/events/ai-innovation-challenge.html

10
Scientists are disrupting the way we live!
Source: https://www.ladn.eu/tech-a-suivre/mobilite-2030-vehicules-volants-open-data/

11
Interest in Machine learning continues to grow
https://www.wipo.int/edocs/pubdocs/en/wipo_pub_1055.pdf

12
Source: https://www.bbc.com/news/technology-35785875

13
MACHINE LEARNING AND AI IS REVOLUTIONIZING FINANCE

14
Market impact at the speed of light!
14

15
Machine Learning & AI in finance: A paradigm shift
15
Stochastic
Models
Factor Models
Optimization
Risk Factors
P/Q Quants
Derivative pricing
Trading Strategies
Simulations
Distribution
fitting
Quant
Real-time analytics
Predictive analytics
Machine Learning
RPA
NLP
Deep Learning
Computer Vision
Graph Analytics
Chatbots
Sentiment Analysis
Alternative Data
Data Scientist

16
CFA Institute has adopted Fintech and AI content in its curriculum
Ref: https://www.cfainstitute.org/-/media/documents/support/programs/cfa/cfa-program-level-iii-fintech-in-investment-management.ashx

17
The Virtuous Circle of
Machine Learning and AI
17
Smart
Algorithms
Hardware
Data

18
The rise of Big Data and Data Science
18
Image Source: http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4-Vs-of-big-data.jpg

19
Smart Algorithms
19
Distributing Computing Frameworks Deep Learning Frameworks
1. Our labeled datasets were thousands of times too
small.
2. Our computers were millions of times too slow.
3. We initialized the weights in a stupid way.
4. We used the wrong type of non-linearity.
- Geoff Hinton
“Capital One was able to determine fraudulent credit
card applications in 100 milliseconds”*
* http://go.databricks.com/hubfs/pdfs/Databricks-for-FinTech-170306.pdf

20
Hardware
Speed up calculations with
1000s of processors
Scale computations with
infinite compute power

• Bank of America
• Ravenpack
• Northfield
Examples on how AI and ML are used in Finance
#Disrupt19

Use Cases in NLP
Risk Management
Power risk models by
informing clients about
their portfolio exposures
to headline risk and
public disclosures.
Compliance
Reduce costs in trade
surveillance and
compliance by
reducing the number
of false-positives
chased by analysts
and officers.
Benchmarks
Create innovative
investable indexes
powered by AI and
Big Data.
Alpha Generation
Create trading signals
by ingesting event and
sentiment data; identify
securities that are likely
to suffer from short
squeezes or reversals.

Risk Systems That Read®
• Northfield uses machine learning based analysis of news text
to describe how current conditions in financial markets are
different than usual.
• Typically, over 8000 articles per day containing more than
20,000 “topics” (companies, industries, countries) are
processed.
• The nature and magnitudes of these difference are used to
revise expectations of financial market risks for all global
equities and credit instruments on a daily basis.

25
• Sentiment Analysis App:
▫ http://ec2-34-220-235-127.us-west-2.compute.amazonaws.com/
• Credit Risk App:
▫ http://ec2-54-202-242-75.us-west-2.compute.amazonaws.com/
Sample Apps

27
• Machine learning is the scientific study of algorithms and statistical
models that computer systems use to effectively perform a specific task
without using explicit instructions, relying on patterns and inference
instead1
• Artificial intelligence is intelligence demonstrated by machines, in
contrast to the natural intelligence displayed by humans and animals1
Definitions: Machine Learning and AI
27
1. https://en.wikipedia.org/wiki/Machine_learning
2. Figure Source: http://www.fsb.org/wp-content/uploads/P011117.pdf

28
1. Data
2. Goals
3. Machine learning algorithms
4. Process
5. Performance evaluation
Key steps involved

30
Dataset, variable and Observations
Dataset: A rectangular array with Rows as observations and
columns as variables
Variable: A characteristic of members of a population ( Age, State
etc.)
Observation: List of Variable values for a member of the
population

31
Variables
A variable could be:
▫ Categorical
– Yes/No flags
– AAA,BB ratings for bonds
▫ Numerical
– 35 mpg
– $170K salary

32
Longitudinal
▫ Observations are dependent
▫ Temporal-continuity is required
Cross-sectional
▫ Observations are independent
Datasets

33
Data
Cross
sectional
Numerical Categorical
Longitudinal
Numerical
Summary
33

35
• Descriptive Statistics
▫ Goal is to describe the data at hand
▫ Backward-looking
▫ Statistical techniques employed here
• Predictive Analytics
▫ Goal is to use historical data to build a model for prediction
▫ Forward-looking
▫ Machine learning & AI techniques employed here
Goal
35

36
• How do you summarize numerical variables ?
• How do you summarize categorical variables ?
• How do you describe variability in numerical variables ?
• How do you summarize relationships between categorical and
numerical variables ?
• How do you summarize relationships between 2 numerical
variables?
Descriptive Statistics – Cross sectional datasets
36

37
Goal is to extract the various components
Longitudinal datasets
37

38
• Given a dataset, build a model that captures the
similarities in different observations and assigns
them to different buckets.
• Given a set of variables, predict the value of
another variable in a given data set
▫ Predict salaries given work experience, education etc.
▫ Predict whether a loan would be approved given fico
score, current loans, employment status etc.
Predictive Analytics : Cross sectional datasets
38

39
• Given a time series dataset, build a model that can be used to
forecast values in the future
Predictive Analytics : Time series datasets
39

40
Goal
Descriptive
Statistics
Cross
sectional
Numerical vs
Categorical
Categorical vs
Categorical
Numerical vs
Numerical
Time series
Predictive
Analytics
Cross-
sectional
Segmentation Prediction
Predict a
number
Predict a
category
Time-series
Summary
40

42
Machine Learning
Unsupervised Supervised
Reinforcement Semi-Supervised
Machine Learning

43
Goal
Descriptive
Statistics
Cross
sectional
Numerical vs
Categorical
Categorical vs
Categorical
Numerical vs
Numerical
Time series
Predictive
Analytics
Cross-
sectional
Predict a
number
Predict a
category
Time-series
Machine Learning Algorithms
43

44
Supervised Algorithms
▫ Given a set of variables 𝑥", predict the value of another variable 𝑦 in
a given data set such that
▫ If y is numeric => Prediction
▫ If y is categorical => Classification
▫ Example: Given that a customer’s Debt-to-Income ratio increased 20%, what are
the chances he/she would default in 3 months?
Machine Learning
44
x1,x2,x3… Model F(X) y

45
Unsupervised Algorithms
▫ Given a dataset with variables 𝑥", build a model that captures the
similarities in different observations and assigns them to different
buckets => Clustering
▫ Example: Given a list of emerging market stocks, can we segment them
into three buckets?
Machine Learning
45
Obs1,
Obs2,Obs3
etc.
Model
Obs1- Class 1
Obs2- Class 2
Obs3- Class 1

46
Supervised
Learning
algorithms
Parametric
models
Non-
Parametric
models
Supervised learning Algorithms - Prediction
46

47
• Parametric models
▫ Assume some functional form
▫ Fit coefficients
• Examples : Linear Regression, Neural Networks
Supervised Learning models - Prediction
47
𝑌 = 𝛽' + 𝛽) 𝑋)
Linear Regression Model Neural network Model

48
• Non-Parametric models
▫ No functional form assumed
• Examples : K-nearest neighbors, Decision Trees
Supervised Learning models
48
K-nearest neighbor Model Decision tree Model

49
• Given estimates +𝛽', +𝛽), … , +𝛽.We can make predictions using
the formula
/𝑦 = +𝛽' + +𝛽) 𝑥) + +𝛽0 𝑥0 + ⋯ + +𝛽. 𝑥.
• The parameters are estimated using the least squares approach
to minimize the sum of squared errors
𝑅𝑆𝑆 = 4
"5)
6
(𝑦" − /𝑦")0
Multiple linear regression
49

50
• Parametric models
▫ Assume some functional form
▫ Fit coefficients
• Examples : Logistic Regression, Neural Networks
Supervised Learning models - Classification
50
Logistic Regression Model Neural network Model

51
• Non-Parametric models
▫ No functional form assumed
• Examples : K-nearest Neighbors, Decision Trees
Supervised Learning models
51
K-nearest neighbor Model Decision tree Model

52
Unsupervised Algorithms
▫ Given a dataset with variables 𝑥", build a model that captures the
similarities in different observations and assigns them to different
buckets => Clustering
Machine Learning
52
Obs1,
Obs2,Obs3
etc.
Model
Obs1- Class 1
Obs2- Class 2
Obs3- Class 1

53
• These methods partition the data into k clusters by assigning each data point to its
closest cluster centroid by minimizing the within-cluster sum of squares (WSS), which
is:
4
:5)
;
4
"∈=>
4
?5)
@
(𝑥"? − 𝜇:?)0
where 𝑆: is the set of observations in the kth cluster and 𝜇:? is the mean of jth
variable of the cluster center of the kth cluster.
• Then, they select the top n points that are the farthest away from their nearest
cluster centers as outliers.
K-means clustering
53

54
Euclidean distance:
Distance functions

55
Correlation distance:
Distance functions

56
Machine
Learning
Supervised
Prediction
Parametric
Linear
Regression
Neural
Networks
Non-
parametric
KNN
Decision
Trees
Classification
Parametric
Logistic
Regression
Neural
Networks
Non
Parametric
Decision
Trees KNN
Unsupervised
algorithms
K-means
Associative
rule mining
56

57
Anomaly Detection vs Unsupervised Learning
57

58
Machine Learning movers and shakers
Deep
Learning
Automatic
Machine
Learning
Ensemble
Learning
Natural
Language
Processing

59
http://www.asimovinstitute.org/neural-network-zoo/

61
The Process
61
Data
ingestion
Data
cleansing
Feature
engineering
Training
and testing
Model
building
Model
selection

62
• What transformations do I need for the x and y variables ?
• Which are the best features to use?
▫ Dimension Reduction – PCA
▫ Best subset selection
– Forward selection
– Backward elimination
– Stepwise regression
Feature Engineering
62

63
Data
Training
80%
Testing
20%
Training the model
63

65
Evaluating
Machine learning
algorithms
Supervised -
Prediction
R-square RMS MAE MAPE
Supervised-
Classification
Confusion Matrix ROC Curves
Evaluation framework
65

66
• The prediction error for record i is defined as the difference
between its actual y value and its predicted y value
𝑒" = 𝑦" − /𝑦"
• 𝑅0
indicates how well data fits the statistical model
𝑅0
= 1 −
∑"5)
6
(𝑦" − /𝑦")0
∑"5)
6
(𝑦" − E𝑦")0
Prediction Accuracy Measures

67
• Fit measures in classical regression modeling:
• Adjusted 𝑅0 has been adjusted for the number of predictors. It increases
only when the improve of model is more than one would expect to see by
chance (p is the total number of explanatory variables)
𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑 𝑅0 = 1 −
⁄∑"5)
6
(𝑦" − /𝑦")0 (𝑛 − 𝑝 − 1)
∑"5)
6
𝑦" − E𝑦"
0 /(𝑛 − 1)
• MAE or MAD (mean absolute error/deviation) gives the magnitude of the
average absolute error
𝑀𝐴𝐸 =
∑"5)
6
𝑒"
𝑛

68
▫ MAPE (mean absolute percentage error) gives a percentage score of
how predictions deviate on average
𝑀𝐴𝑃𝐸 =
∑"5)
6
𝑒"/𝑦"
𝑛
×100%
• RMSE (root-mean-squared error) is computed on the training and
validation data
𝑅𝑀𝑆𝐸 = 1/𝑛 4
"5)
6
𝑒"
0

69
• Consider a two-class case with classes 𝐶' and 𝐶)
• Classification matrix:
Classification matrix
Predicted Class
Actual Class 𝐶' 𝐶)
𝐶'
𝑛','= number of 𝐶' cases
classified correctly
𝑛',)= number of 𝐶' cases
classified incorrectly as 𝐶)
𝐶)
𝑛),'= number of 𝐶) cases
classified incorrectly as 𝐶'
𝑛),)= number of 𝐶) cases
classified correctly

70
• Estimated misclassification rate (overall error rate) is a main
accuracy measure
𝑒𝑟𝑟 =
𝑛',) + 𝑛),'
𝑛',' + 𝑛',) + 𝑛),' + 𝑛),)
=
𝑛',) + 𝑛),'
𝑛
• Overall accuracy:
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 1 − 𝑒𝑟𝑟 =
𝑛',' + 𝑛),)
𝑛
Accuracy Measures

71
• The ROC curve plots the pairs {sensitivity, 1-
specificity} as the cutoff value increases from 0
and 1
• Sensitivity (also called the true positive rate, or
recall in some fields) measures the proportion of
positives that are correctly identified (e.g., the
percentage of sick people who are correctly
identified as having the condition).
• Specificity (also called the true negative rate)
measures the proportion of negatives that are
correctly identified as such (e.g., the percentage of
healthy people who are correctly identified as not
having the condition).
• Better performance is reflected by curves that are
closer to the top left corner
ROC Curve

72
1. Data
2. Goals
3. Machine learning algorithms
4. Process
5. Performance Evaluation
Recap

73
Data
Cross
sectional
Longitudinal
Numerical
Handling Data
73

74
Goal
Descriptive
Statistics
Cross
sectional
Numerical vs
Categorical
Categorical vs
Categorical
Numerical vs
Numerical
Time series
Predictive
Analytics
Cross-
sectional
Predict a
number
Predict a
category
Time-series
Goal
74

75
Machine
Learning
Supervised
Prediction
Parametric
Linear
Regression
Neural
Networks
Non-
parametric
KNN
Decision
Trees
Classification
Parametric
Logistic
Regression
Neural
Networks
Non
Parametric
Decision
Trees KNN
Unsupervised
algorithms
K-means
Associative
rule mining
75

76
The Process
76
Data
ingestion
Data
cleansing
Feature
engineering
Training
and testing
Model
building
Model
selection

77
Evaluating
Machine learning
algorithms
Supervised -
Prediction
R-square RMS MAE MAPE
Supervised-
Classification
Confusion Matrix ROC Curves
Evaluation framework
77

Machine Learning Workflow
Data Scraping/
Ingestion
Data
Exploration
Data Cleansing
and Processing
Feature
Engineering
Model
Evaluation
& Tuning
Model
Selection
Model
Deployment/
Inference
Supervised
Unsupervised
Modeling
Data Engineer, Dev Ops Engineer
Data Scientist/QuantsSoftware/Web Engineer
• AutoML
• Model Validation
• Interpretability
Robotic Process Automation (RPA) (Microservices, Pipelines )
• SW: Web/ Rest API
• HW: GPU, Cloud
• Monitoring
• Regression
• KNN
• Decision Trees
• Naive Bayes
• Neural Networks
• Ensembles
• Clustering
• PCA
• Autoencoder
• RMS
• MAPS
• MAE
• Confusion Matrix
• Precision/Recall
• ROC
• Hyper-parameter
tuning
• Parameter Grids
Risk Management/ Compliance(All stages)
Analysts&
DecisionMakers

#Disrupt19
Credit Risk Decision Making Using Lending Club Data

80
1. Case Intro
2. Data Exploration of the Credit risk data set
3. Problem Definition and Machine learning
4. Performance Evaluation
5. Deployment
Case study 1

81
Credit risk in consumer credit
Credit-scoring models and techniques assess the risk in
lending to customers.
Typical decisions:
• Grant credit/not to new applicants
• Increasing/Decreasing spending limits
• Increasing/Decreasing lending rates
• What new products can be given to existing applicants ?

82
Credit assessment in consumer credit
History:
• Gut feel
• Social network
• Communities and influence
Traditional:
• Scoring mechanisms through credit bureaus
• Bank assessments through business rules
Newer approaches:
• Peer-to-Peer lending
• Prosper Market place

83
The Data
83
https://www.kaggle.com/wendykan/lending-club-loan-data

84
Credit Risk pipeline
Data Ingestion
from Lending
Club
Pre-Processing
Feature
Engineering
Model
Development
and Tuning
Model
Deployment
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

86
1. Whitepapers at www.quantuniversity.com
2. https://blogs.cfainstitute.org/investor/tag/machine-learning/
3. https://techcrunch.com/
4. https://www.technologyreview.com/
5. https://www.bbc.com/timelines/zypd97h
6. https://www.bbc.com/timelines/zq376fr
Additional Reading

88
Claim:
• Machine learning is good for credit-card fraud detection
Caution:
• Beware of imbalanced class problems
• A model that gives 99% accuracy may still not be good enough
1.Machine learning is not a generic solution to all problems
88

89
Claim:
• Our models work on all the datasets we have tested on
Caution:
• Do we have enough data?
• How do we handle bias in datasets?
• Beware of overfitting
• Historical Analysis is not Prediction
2. A prototype model is not A production model
89

90
Prototyping vs Production: The reality
https://www.itnews.com.au/news/hsbc-societe-generale-run-
into-ais-production-problems-477966
Kristy Roth from HSBC:
“It’s been somewhat easy - in a funny way - to
get going using sample data, [but] then you hit
the real problems,” Roth said.
“I think our early track record on PoCs or pilots
hides a little bit the underlying issues.
Matt Davey from Societe Generale:
“We’ve done quite a bit of work with RPA
recently and I have to say we’ve been a bit
disillusioned with that experience,”
“the PoC is the easy bit: it’s how you get that
into production and shift the balance”

91
Claim:
• It works. We don’t know how!
Caution:
• Lots of heuristics; still not a proven science
• Interpretability, Fairness, Auditability of models are important
• Beware of black boxes; Transparency in codebase is paramount
with the proliferation of opensource tools
• Skilled data scientists with knowledge of algorithms and their
appropriate usage are key to successful adoption
3. We are just getting started!
91

92
Claim:
• Machine Learning models are more
accurate than traditional models
Caution:
• Is accuracy the right metric?
• How do we evaluate the model? Accuracy
or F1-Score?
• How does the model behave in different
regimes?
4. Choose the right metrics for evaluation
92
Source:
https://en.wikipedia.org/wiki/Confusion_matrix

93
Claim:
• Machine Learning and AI will replace humans
in most applications
Caution:
• Just because it worked some times doesn’t
mean that the organization can be on
autopilot
• Will we have true AI or Augmented
Intelligence?
• Model risk and robust risk management is
paramount to the success of the
organization.
• We are just getting started!
5. Are we there yet?
93
https://www.bloomberg.com/news/articles/2017-10-
20/automation-starts-to-sweep-wall-street-with-tons-of-
glitches

94
Can Machine Learning algorithms be gamed?
https://www.youtube.com/watch?time_continue=36&v=MIbFv
K2S9g8
https://arxiv.org/abs/1904.08653

#Disrupt19
Does a Loan have a PMI or not? A Freddie Mac Case study

• Freddie Mac The Case study Setup
• Design Choices
• The Pipeline
• Demo
#Disrupt19
Agenda

97
• Freddie Mac was created in 1970 to expand the secondary
market for mortgages in the US. Freddie Mac buys mortgages
on the secondary market, pools them, and sells them as
a mortgage-backed security to investors on the open market.
Introduction
97
https://a16z.com/2018/05/19/mortgage-process-players-
problems-opportunities/

98
• Freddie mac data
Goal
98
http://www.freddiemac.com/research/datasets/sf_loanlevel_d
ataset.page

99
Pipeline
Data
Ingestion
Pre-
Processing
EDA
Model
Building
Performance
Evaluation

• Stock data + Setup
• Design Choices
• The Pipeline
• Demo
#Disrupt19
Agenda

102
Introduction
102
Source: https://novelinvestor.com/sector-performance/

103
• Data: https://www.amazon.com/Analytics-Edge-Dimitris-
Bertsimas/dp/098991089X
• Given stock data, can we cluster intro groups based on returns?
▫ Hierarchical clustering
▫ K-means clustering
Goal
103

#Disrupt19
Sentiment Analysis Using Natural Language Processing in Finance

• What is Sentiment Analysis?
• The Case study Setup
• Design Choices
• The Pipeline
• Demo
#Disrupt19
Agenda

106
What is NLP ?
AI
Linguistics
Computer
Science

107
• Q/A
• Dialog systems - Chatbots
• Topic summarization
• Sentiment analysis
• Classification
• Keyword extraction - Search
• Information extraction – Prices, Dates, People etc.
• Tone Analysis
• Machine Translation
• Document comparison – Similar/Dissimilar
Sample applications

109
• The process of computationally identifying and categorizing
opinions expressed in a piece of text, especially in order to
determine whether the writer's attitude towards a particular
topic, product, etc. is positive, negative, or neutral.
Sentiment Analysis
#Disrupt19

110
• Understanding sentiments in Earnings call transcripts
Goal
110

111
• Interpreting emotions
• Labeling data
Options
• APIs
• Human Insight
• Expert Knowledge
• Build your own
Challenges

112
NLP pipeline
Data Ingestion
from Edgar
Pre-Processing
Invoking APIs to
label data
Compare APIs
Build a new
model for
sentiment
Analysis
• Amazon Comprehend API
• Google API
• Watson API
• Azure API

113
The reproducibility challenge

114
What’s needed for reproducibility
Code Data
Environment Process

115
QuSandbox solution suite for ML/AI applications
Model
Analytics
Studio
QuSandbox
Research
hub

116
Prototype
Standardize
workflow
Productionize
and share
DEMO with QuSandbox
116
QuSandbox Model Analytics Studio ResearchHub

#Disrupt19
Building your Data science applications which uses AI/ML in 10 steps

119
1. Articulate your business problem
Data science in 10 steps

120
2. The Data questions
1. Do you know what data you need ?
2. Do you know if the data is available?
3. Do you have the data ?
4. Do you have the right data?
5. Will you continue to have the data?

121
3. Develop a data acquisition and data prep strategy
1. Do you know how to get the data ?
2. Who gets the data?
3. How do you process it?
4. How do you access it?
5. How do you version and govern the data?

122
4. Explore and evaluate your data and get it in the right format

123
5. Define your goal:
1. Summarization
2. Fact finding
3. Understanding relationships
4. Prediction

124
6. Shortlist (not “Choose” ) the
techniques/methodologies/algorithms

125
7. Evaluate/establish business constraints and narrow down your
choices of techniques/methodologies/algorithms
1. Cloud/Cost/Expertise/Cost-Value
2. Build/buy/access
Outcomes
Time
Quality
Cost

126
8. Establish criteria to know if the methodology/models/algorithms
work
1. Is the process replicable?
2. What performance metrics do we choose?
3. Can you evaluate the performance and validate if the models meet
the criteria?
4. Does it provide business value?

127
9. Fine tune your algorithms and algorithm selection
1. Hyper parameter tuning
2. Bias-variance tradeoff
3. Handling imbalanced class problems
4. Ensemble techniques
5. AutoML
https://support.sas.com/resources/papers/proceedings17/SAS0514-2017.pdf

128
10. How will this process reach decision makers
1. Deployment choices (On-prem/Cloud)
2. Frequency of data/model updates
3. Governance/Role/Responsibilities
4. Speed, Scale, Availability, Disaster recovery, Rollback, Pull-Plug

129
How do you monitor the efficacy of your solution?
1. Retuning
2. Monitoring
3. Model decay
4. Data augmentation
5. Newer innovations
Data science in 10 steps - Bonus

Thank you!
Sri Krishnamurthy, CFA, CAP
Founder and CEO
QuantUniversity LLC.
srikrishnamurthy
www.QuantUniversity.com
Contact
Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be
distributed or used in any other publication without the prior written consent of QuantUniversity LLC.
130

CFA-NY Workshop - Final slides

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to CFA-NY Workshop - Final slides

Similar to CFA-NY Workshop - Final slides (20)

More from QuantUniversity

More from QuantUniversity (18)

Recently uploaded

Recently uploaded (20)

CFA-NY Workshop - Final slides