AI and machine learning have the power to transform entire industries. Companies in consumer facing industries like banking, healthcare, and e-commerce that aren’t using these new tools and advances today run the real risk of failure as we gear towards an automated future. Dr. Sourav Dey, Head of Machine Learning at Manifold, will talk about the importance for today’s enterprises to adopt ML and data science and how they can do it. Using specific real world use cases, Sourav will detail his 6 step process he calls, “Lean AI” for enterprises to implement AI services to their business. He will start with how to cast business problems as machine learning problems—from identifying which business problem to solve, to hypothesis generation, to AB testing. He’ll explain how to prioritize business value vs. data value, and explain how to transform, segment, and measure variables to obtain meaningful data unique to each company. Finally, he’ll share examples from companies he’s worked with that have implemented the 6 steps and different AI technologies to analyze specific areas of their business and resolve their toughest problems. SOURAV DEY, Managing Director and Head of Data Science, Manifold

1. LEAN AI: SIX STEPS FOR IMPLEMENTING
AI IN THE ENTERPRISE
DR. SOURAV DEY | @resdntalien
DATAWORKS SUMMIT
JUNE 20, 2018

2. © 2018 Manifold. All rights reserved. | 2
Manifold is an engineering services firm that
accelerates AI development for Global 500
and high-growth companies.
ABOUT US

3. © 2018 Manifold. All rights reserved. | 3
THE POINT OF THIS TALK
•Share our Lean AI mental model for applied AI
•Make it real using some case studies from our work

4. © 2018 Manifold. All rights reserved. | 4
LEAN AI PLAYBOOK
#TackleBigRisksEarly

5. © 2018 Manifold. All rights reserved. | 5
UNDERSTAND

6. © 2018 Manifold. All rights reserved. | 6
AI VALUE ≤ BUSINESS VALUE X DATA QUALITY X PREDICTIVE SIGNAL
AI UNCERTAINTY PRINCIPLE
Multiplicative! If any term goes to 0, value goes to 0!

7. © 2018 Manifold. All rights reserved. | 7
BATTLE UNCERTAINTY WITH AN AI SPEC
Business Problem Workshop:
• What is the ROI?
• If this could be predicted,
how would it help your
company?
• Etc.
Data Audit:
• Where are you data
sources?
• How much data?
• How rare is the event?
• Is the data labelled well?
• Is it joinable?
• It is trustworthy?
• tc.

8. CASE STUDY #1
“We want to use AI to be more customer centric. 80% of revenue
comes from 20% of customers. How can we identify our most loyal
customers and deliver more value to them?”
LEADING BABY REGISTRY IN THE U.S.

9. © 2018 Manifold. All rights reserved. | 9
CASE STUDY #1: AI SPECIFICATION
• We will create models to predict
if a customer is going to adopt,
activate, and $LTV after 9
months
• We will generate this prediction 1
day after signup, 2 days, 7 days,
30 days, N days
• We will only use transactional DB
as the only data source—there
seems to be enough data in there
to create meaningful features
LEADING BABY REGISTRY IN THE U.S.
9 months

10. CASE STUDY #2
“We want to use AI to be more efficient across our operations.
The vision is to create a system for making better decisions.”
LEADING INDUSTRIAL SERVICES COMPANY

11. © 2018 Manifold. All rights reserved. | 11
CASE STUDY #2: AI SPECIFICATION
Lookback
= 2 days
Horizon = 5 days
LEADING INDUSTRIAL SERVICES COMPANY
• Predict major faults where machine is continuously
down for >2 hours.
• Major faults almost always lead to customer calls,
truck rolls, and downtime.
• Predict whether major fault will happen over a horizon
of 1, 2, … ,5 days.
• Use machine-generated data as input features, e.g.
~30 continuous time series, ~20 discrete time series.
• Use demographic data about machines, e.g. unit type,
location, etc.
• Do not use human-generated service data because of
data quality issues.

12. © 2018 Manifold. All rights reserved. | 12
MODEL

13. © 2018 Manifold. All rights reserved. | 13
MODELLING IS JUST
THE TIP OF THE
PYRAMID
Source: Monica Rogati

14. © 2018 Manifold. All rights reserved. | 14
BUILD A BASELINE MODEL, NO EXCEPTIONS
It’s all about learning!
Then iterate, iterate, iterate.
• classification > regression
• class errors are easier to understand learn from
• even for continuous targets, you may want to do a binary (or multiclass)
classifier before regression
• random forest > gradient boosted trees > deep learning
• few parameters to tune, robust to overfitting, quick to train
• interpretable feature importance to learn from
• pick a few features to start, then create more features

15. © 2018 Manifold. All rights reserved. | 15
EVALUATE TO LEARN
• Aggregate Metrics
• Cross-Validated ROC and AUC = your
score to improve by iterative modelling
• feature importance done properly
• Individual Metrics
• prediction probability distribution
• “Four corners and the middle analysis”
• most accurate negatives
• most accurate positives
• least accurate negatives
• least accurate positives
• least certain estimates

16. © 2018 Manifold. All rights reserved. | 16
CASE STUDY #1: MODEL ITERATION
NO NEED FOR HYPER OPTIMIZATION
Predict LTV > 0 7 days after signup only using 2 easiest
features to engineer: “platform used to sign up” and “referrer”
Predict LTV > 0 7 days after signup only using 11 most
important features which includes early activity features
AUC = 0.65 AUC = 0.90

17. © 2018 Manifold. All rights reserved. | 17
CASE STUDY #2: MODEL ITERATION
DIMINISHING RETURNS
Feature Matrix
Deep Learning
(CNNs)
Tree Methods
(RF and GBT)
Feature
Engineering
Mixed Effects
Models
(MERF)
#1
#2
#3

18. © 2018 Manifold. All rights reserved. | 18
USER FEEDBACK

19. © 2018 Manifold. All rights reserved. | 19
GET USER FEEDBACK ASAP
• Multiple structured sessions with
final end users
• Use prototype tooling — e.g.,
nothing, Excel, Jupyter notebooks
• Observe their workflow and how
they integrate predictions

20. © 2018 Manifold. All rights reserved. | 20
TRUST NOBODY, ESPECIALLY MODELS
• Does the aggregate predicted
failure rate for a daily cohort
match the historical average I’m
familiar with?
• If sensor A goes above X psi,
likelihood of failure goes up, what
does the model say?
• Sensitivity analysis can show that
modelling is not magic; it’s
heuristics you know codified into a
mathematical model.
MODELS HAVE TO EARN TRUST

21. © 2018 Manifold. All rights reserved. | 21
DELIVER SOLUTIONS, NOT MODELS
• The raw predictions almost
always need post processing
before they are useful.
• It is our job as AI engineers to
create workflow tools or APIs that
help users derive value from the
AI.
• BUILD THE UI FOR THE AI

22. © 2018 Manifold. All rights reserved. | 22
CASE STUDY #1: PRODUCT CHANGE DECISION
Problem
I want to change the product, so that if a user takes certain key actions (that you helped
me identify), they will get a free box full of goodies. I want to run this promotion for a few
weeks and use your model to determine if this is a promotion worth running long term.
I.e., is the cost of the box worth it? Will the higher CAC be offset by higher LTV?
Solution
• Temporal AB test with predicted LTV
• Need to use a model without the features that are incentivized, i.e. need to remove
confounding features.

23. © 2018 Manifold. All rights reserved. | 23
CASE STUDY #2: DIRECTED TRIAGE
Problem
Most high probability of fault units are known stressed units, so just looking at raw
predictions leads to many false alarms and erosion of trust.
Also, the human can spend lots of time looking at data and may not be able to see what
the AI sees. We want triage to be directed.
Solution
• Rules on historical predictions to find “interesting events”, e.g. day on day % prob
change
• Explainable AI using TreeSHAP that identifies which factors are driving the
increased probability of failure

24. © 2018 Manifold. All rights reserved. | 24
LOTS MORE WE CAN’T COVER TODAY
Don’t be a pirate, be the Navy.
Embed high cardinality
categorical variables!
Use Docker, damnit.

25. THANK YOU
Dr. Sourav Dey | sdey@manifold.ai | @resdntalien