The document is a presentation about AI and enterprise software investing in 2019. It discusses the current AI revolution and why progress is happening now due to increases in data, computing power, and algorithms like deep learning. It provides a framework for investing in enterprise AI and discusses challenges like differentiating from large cloud providers. Key metrics for evaluating AI investments are discussed, like error rates, ROI, human intervention levels, and how the AI improves over time with more data.

1. Beyond the Hype:
AI & Enterprise Software Investing in 2019
April 16, 2019

2. [ 2 ] C O N F I D E N T I A L / T R A D E S E C R E T
Introduction to the Venture Capital Landscape
• AI – specific questions
• Questions on the broader venture ecosystem
• How to get into venture
The Current AI Revolution:
Why Now and Technical Progress
Questions from the Audience
AI Investing:
Guiding Themes, Key Questions, and Sample Markets
The AI Metrics that Matter and How We Decide
TODAY’S AGENDA
[ 2 ] C O N F I D E N T I A L / T R A D E S E C R E T

3. [ 3 ] C O N F I D E N T I A L / T R A D E S E C R E T
• Principal, Scale Venture Partners (Joined in Summer 2015)
• Econ Researcher @ NY Fed; Data Science @ Analysis Group;
Sales Strategy & Business Operations @ Salesforce
• Focus on AI applications, frontier tech, and vertical software
My Background
Background
Investments

4. [ 4 ] C O N F I D E N T I A L / T R A D E S E C R E T
Introduction to the Venture Capital Landscape

5. [ 5 ] C O N F I D E N T I A L / T R A D E S E C R E T
Scale Venture Partners Overview
5 Partners
1.3B Assets
Under Management
Closed $400M
Fund VI $400M - July 2018
The Facts
Sector
Enterprise software focus
Stage
Early, in-revenue and
ready to scale
Sourcing
Proactive, driven by own
research on tech trends
Team
Active board members focused
on scaling the business
The Strategy
Funding the Future of Work
The Portfolio
39 Exits
Creating market cap of $17.5B

6. [ 6 ] C O N F I D E N T I A L / T R A D E S E C R E T
ULTRA LATEEARLY LATE
A Simplified Version of the Venture Landscape
Early
Openview Partners
Mid
All Stage: Balanced/ Conglomerates
MIDEARLY
Larger but still early and mid
All Stage: Strategics

7. [ 7 ] C O N F I D E N T I A L / T R A D E S E C R E T
ScaleVP: Investing in AI
Standard
inputs/outputs
New ways of input
System of
RECORD Transactions/balances, etc.
Infrastructure
SOFTWARE
System of
ENGAGEMENT
… …
Customers
*Investment made 9/15/17; not yet announced
?
System of
PREDICTION

8. [ 8 ] C O N F I D E N T I A L / T R A D E S E C R E T
The Current AI Revolution: Why Now and Technical
Progress

9. [ 9 ] C O N F I D E N T I A L / T R A D E S E C R E T
Computer
VISION
SPEECH
Recognition
Machine
LEARNING
Natural
LANGUAGE
Processing
What is AI?
AI is not a single
technology, but a
set of computing
disciplines and
techniques that
cumulatively “seem
to enable
computers to think
and learn like
people”
ARTIFICIAL
INTELLIGENCE
MACHINE
LEARNING
DEEP
LEARNING
AI characterized as human (but isn’t)

10. [ 10 ] C O N F I D E N T I A L / T R A D E S E C R E T
Progress of AI Is Regularly Overestimated
1950s
An attempt will be made to
find how to make machines use
language, form abstractions and
concepts now reserved for humans.
– Dartmouth Conference, 1956
“ The existence of AI is
beginning to be felt outside of academia,
and in a few years the computer and
society as we know it is likely to be
dramatically transformed.
– Byte Magazine, 1980
Present Day1980s
““ Artificial intelligence
on a level with human intelligence is a
very strong likelihood in the near future
– Newsweek, 2016

11. [ 11 ] C O N F I D E N T I A L / T R A D E S E C R E T
So Why Are Predictions Often Wrong?
1
2
3
AI gives “probable”
answers not definite
answers, which
means some
answers
will be wrong
It is not clear
how many wrong
answers are too many
All is
too exciting!

12. [ 12 ] C O N F I D E N T I A L / T R A D E S E C R E T
But This Time is Different….Really!
Data
INTERNET
Compute
GPU + CLOUD
COMPUTE
Algorithms
RISE OF
DEEP LEARNING
1 2 3

13. [ 13 ] C O N F I D E N T I A L / T R A D E S E C R E T
Compute Used in AI Training Runs is Increasing Faster Than Moore’s Law
Linear Scale Log Scale
The amount of
compute used in the
largest AI training
runs has been
increasing
exponentially with a
3.5 month doubling
time (by comparison,
Moore’s Law had an
18 month
doubling period!)
Source: OpenAI Foundation Research

14. [ 14 ] C O N F I D E N T I A L / T R A D E S E C R E T
Clear Progress in Language, Speech, and Vision
Vision
3%
Errors
5%
Errors
Humans
26%
Errors
Speech
2013 2015 2016
Deep LearningStatistical
Word
Error Rate
(%)
Time
23%
8.9%
5.9%
Driven by Deep Learning
Natural Language
Processing (NLP)
2007 2019
Human
Level
Deep Learning
StatisticalBLEU
Score

15. [ 15 ] C O N F I D E N T I A L / T R A D E S E C R E T
A General Framework for Investing in Enterprise AI

16. [ 16 ] C O N F I D E N T I A L / T R A D E S E C R E T
Yet AI Will Not Re-Write Every Business Application Like SaaS did to On-Prem
SaaS
Same SaaS Company
Acquires AI Company
On Prem
CRM Market HR Market
SaaS
More Modern
SaaS Company
Acquires AI Company
On Prem

17. [ 17 ] C O N F I D E N T I A L / T R A D E S E C R E T
Because AI Does Not Replace a Core Business System of Record
A Map of Enterprise Software
Standard
inputs/outputs
New ways of input
System of
RECORD Transactions/balances, etc.
Infrastructure
SOFTWARE
System of
ENGAGEMENT
… …
Customers
System of
PREDICTION
Where AI will have more of an Impact Where AI will have less of an impact

18. [ 18 ] C O N F I D E N T I A L / T R A D E S E C R E T
Meaning every AI market now becomes a question of whether
the horizontal cloud vendor will stop the up-start vertical player in its tracks
And Cloud Vendors Are Ferocious Competitors With Intimidating Amounts of Data

19. [ 19 ] C O N F I D E N T I A L / T R A D E S E C R E T
How to Differentiate vs. Horizontal Cloud Providers

20. [ 20 ] C O N F I D E N T I A L / T R A D E S E C R E T
There Are a Variety of Clever Hacks for Companies to Gather Proprietary Data
Sell Workflow First, AI Second
Incent Data Sharing With Clever Pricing
Scrape Data to Get Started for MVP
Partner With a Relevant Institution in the Field
Radiology and
Pathology Scans
Camera Data
on Crimes
Committed

21. [ 21 ] C O N F I D E N T I A L / T R A D E S E C R E T
But Beating the Cloud Vendors Requires More than Just a High Volume of Data
The
“Data is the New Oil”
argument is a fallacy
because unlike oil, data is
not a commodity whose value
is derived solely from quantity.
Evolution of Data Strategies
2007 2011 2017
Workflow Tools
(No Data)
Data Aggregation
Data-driven Virtuous
Loops (AI/ML)
Data Moats
Accessibility
How easy was it to get?
Time
How quickly can the data be amassed and used in the model?
Cost
How much money is needed to acquire and/or label this data?
Uniqueness
Is similar data widely available to others who could then build
a model and achieve the same result?
Dimensionality
How many different attributes are described in a data set?
Statistical Distribution of Data
How widely do the values of attributes vary,
such that they may account for edge cases and rare exceptions?
Perishability
Will the data be useful for a long time?
Customer Cloud Architecture
To pool learnings across customer.
Virtuous Loop
Can outcomes be used as inputs to improve the algorithm?
Algorithmic Time to Sufficiency
If only a few data points needed to train model, easy to replicate.
Source: Credit to Zetta Venture Partners for publishing this argument

22. [ 22 ] C O N F I D E N T I A L / T R A D E S E C R E T
Closing the Loop with Humans Can Be a Source of Competitive Data Advantage
Classic example of a “Closed Loop” where the
system learns from previous translated jobs
Great metrics comparing the
i) horizontal tool vs. ii) vertical tool vs. iii) vertical tool with humans

23. [ 23 ] C O N F I D E N T I A L / T R A D E S E C R E T
Recognizing Technical Limitations and Timing Risk

24. [ 24 ] C O N F I D E N T I A L / T R A D E S E C R E T
Keep in Mind that AI will have Multiple Different S Curves of Adoption
% of Total
Adoption
1956 Dartmouth
Conference
2019 2025
TODAY
Fraud Detection
Solutions
Speech
Recognition
AI-Based
Medical
Imaging
Autonomous
Cars

25. [ 25 ] C O N F I D E N T I A L / T R A D E S E C R E T
For Each Adoption Curve, Which Sub-Markets Have the Least Timing Risk?
Requires 100m+
in capital
Monetization scales
with number of
miles being driven
Requires making
a bet on hardware
Less timing risk,
but GTM risk
of selling to OEMs

26. [ 26 ] C O N F I D E N T I A L / T R A D E S E C R E T
And Intuition Is a Poor Guide to Understanding Degree of Technical Difficulty
Which is Easier?
Tech Risk is Hard to Assess
Flying Plane Driving Car Welding a Car Picking Up Fruit
Humans
Machines
Hard
Autopilot for 20 years
Easier
Autonomous Driving
Still in Development
Hard
Robot for many years
Easier
Challenging for Robots
vs vs

27. [ 27 ] C O N F I D E N T I A L / T R A D E S E C R E T
And Yet: Machines Sometimes Are Not Quite Good Enough
Summarization and understanding of human politics are beyond the ability of AI today
Legal Case Summaries Scheduling Meetings/ Calendaring
Human Written Machine Written
BACKGROUND: Defendant was convicted, after a jury trial in the
Washington Superior Court, Thurston County, Richard A. Strophy, J.,
of first-degree assault while armed with deadly weapon. Defendant
appealed. The Washington Court of Appeals reversed. On review,
the Washington Supreme Court, 147 Wash.2d 424, 54 P.3d 656,
reversed and reinstated defendant's conviction. Certiorari was granted.
HOLDINGS: The Supreme Court, Justice Scalia, held that:
 Out-of-court statements by witnesses that are testimonial are
barred, under the Confrontation Clause, unless witnesses are
unavailable and defendants had prior opportunity to cross-examine
witnesses, regardless of whether such statements are deemed
reliable by court, abrogating Ohio v. Roberts, 448 U.S. 56, 100 S.Ct.
2531, 65 L.Ed.2d 597.
At his trial, the State played for
the jury Sylvia's tape-recorded
statement to the police describing
the stabbing, even though he had no
opportunity for cross-examination.
The Washington Supreme Court
upheld petitioner's conviction after
determining that Sylvia's statement
was reliable.
Petitioner Michael Crawford
stabbed a man who allegedly
tried to rape his wife, Sylvia.

28. [ 28 ] C O N F I D E N T I A L / T R A D E S E C R E T
The AI Metrics That Matter

29. [ 29 ] C O N F I D E N T I A L / T R A D E S E C R E T
At Some Level, an “AI Deal” is Evaluated Just Like Any Other Deal
The Target Market
Founding Team
The Product Vision
A
Customer Traction
Market Size
Management Team
ARR Growth
B
ARR Growth
Success Metrics
Number 1 in Market
C

30. [ 30 ] C O N F I D E N T I A L / T R A D E S E C R E T
All Should Be Evaluated At the Customer Level Over Time
Yet Traditional SaaS Metrics Don’t Give The Complete Picture
What is “good enough”
varies by deal and market
The more tangible the better
Helps answer whether the AI error
rate is “Reasonable Enough”
Solves the “revealed preference” problem
of buyers not knowing what they want
ROI/Usage Data
Shows whether it is a true
closed-loop, virtuous cycle
Can proxy using gross margin
Should decline over time
Answers the question of scalability
vs. one-off consulting services
Human Intervention Data
Need to fully understand meaning
of numerator and denominator
Should improve over time
Somewhat of a vanity metric as it doesn’t
address the value of solved resolutions
Containment /
Self Service Resolution

31. [ 31 ] C O N F I D E N T I A L / T R A D E S E C R E T
An Easy-To-Measure Customer ROI Helps Us Overcome The AI Hype
Change in time-to-fill Lift in screening rate Lift in interview rate Lift in offer rate Lift in % women
Max -65% 100% 65% 23% 48%
Third Quartile -26% 30% 26% 12% 24%
Median -19% 25% 20% 8% 19%
First Quartile -15% 20% 16% 5% 12%
Min -8% 14% 9% 2% 4%
Note: Numbers are modified as true numbers are confidential

32. [ 32 ] C O N F I D E N T I A L / T R A D E S E C R E T
A Successful Data Acquisition Strategy Means Human Intervention Declines
With Additional Data, MQM Increases
Improved Editor Productivity Falling Cost Per Word
Fewer Humans Per QuerryDeclining Human Intervention Ratio
Improving F1 Score from Quality
Estimation

33. [ 33 ] C O N F I D E N T I A L / T R A D E S E C R E T
Key Questions:
 Does this asymptope
over time?
 Are the requests the
AI handles high value?
 What does the
distribution of requests
look like? Any
clustering?
A Successful Data Acquisition Strategy Means the AI Gets Better Over Time
Queries Per User By WeekContainment/ Self Service Resolution Rate

34. [ 34 ] C O N F I D E N T I A L / T R A D E S E C R E T[ 34 ] C O N F I D E N T I A L / T R A D E S E C R E T[ 34 ] C O N F I D E N T I A L / T R A D E S E C R E T[ 34 ] C O N F I D E N T I A L / T R A D E S E C R E T
THE ASK
Please Put Me
In Touch With
Interesting AI
Companies and
Investors!
Jeremy Kaufmann
Principal, Scale Venture
Partners
jeremyk@scalevp.com

35. [ 35 ] C O N F I D E N T I A L / T R A D E S E C R E T
Questions?