This document summarizes an ML driven sales and marketing presentation. It discusses using ML to score prospect engagement and readiness to buy. Models are trained on prospect activity data and used to triage prospects for nurturing or calls. Marketers can then funnel prospects to campaigns based on their scores. The presentation also covers interpreting model predictions, monitoring model quality over time, and lessons learned in building ML products for B2B.

1. ML driven Sales and Marketing for Everyone
Dynamic Talks, San Francisco
Feb-27, 2020
@being_bayesian
Srivatsan Ramanujam,
Director of Engineering, Salesforce Einstein
Previously ML at Pivotal, Sony, IBM Almaden Research and UT Austin CS

2. Building a strong Sales
pipeline starts with
Marketing
• Generating demand, nurturing
leads and qualifying them
Winning more deals requires
efficient forecasts
• Identify risks early to meet
targets
Too many leads to pursue under limited time and budget
Integrated Sales and Marketing Funnel

3. Who is ready to buy and who needs to be nurtured?
Triaging Prospects
Ignore Nurture Call
We call this the Behavior Score of prospects which is indicative of their readiness to buy
Jan-2020 Dec-2019 Feb-2020

4. Marketers typically have a few
hundred prospects they are
qualifying every week
• Ability to sort prospects by Behavior
Score is key to productivity
Marketers are also interested in
what factors are driving the
readiness to buy of prospects
• Record level insights as well as
broad trends across the business are
both of interest
List Views with Prospect Scores and Insights
Qualifying Prospects

5. Marketers can funnel
prospects to different
campaigns based on
their Behavior Score
• Hot prospects might be
good candidates for a
sales call
• Inactive prospects could
be nurtured with right
content to re-engage
them
The power of workflows
Nurturing Prospects
https://trailhead.salesforce.com/en/content/learn/modules/pardot-engagement-studio/engagement-studio-best-practices

6. Modeling Prospect Engagement

7. Starting Simple
ML may not be the first solution!

8. We did not attempt to model long range outcomes such as an opportunity being won/lost, as we have
another product that scores opportunities
Prospect’s Lifecycle
Defining the Target

9. A vast majority of prospects that
link to an opportunity, do so
within a time range which we
define as the ‘forecast horizon’
Different orgs have different
durations for their sales cycles
• We learn org specific forecast
horizons to account for this
Unconverted until now != will never convert
Censored Prospect Outcomes

10. Non-overlapping grid search, validation and scoring sets
Modeling Datasets

11. Capturing prospect’s outcomes throughout its lifetime
Snapshotted Prospect Activity Count Feature Vectors
The activity counts for each prospects are further bucketed into time-bins (ex: activity counts in the
last day, week, month, year etc.)

12. Classifiers, Rankers and Blended Models with Email Embeddings
Models
XGBoost LTRXGBoost Classifier Structured Attributes + Sales Activity Email Embeddings

13. Insights: Model Explainability

14. Accuracy vs. interpretability trade off?
Earning Trust: Interpretable Predictions
Simple Explainable Model Scores and Insights
Blackbox ML model
?
Title = Sales Intern, Engagement = Low => Low Score Title = CXO, Engagement = Low => High Score
Challenge: simple models cannot capture feature interactions
Challenge: complex models are harder to explain
Modeling Dataset

15. Don’t trade-off accuracy for explainability
Predictors and Explainers: Global and Local Surrogates
Simple Surrogate Model
“Explainer”
Modeling Dataset with PredictionsBlackbox ML modelModeling Dataset
Challenge:
maintaining
multiple models,
keeping models
in sync
Techniques such as LIME are based on local surrogates, where an explainable model is trained on the predictions of the black box model on
perturbations of original dataset weighted by their proximity to the instance of interest

16. SHapley Additive exPlanations
SHAP: Unifying Predictor and Explainer
Blackbox ML model
88
35
72
The higher an attribute’s influence on the score, the
greater is its importance
Title is “CXO”
Annual Revenue > $100000
.
.
Modeling Dataset
For a deeper dive on SHAP, we refer interested readers to Lundberg et. al, the blog post by Gabriel Tseng and the paper on Shapley sampling
values by Strumbelj et. al.

17. One model to rule them all
SHAP Powered Local Insights
Record Level Insights

18. Understanding Factors Driving Conversion at the Company Level
SHAP Powered Global Insights

19. Quality Scores and Insights for
Everyone

20. Scores and Insights for all Customers
Model Tournament

21. Architecture: Modeling and
Monitoring

22. We leverage many open source technologies
Architecture Overview

23. Serving Data Scientists, DevOps and Product Managers
Quality Monitoring Framework
Product Managers
Data Scientists
DevOps

24. Per tenant and cross-tenant model metrics
Tracking Model Quality

25. Closing Thoughts

26. Learnings
Building ML Products in B2B
Clean data is foundational for building ML
based systems
• Resort to expert encoded rules if the data is noisy
Do not underestimate the power of simple
• Rolling out simpler models quickly could help get
timely feedback that would buy you much needed
time to build more sophisticated solutions
Invest early and heavily in monitoring tools
• Real world datasets are constantly evolving
(GDPR, distributional shift etc.)
• Be wary of data leaks, counter-intuitive insights
etc.
ML product development is iterative
• Conventional thinking: create datasets, run
experiments, deploy to production (one-way,
waterfall)
• Reality: create datasets, run experiments,
deploy, find issues, fix dataset, run experiment,
fail to reproduce results, fix bugs, deploy, find
issues again, make product decisions...

27. Keita Himanshi
Checkout our Engineering blog for more:
1.Machine Learning Driven Sales and Marketing for Everyone with Einstein Behavior Scoring (Part 1)
2.Machine Learning Driven Sales and Marketing for Everyone with Einstein Behavior Scoring (Part 2)
Brad Kas Arie Vatsan Swapna Narae Itamar Mark Subha Dov

28. Forward-Looking Statement
Statement under the Private Securities Litigation Reform Act of 1995:
This presentation contains forward-looking statements about the company’s financial and operating results, which may include expected GAAP and non-GAAP financial and other operating
and non-operating results, including revenue, net income, diluted earnings per share, operating cash flow growth, operating margin improvement, expected revenue growth, expected
current remaining performance obligation growth, expected tax rates, the one-time accounting non-cash charge that was incurred in connection with the Salesforce.org combination; stock-
based compensation expenses, amortization of purchased intangibles, shares outstanding, market growth and sustainability goals. The achievement or success of the matters covered by
such forward-looking statements involves risks, uncertainties and assumptions. If any such risks or uncertainties materialize or if any of the assumptions prove incorrect, the company’s
results could differ materially from the results expressed or implied by the forward-looking statements we make.
The risks and uncertainties referred to above include -- but are not limited to -- risks associated with the effect of general economic and market conditions; the impact of geopolitical events;
the impact of foreign currency exchange rate and interest rate fluctuations on our results; our business strategy and our plan to build our business, including our strategy to be the leading
provider of enterprise cloud computing applications and platforms; the pace of change and innovation in enterprise cloud computing services; the seasonal nature of our sales cycles; the
competitive nature of the market in which we participate; our international expansion strategy; the demands on our personnel and infrastructure resulting from significant growth in our
customer base and operations, including as a result of acquisitions; our service performance and security, including the resources and costs required to avoid unanticipated downtime and
prevent, detect and remediate potential security breaches; the expenses associated with new data centers and third-party infrastructure providers; additional data center capacity; real estate
and office facilities space; our operating results and cash flows; new services and product features, including any efforts to expand our services beyond the CRM market; our strategy of
acquiring or making investments in complementary businesses, joint ventures, services, technologies and intellectual property rights; the performance and fair value of our investments in
complementary businesses through our strategic investment portfolio; our ability to realize the benefits from strategic partnerships, joint ventures and investments; the impact of future gains
or losses from our strategic investment portfolio, including gains or losses from overall market conditions that may affect the publicly traded companies within the company's strategic
investment portfolio; our ability to execute our business plans; our ability to successfully integrate acquired businesses and technologies, including delays related to the integration of
Tableau due to regulatory review by the United Kingdom Competition and Markets Authority; our ability to continue to grow unearned revenue and remaining performance obligation; our
ability to protect our intellectual property rights; our ability to develop our brands; our reliance on third-party hardware, software and platform providers; our dependency on the development
and maintenance of the infrastructure of the Internet; the
effect of evolving domestic and foreign government regulations, including those related to the provision of services on the Internet, those related to accessing the Internet, and those
addressing data privacy, cross-border data transfers and import and export controls; the valuation of our deferred tax assets and the release of related valuation allowances; the potential
availability of additional tax assets in the future; the impact of new accounting pronouncements and tax laws; uncertainties affecting our ability to estimate our tax
rate; the impact of expensing stock options and other equity awards; the sufficiency of our capital resources; factors related to our outstanding debt, revolving credit facility, term loan and
loan associated with 50 Fremont; compliance with our debt covenants and lease obligations; current and potential litigation involving us; and the impact of climate change.
Further information on these and other factors that could affect the company’s financial results is included in the reports on Forms 10-K, 10-Q and 8-K and in other filings it makes with the
Securities and Exchange Commission from time to time. These documents are available on the SEC Filings section of the Investor Information section of the company’s website at
www.salesforce.com/investor.
Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements, except as required by law.