Customer case study at Databricks Unified Analytics Workshop: Data Science challenges @ Outreach and how we leverage Databricks. Event: https://pages.databricks.com/Unified-Analytics-Seattle-Reg.html

1. Databricks Workshop - April 11th 2019
Case Study: Outreach
Andrew Brooks, Li Dong, Jiwei Cao

2. Content Overview
1. Introduction to Outreach
a. #1 Sales Engagement Platform
b. Data Science + Sales Engagement
2. Outreach with Databricks
a.Case Study - Production: Out-of-office Data Extraction
b.Case Study - Research: Intent Classification
2

3. Introduction to Outreach
The #1 Sales Engagement Platform

4. Wait, what is a Sales Engagement Platform?
4

5. It’s a book.
5

6. It’s also a new category of software.
Add content
6
Source: https://blog.topohq.com/sales-engagement-the-definitive-guide/

7. How about an Example?
7
Automates execution of some sales tasks:
Emails, Linkedin Messages etc.
Schedules and reminds the rep when it is the
right time to do the manual tasks (e.g. phone
call, custom manual email)
SEP Encodes and Automates Sales
Activities into Workflows/Pipelines

8. Data Science + Sales Engagement
Outreach ML Features:
- Automation - Information Extraction
- A/B testing
- Advanced Analytics (dashboard & reporting)
Optimization:
- Intent & Topic Detection
- Content & Action Recommendation
- Prioritization & Forecasting
8
Highlighted use case
Phone
Email
LinkedIn
Meetings
Data Sources:

9. Case Study: Out-of-office Data Extraction

10. Automation - Information Extraction
Highlighted use case: Out of Office return date and referral extraction
10

11. 11
Production Architecture
Highlighted use case: OOO Information Extraction

12. 12
Something to improve
- Separate stacks to develop ML model and deploy model
- jupyter notebooks
- databricks notebooks
- Docker, K8s(production)
- Lack of ML model life-cycle management
- model training
- experiment(alpha, beta)
- production to GA
- model iterations / releases
Production Architecture
Highlighted use case: OOO Information Extraction

13. 13
Production (Next Step…)
Highlighted use case: OOO Information Extraction

14. Case Study: Intent Classification

15. Intent Classification: Provide More Value Metrics
15

16. Intent Classification: Problem Solving
Steps to solve it with NLP/Machine Learning:
1. Annotate some emails
2. Setup the Experiment Environment(Spark, NLP/ML-Packages)
3. Write Code and Running Experiments
4. Analyze Experiment Results
16
Goal: Classify email replies into 3 categories: positive, objection and unsubscription.

17. Doesn’t Look Complicated, But Painful
Pain Points:
● Difficult to setup and maintain a proper environment.
● Can’t run multiple Experiments at the same time
● Experiment Results are scattered in multiple files. Hard to navigate and analysis.
17

18. Intent Classification: Problem Solving
Steps to explore ideas with Databricks:
1. Annotate some emails
2. Setup the Experiment Environment(Spark, NLP/ML-Packages)
3. Write Code and Running Experiments
4. Analyze Experiment Results
5. Visualization the model prediction
18
Goal: Classify email replies into positive, objection and unsubscription.

19. Visualize the Model Prediction: t-SNE
19
Before Training After Training

20. How does Databricks help us
• Setup a Dedicated Environment with Less Effort
• Running Experiments at Scale
• Analyze Experiment Results at One Place
20

21. Q & A