The document discusses the development of a pricing request classifier using email data and machine learning techniques, focusing on identifying pricing requests among various email types. It highlights challenges in data labeling due to the rarity of pricing requests and outlines approaches like word2vec and LDA to improve label generation and feature vectors. Upcoming improvements include using labeled data for better filtering methods and incorporating user feedback.

1. 1
Sammy Nammari
Identifying Pricing Request
Emails Using Spark and
Machine Learning

2. Agenda
• Inbox and email data
• Building a pricing request classifier

3. Agenda
• Inbox and email data
• Building a pricing request classifier

4. The image part with relationship ID rId3 was not found in the file.
Salesforce Inbox
Productivity apps powered by AI
4
Video

5. What sorts of emails do salespeople receive?
• Emails from customers
o Meeting requests, pricing requests, competitor mentioned, etc.
• Emails from coworkers
• Marketing emails
• Newsletters
• Telecom, Spotify, iTunes, Amazon purchases
• Etc

6. Pricing requests
​ We want to identify pricing requests from customers
Hey Ascander,
How much would it
cost to add ten seats
to the plan?
Thanks,
Gabe
Hello Eddie,
Can you send me that
really important
document?
Thanks,
Alexis
Welcome to Spotify!
Your new subscription
is active.
Enjoy the music.

7. Agenda
• Inbox and email data
• Building a pricing request classifier

8. Building a pricing request classifier
Challenges:
• No labels, and currently no mechanism to automatically generate
labels
• Pricing requests are a relatively rare event
• Emails are sensitive — can’t mechanical turk

9. Data pipelines
• Data labeling
• Generating feature vectors and model training

10. Data labeling pipeline
Word2Vec
Graph
Filtering /
Sampling
Labeling tool
Emails
Labeled
Training
Data

11. Generating feature vectors and model training
Feature
Engineering
Model
Training
Model
Evaluation
LDA
Text
Processing
/ TF-IDF
Labeled
Training
Data
Emails

12. Data used
​ Billions of emails that we process over time
​ ~2.5 million internal emails that we have anonymized and
have explicit permission to label

13. Structure of an email
Hey Alexis,
Let’s meet with Ascander on Friday to discuss
the $10,000/year rate. Ascander’s phone
number is (123) 456-7890.
Thanks,
Noah Bergman
Engineer at Salesforce
(123) 456-7890
The contents of this email and any attachments
are confidential and are intended solely for
addressee…
From: Alexis alexis@salesforce.com
Date: April 1, 2017
Subject: Important Document
Noah, how much does your product cost?

14. Structure of an email
INTRO
SIGNATURE
CONFIDENTIALITY NOTICE
REPLY CHAIN
BODY
Hey Alexis,
Let’s meet with Ascander on Friday to discuss
the $10,000/year rate. Ascander’s phone
number is (123) 456-7890.
Thanks,
Noah Bergman
Engineer at Salesforce
(123) 456-7890
The contents of this email and any attachments
are confidential and are intended solely for
addressee…
From: Alexis alexis@salesforce.com
Date: April 1, 2017
Subject: Important Document
Noah, how much does your product cost?

15. Structure of an email
INTRO
SIGNATURE
CONFIDENTIALITY NOTICE
REPLY CHAIN
BODY
Hey Alexis,
Let’s meet with Ascander on Friday to discuss
the $10,000/year rate. Ascander’s phone
number is (123) 456-7890.
Thanks,
Noah Bergman
Engineer at Salesforce
(123) 456-7890
The contents of this email and any attachments
are confidential and are intended solely for
addressee…
From: Alexis alexis@salesforce.com
Date: April 1, 2017
Subject: Important Document
Noah, how much does your product cost?

16. Labeling Data

17. Labeling data
• No labels, and currently no mechanism to infer labels
• Pricing requests are very important, but relatively rare events
Hand-labeling impractical

18. Labeling data – high-recall filter
​ How can we get a higher yield of positive labels when labeling by hand?

19. Labeling data – high-recall filter
How do we build this green circle?
• Relationship graph (GraphX)
• Word2Vec

20. Labeling data – Word2Vec
What would be the total cost of a …
How much would it cost to add ten seats to the plan?
Does it cost a lot of money to …
Neural network that finds words similar to cost based on the context that it
appears in

21. Labeling data – Word2Vec
• Train Word2Vec on unlabeled emails
• find words close in distance to “price”, “cost”, “license”, etc

22. Things we calculated after we got labels
• Our original dataset was 0.17% positive labels
• Graph + Word2Vec reduced our dataset to 2% of its original size, and
increased the positive label rate to 11.2%, with a recall of 0.93
We’ve introduced some bias, but hand-labeling is now tractable!

23. Improving the output produced by Word2Vec
INTRO
SIGNATURE
CONFIDENTIALITY NOTICE
REPLY CHAIN
BODY
Hey Alexis,
Let’s meet with Ascander on Friday to discuss
the $10,000/year rate. Ascander’s phone
number is (123) 456-7890.
Thanks,
Noah Bergman
Engineer at Salesforce
(123) 456-7890
The contents of this email and any attachments
are confidential and are intended solely for
addressee…
From: Alexis alexis@salesforce.com
Date: April 1, 2017
Subject: Important Document
Noah, how much does your product cost?

24. Improving the output produced by Word2Vec
“Let’s meet with Ascander on Friday to discuss the $10,000/year rate.
Ascander’s phone number is (123) 456-7890.”
Names, monetary values and phone numbers are noisy

25. Improving the output produced by Word2Vec
word2VecModel.findSynonyms(“cost”, 5)
$10
price
$85/month
$19.99
$15,000/year

26. Improving the output produced by Word2Vec
“Let’s meet with Ascander on Friday to discuss the $10,000/year rate.
Ascander’s phone number is (123) 456-7890.”

27. Improving the output produced by Word2Vec
“Let’s meet with NAME on Friday to discuss the MONEY rate. NAME
phone number is PHONE_NUMBER.”

28. Improving the output produced by Word2Vec
word2VecModel.findSynonyms(“cost”, 5)
MONEY
price
license
nominal
budget

29. Interleaving ngrams with unigrams
interleaveNgrams("hello my name is sammy", 2)
produces:
“hello hello^my my my^name name name^is is is^sammy sammy”

30. Improving the output produced by Word2Vec
word2VecModel.findSynonyms(“cost”, 5)
MONEY^per^month
price^of
license
month^to^month
price

31. Generating Feature Vectors

32. Latent Dirichlet Allocation (LDA)
takes a collection of text documents and seeks to group them by topic
NORTH KOREA MISSILE LAUNCH TESTS
TRUMP’S CHINA OUTREACH 50% — North Korea
30% — National Security
10% — China
10% — Trump

33. LDA
• Cannot (well, very hard to) select topics you want to identify in
advance
• Can’t know what each topic is
Instead, include the entire topic distribution in the feature vector

34. Improving the topics identified by LDA
INTRO
SIGNATURE
CONFIDENTIALITY NOTICE
REPLY CHAIN
BODY
Hey Alexis,
Let’s meet with Ascander on Friday to discuss
the $10,000/year rate. Ascander’s phone
number is (123) 456-7890.
Thanks,
Noah Bergman
Engineer at Salesforce
(123) 456-7890
The contents of this email and any attachments
are confidential and are intended solely for
addressee…
From: Alexis alexis@salesforce.com
Date: April 1, 2017
Subject: Important Document
Noah, how much does your product cost?

35. Improving the topics identified by LDA
• Common information blends topics together
• Reply chains add topics and oversample
In the past, we’ve identified “Sent from my
iPhone” as a topic!
BODY
INTRO
SIGNATURE
CONFIDENTIALITY NOTICE
REPLY CHAIN

36. Upcoming improvements
•Use labeled data to generate high-recall filter
• Investigate alternative methods of computing ngram word vectors
• Factor in user feedback

37. Some lessons learned
• High-recall filter
• Normalizing tokens
• Interleaving ngrams with unigrams
• Extracting bodies
• Filtering out reply chains