In this talk I am proposing the technique of combining human input with data programing and weak supervision to create a high quality model that evolves with feedback. We apply dark data extraction method: snorkel, developed at Stanford (https://hazyresearch.github.io/snorkel/) to create an honor code violation detector (HCVD). Snorkel is a framework that uses inputs from SME’s and business partners and converts them into heuristic noisy rules. It combines the rules using a generative model to determine high and low quality rules and outputs a high accuracy training data based on combined rules. HCVD detects key phrases (example: do my online quiz) that indicate honor code violation. We run this model daily and place the HCVD texts (around 2%) in front of humans, the feedback from the humans is periodically checked and the rules are edited to change the weak supervision to produce a fresh training set for modeling. This is an ongoing and iterative process that uses interactive machine learning to evolve the Natural Language Comprehension model as new data gets collected.

1. Confidential Material – Chegg Inc. © 2005 - 2016. All Rights Reserved.© 2005 – 2017 by Chegg Inc. All Rights Reserved. 1
Natural Language Comprehension: Human Machine Collaboration.
Sanghamitra Deb, Data Scientist
Gabriela Brown, Summer Intern

2. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.2
Chegg

3. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.3
What is Chegg Tutors?

4. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.4
Unstructured data in Business

5. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.5
Dark Data at Chegg
Chats between tutors and students
Chegg Study Q&A

6. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.6
Bringing light to Dark Data
DeepDive and snorkel processes such documents from
public and dark web to extract evidential data, such as
names, addresses, phone numbers, job types, job
requirements, information about rates of service, etc.
Wikipedia extractions
Detecting Online Sex Trafficking
Professor Chris Re

7. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.7
Student looking for tutors
I need a 10 page essay
written on the
deforestation of the
amazon rainforest. must
have 7 resources.

8. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.8
Students intents: Fraud
• Do my homework
• Take online quiz for me
• Do my scheduled take home exam
Universities typically have strict honor policies, stating that your homework,
exams, take home etc should be completed by the student without any
external help.
A small number of students come to platform to get their homework done
or ask someone to take their exam for them. This is a strict violation of
honor code.
Examples

9. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.9
Typical NLP Machine Learning Flow
High Performing Machine Learning Models
could require 100,000 labelled data !!

10. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.10
Traditional Feature Engineering
Winning solution!!
Feature Engineering

11. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.11
Generating a training set
• Human reading and labeling
• Several hundreds of expert hours
• Difficult to scale with evolving
business questions

12. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.12
The snorkel pipeline

13. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.13
Human Machine Collaboration
Knowledge
transfer
What is
important to
product and
business
Language,
business needs
and teams
evolve.
Data Scientist
Product/Businesss
SME
Iterate
Knowledge
transfer
What is
important to
product and
business
SME
Data Scientist
Product/Businesss
• Create Filters
• Create rules
• Redefine Filters
• Redefine rulesReplaces manual generation of labelled data

14. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.14
Automated Features

15. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.15
Creating Filters: Candidate Extraction

16. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.16
Creating Filters: Candidate Extraction

17. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.17
Observing the candidates
Humans/SME’s look at ~100-200 of them and label them.

18. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.18
Creating Rules
v
I will pay someone to write my essay.
Reference to the tutor + verb followed by “my”
This is an honor code violation✓

19. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.19
What do the rule functions look like?
Several tens of rules create the training set
The rules are judged based on the labels provided
By humans or SME’s

20. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.20
Developing the training set: one rule
0
20
40
60
80
100
120
140
160
180
200
1 0 -1
Training set 1: Class 1
0: unlabelled data
-1: Class 2

21. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.21
Developing the training set: four rules
0
20
40
60
80
100
120
140
160
180
200
1 0 -1
Training set 1: Class 1
0: unlabelled data
-1: Class 2

22. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.22
Developing the training set: eight rules
0
20
40
60
80
100
120
140
160
180
200
1 0 -1
Training set 1: Class 1
0: unlabelled data
-1: Class 2

23. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.23
Developing the training set: one rule
0
20
40
60
80
100
120
140
160
180
200
1 0 -1
Training set 1: Class 1
0: unlabelled data
-1: Class 2

24. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.24
Developing the training set: twenty rules
0
20
40
60
80
100
120
140
160
180
200
1 0 -1
Training set 1: Class 1
0: unlabelled data
-1: Class 2

25. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.25
Performance
Evaluation Metrics
Positive accuracy 68.3%
Negative accuracy 90.7%
Precision 71.8%
Recall 68.3%

26. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.26
Production: Iterations
• Snorkel codes run on the opportunities sent the day before, humans check the list and update a file with
real honor code violations.
• After doing unsupervised learning (topic modeling, word2vec) on the positive and negative HCV’s from
human generated data the rules are changed to improve positive accuracy.
In dynamic two sided
market places language
and behavior changes
continuously, hence
having iterations every
3-4 months keeps the
model fresh

27. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.27
Generalization: Matching Problem for Chegg Tutors
• Feature Generation for student tutor pairs.
• Chegg tutors is a two sided market place with students and tutors
being paired based on their overlapping characteristics.
• Generating features is an important part of creating this
recommendation system. Snorkel helps generate key phrases
associated with student-tutor pairs.

28. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.28
Behind training set generation: PGM’s
https://arxiv.org/pdf/1605.07723.pdf
Model the rules as
independent similar to
Naïve Bayes
Consider interdependencies between the rules.
Similar fix reinforce
exclude

29. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.29
Noisy sources of truth
credit:
https://hazyresearch.github.io/snorkel/

30. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.30
Generalization

31. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.31
Thank you
sdeb@chegg.com
@sangha_deb

32. Example Slide
Chegg Inc. © 2005 – 2017. All Rights Reserved.32
Stanford NLP & Tools
Intent of Honor Code ViolationOpportunities
• Are students and tutors having
classes offsite?
• Are tutors comitting fraud?
• Are students/tutors using offensive
language?
• Do students want lessons
immediately or they are willing to
wait? …...
Other business questions
Other datasets: Chat