The document discusses weak supervision, which uses unreliable labels to train machine learning models. Weak supervision works by creating many labeling functions that assign probabilistic labels to data, rather than definitive labels. These functions can be rules, user reviews, model predictions, and other heuristics. A generative model then learns the accuracy of labeling functions to determine the true labels. This technique can achieve good results with only a small number of true labels. However, it is difficult to evaluate and influence the importance of labeling functions without labels. The document promotes creating many diverse labeling functions to take advantage of weak supervision.

1. Weak Supervision
Making use of unreliable labels
Eddie Bell
twitter.com/ejlbell

2. Who am I?

3. Fraud

4. I probably don’t need to talk about
labels to a room of data scientists

5. Ravelin has some labels of good quality
and many labels of unknown quality

6. High quality labels Low quality labels
Unsupervised ❌ ❌
Supervised ✅ ❌
Semi-supervised ✅ ❓
Weakly-supervised ✅ ✅

7. Supervised Weakly-supervisedSemi-supervised

9. How does it work?

11. Labelling functions

12. Labelling functions
● rules
● reviews from humans
● graph methods
● similarity methods
● perturbations / transformations
● model predictions

13. Probabilistic Labels

14. Generative model
Learning the Structure of Generative Models without Labeled Data
- Bach et al.

15. Discriminative model
As long as loss supports probabilistic labels

16. Is it any good?

17. We got an additional 2% recall with
10’s of labelling functions

18. But how do you choose labelling functions
without labels to evaluate them?

19. No way to influence label function importance

20. ... or maybe there is?
https://hazyresearch.github.io/snorkel/blog/superglue.html
“In many datasets, especially in real-world applications, there are
subsets of the data that our model underperforms on, or that we care
more about performing well on than others”
“... expert-heads are used to learn slice-specific representations. Then, an
attention mechanism is learned over expert heads to determine when and
how to combine these representations.”
yay!

21. Labelling function
instance density

22. In the low-density setting, sparsity of labels will mean that
there is limited room for even an optimal weighting of the
labeling functions to diverge much from the majority vote.
Conversely, as the label density grows, the majority vote will
eventually be optimal. It is the middle-density regime where
we expect to most benefit from applying the generative model.

23. “These labeling functions do not
need to be precise, and we should
write many of them.”

24. “These labeling functions do not
need to be precise, and we should
write many of them.”

25. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀

26. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀
C
urrentresearch

27. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀
C
urrentresearch
Stream-line your labelling

28. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀
C
urrentresearch
Stream-line your labelling
Sea what it is all about

29. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀
C
urrentresearch
Stream-line your labelling
Sea what it is all about
Betta e-fish-in-sea

30. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀
C
urrentresearch
Stream-line your labelling
Sea what it is all about
Betta e-fish-in-sea
Scale-ableoppo-tuna-ty

31. https://hazyresearch.github.io/snorkel/
🏊 Dive deep into snorkel
🏊‍♀
C
urrentresearch
Stream-line your labelling
Sea what it is all about
Betta e-fish-in-sea
Scale-ableoppo-tuna-ty
The results shore are unbebreathable

32. You shoald at least breefly
mullet over despite its
fishues and crab-e-ats

33. Fin
Fin

34. Thankyou
www.ravelin.com
p.s. we’re hiring data scientists, analysts and a director