Handling problem of hand-labeled training data with data programming and weak supervision Presentation from Confitura 2019 presented by Rafał Wojdan from Sotrender

1. Handling problem of hand-labeled
training data with data programming
and weak supervision
Rafał Wojdan,
Confitura 2019

2. whoami
2
Rafał Wojdan
Senior Machine Learning Engineer @ Sotrender
Trainer @ Sages r.wojdan@sages.com.pl

3. Bottleneck
https://hazyresearch.github.io/sno
rkel/blog/weak_supervision.html
https://blog.easysol.net/building-ai-applications/
Annotated
data

4. Bottleneck
https://hazyresearch.github.io/sno
rkel/blog/weak_supervision.html
https://blog.easysol.net/building-ai-applications/
Annotated
data
MNIST, Imagenet,
Sentiment104
(Twitter)

5. Traditional hand labelling

6. Dealing with lack of annotated data
http://ai.stanford.edu/blog/weak-supervision/

7. Dealing with lack of annotated data
http://ai.stanford.edu/blog/weak-supervision/

8. Active learning
http://burrsettles.com/pub/settles.activelearning.pdf

9. Semi-supervised learning
https://hal.inria.fr/hal-01159171/document

10. Transfer learning
Huge data
Huge model
Data
representation
Objective
Smaller
data
Data
representation
Classifier
Objective

11. Transfer learning - Computer Vision
Huge data
Data
representation
Objective
Photos of
cars
Data
representation
Classifier
Car brand
classification
ResNet - 50

12. Transfer learning - NLP
Huge data
Data
representation
Objective
Customer
reviews
Data
representation
Classifier
Sentiment
analysis

13. 1. Data programming instead of hand labelling
2. Functions instead of labelling guidelines
3. Multiple sources of labels instead one ground truth
Weak supervision
https://arxiv.org/pdf/1711.10160.pdf

14. History
http://ai.stanford.edu/blog/weak-supervision/

15. History
http://ai.stanford.edu/blog/weak-supervision/
Flexibility Control

16. Weak supervision - sources
https://arxiv.org/pdf/1812.00417.pdf
https://ai.googleblog.com/2019/03/harnessing-organizational-k
nowledge-for.html
https://arxiv.org/pdf/1708.00524.pdf

17. Labeling functions vs Hand labelling
http://ai.stanford.edu/blog/weak-supervision/
● Much faster
● More flexible
● High coverage/Scalability
BUT...
● Less accurate
● Overlapping/Correlated
● Conflicting
● Noisy

18. Magic comes in: Generative model
http://cs231n.stanford.edu/slides/2018/cs231n_20
18_ds07.pdf
Solution:
● Generative model: P(L, Y) = P(L | Y)P(Y)
● Algorithm: asynchronous Gibbs sampling
Goal:
● Denosing labels
● Modelling accuracies and
correlations
Result:
The generative model -> re-weighted combination of
the labeling functions.

19. Majority voting vs Generative model
https://arxiv.org/pdf/1711.10160.pdf

20. Majority voting vs Generative model
https://arxiv.org/pdf/1711.10160.pdf
Condorcet’s Jury
Theorem
Not enough
conflicts

21. Generative vs Discriminative - reminder
http://maximustann.github.io/mach/2015/08/11/generative-vs-discriminative-models/

22. Generative and Discriminative
http://ai.stanford.edu/blog/weak-supervision/

23. Noisy labels & noise-aware loss function
Binary case
Labels
Noisy labels
(Generative
model outputs)
Logistic regression
predictions Cross Entropy Noise-aware Cross Entropy
1 [0.9, 0.1] [0.85, 0.15] -(1*log(0.85)) -(0.9*log(0.85) + 0.1*log(0.15))
0 [0.2, 0.8] [0.4, 0.6] -(1*log(0.6)) -(0.2*log(0.4) + 0.8*log(0.6))

24. ● Scalability
● Quick training data preparation for big complex models
● Generalization beyond labelling functions
● Simple domain expertise leverage
Benefits of weak supervision system

25. Wrap up - Snorkel schema
https://arxiv.org/pdf/1711.10160.pdf

26. Data split:
Topic and Product classification examples
https://arxiv.org/pdf/1812.00417.pdf
Generative & Discriminative models gains of training on dev hand-labeled set

27. Majority voting vs Generative model
Topic and Product classification examples
https://arxiv.org/pdf/1812.00417.pdf

28. Benefits from non-serverable features
(real-time event classification)
Topic and Product classification examples
https://arxiv.org/pdf/1812.00417.pdf
Hand-labelling vs weak supervision

29. Weak supervision on images -
cross modal approach
https://arxiv.org/pdf/1903.11101.pdf

30. Weak supervision on images -
cross modal approach
https://arxiv.org/pdf/1903.11101.pdf

31. Weak supervision on images -
cross modal approach
https://arxiv.org/pdf/1903.11101.pdf

32. Weak supervision on images with domain
specific-primitives (DSP)
https://dawn.cs.stanford.edu/2017/09/14/coral/

33. Weak supervision on images with domain
specific-primitives (DSP) - functions
https://dawn.cs.stanford.edu/2017/09/14/coral/

34. Weak supervision on images with domain
specific-primitives (DSP)
https://dawn.cs.stanford.edu/2017/09/14/coral/

35. Thanks for coming!
Questions?