No Training Data? No Problem! Weak Supervision to the Rescue! A talk on NLP Weak Supervision at the Singapore Quantum Black Meetup. This talk talks about 1. ML's insatiable need for large datasets 2. Contemporary ML leaving out domain knowledge from Subject Matter Experts 3. How Weak Supervision, an approach of Data-Centric AI, solves both the problems simultaneously by encoding domain subject matter expertise into programmatic labeling functions. 4. The WRENCH benchmark to compare various weak supervision algorithms on several standard datasets. 5. Snorkel to combine the various labeling functions. 6. COSINE to fine-tune a final transformer based model that overcomes the noise in weak labels 7. Future Directions and Resources Feel free to use the slides but please remember to credit me with a link to my Linkedin profile: www.linkedin.com/in/marie-stephen-leo.

1. No Training Data? No Problem!
Weak Supervision to the Rescue!
Marie Stephen Leo
Based on the Medium post of the same title

2. About Me
Director of Data Science @ Edelman DxI
(World’s largest Public Relations agency)
Part time Data Science Instructor @ General Assembly
✍ Top Writer in Artificial Intelligence @ Medium
🔬 Research Interests:
📝 NLP
🔎 Neural Search
⚙ MLOps
https://medium.com/@stephen-leo
www.linkedin.com/in/marie-stephen-leo
Marie Stephen Leo

3. 📝 Agenda
🚧 The challenge of contemporary Machine Learning
💡 Enter Weak Supervision!
🧰 Weak Supervision Frameworks
🏗 Conclusion & Future Direction

4. 🚧 The challenge of contemporary Machine Learning
● ML requires substantial amounts of manually labeled training data
○ ImageNet contains 14 Million manually annotated images!
● Transfer Learning improves this situation
○ But most models still require few hundreds to thousands of high quality labels to
finetune models such as BERT!
○ For eg., to build a sentiment analysis model, someone should first manually read a
few thousand comments and mention what’s the sentiment of each comment!
● Labeling data is
○ 💰 Costly
○ ⏱ Time Consuming
○ 🏋 Labor Intensive
○ 💢 Prone to human errors and biases
○ 🤷 Not the priority for subject matter experts in the business

5. 🚧 The challenge of contemporary Machine Learning
● At the same time, unlabeled data is vastly abundant!
● Most organizations have an immense depth of domain
knowledge in boolean queries, heuristic rules, or tribal
knowledge that don't get used in ML models.
● What If?
○ Can we leverage vast stores of domain knowledge in our
organisations to solve the labeling problem?
○ Can we label all the unlabeled data programmatically?
○ This would result in ML algorithms learning from domain
subject matter experts rather than some poor intern
labelers with vastly more data than manual labels could
ever collect!

6. Enter Weak Supervision!

7. 💡 Weak Supervision: The shift to Data Centric AI
Data-centric AI is the discipline of systematically engineering the data used to build an AI system -
Andrew Ng (https://datacentricai.org/)
Model Centric AI (The 2010s)
Training Data:
Fixed
Model:
Iterate
+/- 1% accuracy change
Data Centric AI (2020s)
Training Data:
Iterate
+/- 10% accuracy change
Model:
Fixed

8. 💡 Weak Supervision: 💰 A Billion Dollar Industry!

9. 💡 Weak Supervision in one picture - Enabling Data Centric AI!
Reduce the efforts of manual labeling while unlocking the vast knowledge of domain subject matter
experts (SMEs) by leveraging a diversity of weaker, often programmatic supervision sources.

10. 💡 Weak Supervision details
1⃣ Writing Labeling Functions (LFs)
2⃣ Combining them with Label Model (LM)
3⃣ Training a Downstream End Model (EM)
4⃣ Iterate!
During Training, Weak Supervision in general
has 4 steps
Training Data!
During inference, we discard everything and
only use the EM directly to make predictions!
Hence no different from normal ML.

11. 💡 Weak Supervision details - 1⃣ Labeling Functions (LF)
● Any Python function that takes in one datapoint as input and
returns either one label as output or abstains.
● Can be anything! Keywords, heuristics, search queries, outputs of
other models (eg. Zero shot), labels from interns, etc.
● Use the Snorkel python library [https://github.com/snorkel-team/snorkel]
● Not expected to be perfect! The next steps will denoise them.

12. 💡 Weak Supervision details - 2⃣ Label Model (LM)
● If we have n LFs, then each row will get max n labels (LFs can
abstain if they are not sure).
● We need to aggregate the outputs of the n individual LFs so that
each row only has one label.
● Majority Vote is the simplest Label Model.
● There are better ways! We can use the agreements and
disagreements between the various LFs with some matrix math.
○ Does not need any ground truth data at all! [Data Programming
Paper] [MeTaL Paper] [Flying Squid paper] [Poster] [Talk]
○ In practice having a small labeled validation set (~100 rows)
helps to convince yourself (and your boss!) that you’re doing
the correct thing.

13. 💡 Weak Supervision details - 3⃣ End Model (EM)
● The output of the Label Model (LM) is one Weak Label for each row
generated by combining all the weak LFs.
● Use these weak labels as the training data to fine tune a downstream
pre-trained model to generalize beyond the weak labels.
○ Large pre-trained models such as BERT already have
tremendous understanding of our language.
○ Fine Tuning BERT on weak labels is sufficient for it to learn the
task, even beyond the weak labels.
● Since the LFs are programmatic labeling sources, we can run the LFs
and LM on our entire unlabeled corpus to generate many labels.
● The EM benefits from the more extensive training datasets created
and incorporates the domain knowledge of SMEs! Win - Win!
Training Data!

14. 💡 Weak Supervision details - Inference Time
Data Prediction
● But don’t throw away your LFs and LM just yet!
● You can reuse them for model retraining at a regular cadence or model
monitoring for performance degradation over time.
● LF creation work is a one time effort vs labeling every time your model drifts!

15. Weak Supervision Frameworks

16. 🧰 Weak Supervision Frameworks - 🔧 WRENCH
[WRENCH Paper] [Github]

17. 🧰 Weak Supervision Frameworks - 🔧 WRENCH
Despite not using any labeled data to train,
Weak Supervised models with appropriate LFs
can achieve performance that’s close to fully
supervised models on many tasks!
[WRENCH Paper] [Github]

18. 🧰 Weak Supervision Frameworks - Snorkel
[Data Programming (DP) Paper] [MeTaL Paper] [Github] [Poster]
A matrix completion problem that is solved with SGD [Talk]

19. 🧰 Weak Supervision Frameworks - 📐 COSINE
[COSINE Paper] [Github] [WRENCH Implementation]
COSINE is short for COntrastive Self-training for fINE-Tuning Pretrained Language Model
Initialization
Sample
Reweighting
Classification Loss
on high confidence
samples
Contrastive Loss
on high confidence
samples
Confidence
regularization on
all samples

20. 🧰 Weak Supervision Frameworks - 🔎 Heuristic LF selection
● In real world testing, accuracy can vary a lot depending on quality of LFs selected.
● Our solution is to use a small hand labeled validation dataset or iterative active learning to
choose the best LFs from an LF Zoo.
● Highly iterative process, can start with a small number of LFs and refine them over time. The
analysis could also expose gaps in our understanding of the problem domain!

21. Conclusion & Future Direction

22. 🏗 Conclusion
● Shift to Data Centric AI
● Weak Supervision for programmatic data labeling
○ 1⃣ Writing Labeling Functions (LFs)
○ 2⃣ Combining them with Label Model (LM)
○ 3⃣ Training a Downstream End Model (EM)
○ 4⃣ Iterate!
● Weak Supervision frameworks
○ 🔧 WRENCH
○ Snorkel
○ 📐 COSINE
○ 🔎 Heuristic LF selection

23. 🏗 Future Direction
● More research into augmenting domain knowledge LFs
with automated LFs
○ Want To Reduce Labeling Cost? GPT-3 Can Help [Paper] [Github]
○ X-Class: Text Classification with Extremely Weak Supervision [Paper]
[Github]
○ 󰐵 OptimSeed: Seed Word Selection for Weakly-Supervised Text
Classification with Unsupervised Error Estimation: [Paper] [Github]
● The Rise of UI based tools since Weak Supervision relies
heavily on SMEs who may not be coding experts!
○ 🌟 Open Source: Rubrix
○ 💰 Commercial: Snorkel Flow ($1Billion at work!)

24. 📚 Resources
● Medium Post that this talk is based on: Link
● Snorkel Tutorials: Snorkel Website
● Collection of resources on Data Centric AI: Link
● Cool Icons: Flaticon
● Papers: Arxiv
● O’Reilly Book: Link

25. Questions?