This document summarizes Matthias Gallé's presentation on human-centric machine learning at the 2017 Rakuten Technology Conference. It discusses supervised learning where the goal is formalized and large labeled datasets are available, unsupervised learning where patterns are discovered in unlabeled data, and interactive learning which allows users to provide feedback to guide the machine learning model. Specific applications mentioned include machine translation, exploratory search, active learning, and removing the burden of labeling from humans.

1. © 2017 NAVER LABS. All rights reserved.
Matthias Gallé
Naver Labs Europe
@mgalle
Human-Centric Machine Learning
Rakuten Technology Conference 2017

2. Advanced
Chess

3. Supervised Learning
Where f typically such that
𝑓 = argmin 𝑓∈𝐹
1
𝑁
෍
𝑖=1
𝐿 𝑓 𝑥𝑖 , 𝑦𝑖 + 𝜆𝑅 𝑓
I know what I want
(and can formalize it)
I have time & money to label lots of data
X,Y f(x)

4. Example: Machine Translation
Given a text s and its proposed translation p, how to measure its distance with
respect to a reference translation t ?
BLEU: n-gram overlap between t and p
typically: 1 ≤ 𝑛 ≤ 4, precision only, brevity penalty
METEOR
bonus points for matching stems and synonyms
use paraphrases

5. Statistical Machine Translation
P Koehn
(www.statmt.org/book/slides/08-
evaluation.pdf)

6. Consequences of not formalizing correctly
Users do not use your model
Computer-Assisted Translation used rule-based systems for years
Ad-hoc solutions
Quality Prediction
Automatic Post Edition

7. Unsupervised Learning
Where Z(X) capture some prior:
• Compression
• Clustering
• Coverage
• ….
I am not sure what I want
I have a (big) corpus with assumed patterns
X Z(X)

8. Example: Exploratory Search
Whenever your task is:
• Ill-defined:
– Broad / under-specified
– Multi-faceted
• Dynamic:
– Searcher’s understanding inadequate at the beginning
– Searcher’s understanding evolves as results are gradually retrieved.
The answer to what you search is “I know it when I see it”

9. https://en.wikipedia.org/wiki/I_know_it_when_I_see_it

10. Interactive Learning

11. Exploratory Search: examples
E-Discovery
Sensitivity Review
• Vo, Ngoc Phuoc An, et al. "DISCO: A System Leveraging Semantic Search in Document Review." COLING (Demos). 2016.
• Privault, Caroline, et al. "A new tangible user interface for machine learning document review." Artificial Intelligence and Law 18.4 (2010): 459-479.
• Ferrero, Germán, Audi Primadhanty, and Ariadna Quattoni. "InToEventS: An Interactive Toolkit for Discovering and Building Event Schemas." EACL 2017 (2017): 104.

12. Example: Active Learning
Give initiative to the algorithm
allow action of type: “please, label instance x”
Cognitive effort of labeling a document 3-5x higher than labelling a word [1]
Feature labelling:
• type(feedback) ≠ type(label)
• information load of a word label is small
• word sense disambiguation
[1] Raghavan, Hema, Omid Madani, and Rosie Jones. "Active learning with feedback on features and instances." Journal of
Machine Learning Research7.Aug (2006): 1655-1686.

13. Conclusion
If you really want to solve a problem, don’t be prisoner of your
performance indicator
Ask yourself:
1. Does it really capture success?
does it align with human judgment?
2. What does the [machine | human] best?
3. Can you remove the burden from humans by smarter algorithms?

14. Further reading & Acknowledgments
Jean-Michel RendersMarc Dymetman Ariadna Quattoni
http://www.europe.naverlabs.com/Blog

15. Q&A
© 2017 NAVER LABS. All rights reserved.

16. Appendix
© 2017 NAVER LABS. All rights reserved.

17. Statistical Machine Translation
P Koehn
(www.statmt.org/book/slides/08-
evaluation.pdf)