1.
© 2017 NAVER LABS. All rights reserved.
Matthias Gallé
Naver Labs Europe
@mgalle
Human-Centric Machine Learning
Rakuten Technology Conference 2017

2.
Advanced
Chess

3.
Supervised Learning
Where f typically such that
𝑓 = argmin 𝑓∈𝐹
1
𝑁
෍
𝑖=1
𝐿 𝑓 𝑥𝑖 , 𝑦𝑖 + 𝜆𝑅 𝑓
I know what I want
(and can formalize it)
I have time & money to label lots of data
X,Y f(x)

4.
Example: Machine Translation
Given a text s and its proposed translation p, how to measure its distance with
respect to a reference translation t ?
BLEU: n-gram overlap between t and p
typically: 1 ≤ 𝑛 ≤ 4, precision only, brevity penalty
METEOR
bonus points for matching stems and synonyms
use paraphrases

5.
Statistical Machine Translation
P Koehn
(www.statmt.org/book/slides/08-
evaluation.pdf)

6.
Consequences of not formalizing correctly
Users do not use your model
Computer-Assisted Translation used rule-based systems for years
Ad-hoc solutions
Quality Prediction
Automatic Post Edition

7.
Unsupervised Learning
Where Z(X) capture some prior:
• Compression
• Clustering
• Coverage
• ….
I am not sure what I want
I have a (big) corpus with assumed patterns
X Z(X)

8.
Example: Exploratory Search
Whenever your task is:
• Ill-defined:
– Broad / under-specified
– Multi-faceted
• Dynamic:
– Searcher’s understanding inadequate at the beginning
– Searcher’s understanding evolves as results are gradually retrieved.
The answer to what you search is “I know it when I see it”

9.
https://en.wikipedia.org/wiki/I_know_it_when_I_see_it

10.
Interactive Learning

11.
Exploratory Search: examples
E-Discovery
Sensitivity Review
• Vo, Ngoc Phuoc An, et al. "DISCO: A System Leveraging Semantic Search in Document Review." COLING (Demos). 2016.
• Privault, Caroline, et al. "A new tangible user interface for machine learning document review." Artificial Intelligence and Law 18.4 (2010): 459-479.
• Ferrero, Germán, Audi Primadhanty, and Ariadna Quattoni. "InToEventS: An Interactive Toolkit for Discovering and Building Event Schemas." EACL 2017 (2017): 104.

12.
Example: Active Learning
Give initiative to the algorithm
allow action of type: “please, label instance x”
Cognitive effort of labeling a document 3-5x higher than labelling a word [1]
Feature labelling:
• type(feedback) ≠ type(label)
• information load of a word label is small
• word sense disambiguation
[1] Raghavan, Hema, Omid Madani, and Rosie Jones. "Active learning with feedback on features and instances." Journal of
Machine Learning Research7.Aug (2006): 1655-1686.

13.
Conclusion
If you really want to solve a problem, don’t be prisoner of your
performance indicator
Ask yourself:
1. Does it really capture success?
does it align with human judgment?
2. What does the [machine | human] best?
3. Can you remove the burden from humans by smarter algorithms?

14.
Further reading & Acknowledgments
Jean-Michel RendersMarc Dymetman Ariadna Quattoni
http://www.europe.naverlabs.com/Blog

15.
Q&A
© 2017 NAVER LABS. All rights reserved.

16.
Appendix
© 2017 NAVER LABS. All rights reserved.

17.
Statistical Machine Translation
P Koehn
(www.statmt.org/book/slides/08-
evaluation.pdf)