How to build a multi-lingual text classifier ? In this talk, we will introduce one of the biggest challenge we face at dailymotion : how do we accurately categorize our video catalog at scale using the descriptions ? The purpose is to introduce the whole pipeline running at dailymotion which relies on a complex mixing of different methods : machine learning for language detection, NEL to Wikidata knowledge graph, deep learning using sparse representations and NLP with multi-lingual embeddings & robust transfer learning. -> Event : https://www.meetup.com/fr-FR/Paris-NLP/events/lkmbhqyzkbgc/

1. How to build a multi-lingual text
classifier ?
July 24, 2019 Axel de Romblay

2. © 2 0 1 9 CO N FI D E N TI A L
Motivation
W HAT FO R ?
Let’s introduce some important
figures, goals & KPIs

3. © 2 0 1 9 CO N FI D E N TI A L
A multi-lingual
video catalog
Dailymotion hosts hundreds of
millions videos in more than 20
languages. Our purpose is to share
the most compelling music,
entertainment, news and sports
content around.
C HAPTE R
3

4. © 2 0 1 9 CO N FI D E N TI A L
Content
categorization for
a better user
experience
Why do we care at dailymotion about
being able to accurately categorize
content at scale?
• Watching interface
• Search engine
• SEO & acquisition
C HAPTE R
4

5. 3
21
4
© 2 0 1 9 CO N FI D E N TI A L
High precision/coverage
tradeoff
Tag the maximum videos with a
minimum error rate
Fast & up-to-date
annotation
Get updated topics/categories
Relevance & quality
Get relevant & meaningful
topics/categories
Multi-lingual annotation
Tag videos for all the languages
C HAPTE R
5
Criteria for a good classification

6. © 2 0 1 9 CO N FI D E N TI A L
First steps
W HAT ARE T HE REQ UIRE M E NTS ?
Let’s introduce the settings:
language detection, video
annotation using NEL &
unsupervised categorization of
topics.

7. Topic annotation
for English &
French videos
Reference: https://medium.com/dailymotion/topic-
annotation-automatic-algorithms-data-377079d27936

8. © 2 0 1 9 CO N FI D E N TI A L
Topic annotation pipeline
C HAPTE R
8

9. © 2 0 1 9 CO N FI D E N TI A L
• Polyglot python package (based on cld2)
• Naïve Bayesian classifier trained on
millions web pages for each language.
• Optimized to run at scale.
• Supports 196 languages
C HAPTE R
9
Text extraction & language detection
How to detect the language ?

10. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
1 0
Topic generator : Named Entity Linking
with Wikidata knowledge graph
Open source knowledge graph
Multilingual
Updated
50M interconnected entities

11. © 2 0 1 9 CO N FI D E N TI A L
Preprocessing
• Standard
preprocessing : stop
words …
• Tokenization :
detection of
overlapping words.
C HAPTE R
1 1
Topic generator : Named Entity Linking
with Wikidata knowledge graph
Disambiguation
Given a word a, we choose the appropriate
Wikidata entity pa using :
• The commonness of pa : Pr(pa|a)
• A relatedness score between a and pa :
Pruning
We keep relevant wikidata entities using :
• The link probability of a word
• The coherence of the word :

12. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
1 2
Topic filter : feature engineering &
centrality classification problem
Training Set
Candidate topics
Features : coherence, popularity,
disambiguation score, location
score …
Machine Learning

13. Topic
categorization

14. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
1 4
Unsupervised & uncontextual topic
categorization using Wikidata
ØGather the topics: get less classes than topics
ØGet different levels of classification (hierarchy)
§ Set a good number of levels
§ Set good splits for each level (and avoid small & big classes)
ØGet a relevant number of classes for each topic
§ Have at least one class (coverage)
§ Limit the number of classes for each topic
ØGet a good label for each class: match IAB
taxonomy or at least get a wikidata qid
ØCoherence/consistency : similar classes must be
in a same level (ex: countries, teams, ...)
Humans
Not Humans
Football Player
Model
Singer
RM
Cristiano
Ronaldo
Rap
Rock
Count : 20 topics
Count : 11+2+2=15 classes
What criteria for a good categorization ?

15. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
1 5
Preprocessing: selecting a Wikidata subgraph
with product rules
Wikidata
44M topics (vertices) & 4k relations
(types of edges)
Our subgraph
500k topics & 10 relations

16. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
1 6
From a graph of connected classes to
a hierarchy of classes
Compute a granularity
measure G
It will perform a ranking between all the classes
and will allow us to split the classes into
different levels depending on a measure G :
G (c) = number of leaves having a path to the
class c
Compute a correlation
measure C
It will detect and drop correlated classes: this will
allow us to drastically reduce the number of classes
per topic.
C (c1, c2) =

17. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
1 7
A concrete example…

18. © 2 0 1 9 CO N FI D E N TI A L
Multi-lingual
text classifier
HOW W E D O THIS ?
Now we can move on to NLP
algorithms J

19. © 2 0 1 9 CO N FI D E N TI A L
What we want to do
Get a contextual
categorization of our
video catalog
… For all languages !
How we do this
1. Get a robust representation of our video
catalog (multi-lingual embeddings)
2. Train a predictive model on the top of it,
on French & English videos only !
3. Transfer on other languages
4. Evaluate performances
C HAPTE R
1 9
What we have
57% of our video
catalog (French +
English) is
annotated &
categorized into
different levels.
The categorization
is uncontextual (it
only depends on
the topic and not
on the video)
Where do we stand

20. Multi-label
classification

21. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 1
Get a contextual classification for French
& English videos with sparse inputs

22. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 2
Get a contextual classification for French
& English videos with sparse inputs
ü Fast & accurate model
§BOW trained using DataFlow
§Top1-accuracy = 0.9
ü Low memory usage : sparse model
implemented using tf.keras
Pros Cons
Reference: https://medium.com/dailymotion/how-to-design-deep-learning-models-with-sparse-inputs-in-tensorflow-keras-fd5e754abec1
q Not transferable on others languages
§Bow vocabulary for French/English only

23. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 3
Robust multi-lingual embeddings with
BERT using a Tesla V100
Reference: https://github.com/google-research/bert

24. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 4

25. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 5
Fine-tuning on French/English videos by
adding prediction layers

26. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 6
Some qualitative results on other
languages
Ø We display the most confident
predictions on other languages (test
set)
Ø To decrease the number of False
Positive, we can set a threshold to
get 85% precision and deduce the
recall
Ø But how can we make sure that this
threshold is the same on the test
set ?

27. © 2 0 1 9 CO N FI D E N TI A L
The next
step(s)
CO NC LUS IO N
Let’s conclude with the next steps

28. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 8
First find a quantitative metric for
transfer learning.
Two biases:
• Content probably depends on the
language (Korean videos tend to
display more news, English videos
more sports...)
• BERT is supposed to align multi-lingual
embeddings.
Some ideas:
• Translate some videos into English using
Google Translation API
• Apply BOW model to get a groundtruth on
other languages

29. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
2 9
Then tune hyperparameters using state-
of-the-art optimization (BOHB)
Reference:
https://www.automl.org/blog_bohb/

30. © 2 0 1 9 CO N FI D E N TI A L
C HAPTE R
3 0
Push to production environment.
qCode on Github (SQL / Python / TensorFlow /
DataFlow) & dump models / tables (Google Cloud).
qCheck CI passes: run unit tests, style checks, code
reviews…
qBuild docker image and push to quay repository
qDeploy (Kubernetes)
qSchedule the tasks (Airflow)
qMonitor (Datadog & Tableau)

31. Thanks to my
squad
&
Thank you !
Contact: https://www.linkedin.com/in/axel-de-romblay-6444a990/