ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications

Kun Han Xiangang Li Jieping Ye
DiDi AI Labs DiDi AI Labs DiDi AI Labs
Univ. of Michigan,
Ann Arbor
Zang Li
DiDi AI Labs
Kevin Knight
DiDi AI Labs
Univ. of Southern
California

nPart I: Introduction (20min)
nPart II: Natural Language Processing (145min+Break)
l Word representations.
l Sentence representations.
l NLP Benchmarks.
l Multilingual representations. Social bias.
l Text Embedding Applications
l Graph Embedding Techniques and Applications
l Text/Graph Embedding Applications in Customer Service Scenarios
nPart III: Speech (140min+Break)
l Speech recognition: Basic concepts and classic methods
l Speech recognition: Deep learning approaches, end-to-end approaches, and applications
l Multimodal approach: speech + text for emotion recognition
l Conversational understanding: dialogue intent and topic mining
Outline

History of Urban Transportation

Smart Transportation System
Smart Travelers
Smart InfrastructureSmart Vehicles Cloud
Big
Data
Transportation
EngineeringAI

AI
Neural Networks
Machine Learning:
supervised,
unsupervised
Deep Learning
Reinforcement
Learning
NLP & Speech

nNatural Language Processing
lWord representations.
lSentence representations.
lNLP Benchmarks.
lMultilingual representations. Social bias.
lApplications of text and graph-embedding.
nSpeech
lSpeech recognition: Basic concepts and classic methods
lSpeech recognition: Deep learning approaches, end-to-end approaches, and applications
lMultimodal approach: speech + text for emotion recognition
lConversational understanding: dialogue intent and topic mining
Outline

DRIVERS
DiDi
PASSENGERS
DiDi APP
AUTONOMOUS
VEHICLES
Natural Language Processing (NLP) at DiDi

DRIVERS
DiDi
PASSENGERS
DiDi APP
AUTONOMOUS
VEHICLES
Making
communication
effective
Language
Translation
Customer
Service
Where to Go
Where to Stop
Entertainment
Hands-Free
OperationUser
Feedback
Analysis
Natural
Speech
Interaction
Driver care

What NLP
needs do
we have?
What NLP
technologies
are needed?
What data
do we
process?
Customer service In-vehicle interaction
Dialog authoring …
toolkit
Music
recommendation
User feedback dataDialog data
Translation …
toolkit
… entity tagger … emotion taggerdialog speech-act tagger
Customer service …
chatbot
Voice
navigation
User feedback
toolkit
DiDi Businesses
NLP Applications
NLP Toolkits
NLP Algorithms
DiDi Language Data

Machine Translation to Bridge Language Barriers

Driver speaks Japanese
Passenger speaks Chinese
Driver:
DiDi Translate:
Driver:
DiDi Translate:
Passenger:
responds in Chinese…
automatically translated to Japanese

JaàZh SMS translation Relative Score
XXX Translation System 21.7
YYY Translation System 23.5
DiDi Translate (v6n) 20.7
DiDi Translate (v10) 23.5
DiDi Translate (v12) 25.1
DiDi Translate (v12.2) 25.2
DiDi Translate (v12.5) 27.8
DiDi Translate (v14.1.70) 31.9
URLs
URL
pairs
HTML
document pairs
Extracted
text
Sentence
pairs
Filtered
sentence
pairs
Machine
Translation
training
Domain
…
,
…
…
,
…
TRAIN
MACHINE
TRANSLATION
SYSTEM
Japanese Chinese
High-quality
human
translation
examples

Intelligent Customer Service Panorama
Customer Service Brain
AI HI
AI for Customer Service
n Customer Service Volume:
1.2 Million+
n The Proportion of AI Customer Service
75%+
Cost reduction
Enhanced user experience

Driver Care Assistant Intelligent Bot

.., .,
. , . , . . ..,
, . , . , . .
.., .. .
.., , . , . . .
,. . . . . , .
. .
. . ,. , . , . .
. ,.
, . , .
. .,
. , , . . . ,
.
Speech Processing Layout

Voice Interactive
AustraliaJapan China
n Japan & Australia: Accept Orders
n China: Cancel Orders
Voice Interactive

nNature Language Processing
lWord representations.
Count-based vectors, prediction vectors (LSTM, Word2vec), character vectors.
lSentence representations.
Word vectors that vary with context.
Whole-sentence vectors (e.g., Seq2seq, Skip-Thought, ELMO, BERT).
lNLP Benchmarks. GLUE benchmark, etc.
lMultilingual representations. Social bias.
lText Embedding Applications
lGraph Embedding Techniques and Applications
lText/Graph Embedding Applications in Customer Service Scenarios
Natural Language Processing - Outline

Why You Should Care
• You may have to unstructured language data.make predictions based on
manipulate and transform
extract patterns from
negative
negative
positive
positive
negative
…
this movie is super bad à
I liked it à
this movie is not good à
the special effects were awwwwwesome à
a snooze-fest à
input output
…
sample problem
of course, your problems are more interesting … we’ll get to that

There’s Been a Revolution in NLP in the Last Couple of Years
old way
new way
Collect millions of examples & train classifier
negative
negative
positive
positive
negative
positive
…
this movie is super bad à
I liked it à
a snooze-fest à
pretty decent date movie à
input output
…
Apply classifier to new examples
pretty decent date movie
PRE-TRAINED MODEL “knows English”
little classifier
a few training
examples
positive/negative
prediction
positivity
(or any sentence!)
beauty
entertainment
problem
positive?
means “beautiful”? or “fairly”?
weird spellings?
numerical representation

Some Classic NLP Problems
Assign class label to a sentence/text
Input: I like this movie
Output: <positive>
Assign tag to each word in text
Input: John went to New Orleans
Output: PER --- --- LOC LOC
Convert one string into another string – very general!
I like to eat apples. à John went to New Orleans à
PER --- --- LOC LOC
I eat apples. à
( S ( NP I ) ( VP ( VB eat ) ( NP ( NNS apples ) ) ) )
Sentiment classification
Machine translation
Question answering
Pronoun resolution
Spelling correction
Entity tagging / linking
…
Parsing
Relation extraction
Summarization
Word segmentation
…

But new problems come up every day … for example at our place …
…
,
…
…
ABBA the museum
,
…
Japanese docs Chinese docs
Millions of proposed segment pairs
Only 2/3 of these pairs are good.
Can we quickly write a program to
accurately filter out the bad ones?
Classifier? Zillions of hand-labeled pairs…
…
,
…
…
,
…

How to Represent Words in a Useful Way?
1970s 1990s 2010s
60714 (integer index into vocabulary list)
0
0
14
0
123
0
89
14
…
Representation by Linguist
looking inside mind
Representation by Computer Scientist
looking at nothing
Representation by Algorithm
looking at lots of text
“walks” “walks” “walks”
(how can that be useful?)
multiple senses

Distributional Word Vectors based on Co-Occurrence
“one hot”
sky
0
0
0
0
0
0
0
1
0
0
…
0
0
0
0
0
0
co-occurrence
0
0
14
0
123
0
89
14
0
5
…
0
16
0
9
0
1
“one hot”
sun 0
0
0
0
0
0
0
0
0
0
…
0
1
0
0
0
0
co-occurrence
0
0
17
4
209
0
28
14
0
5
…
0
22
0
11
6
0
“one hot”
reading 0
0
0
0
0
0
0
0
0
0
…
0
0
0
0
1
0
co-occurrence
60
130
7
0
2
0
0
146
0
0
…
0
0
16
0
62
1
moon
sky sky
monograph monograph monograph
moon
“train” vectors on
million sentences
of English text
“moon” and “sky”
appeared together in a
sentence 123 times

Dimension Reduction for Better Generalization (and Visualization)
“one hot”
0
0
0
0
0
0
0
1
0
0
…
0
0
0
0
0
0
co-occurrence
0
0
14
0
123
0
89
0
0
5
…
0
16
0
9
0
1
reduced
12.5
16.2
2.7
-5.3
0.1
-16.2
78.2
-0.2
https://medium.com/@TheDataGyan/dimensionality-reduction-with-pca-and-t-sne-in-r-2715683819
baking
boiling
cooking
pepper
salt
mix
pork
chicken
beef

Build phrase meanings from word meanings?
• Compositional Distributional Semantics (CDS)
mistakebig big mistake
f ,
Sum vectors?
Or represent “big” as a
matrix instead, and multiply
big noun noun
N phrase
“big N”
x =
build f
such that
( )
2.5
16.2
2.7
-5.3
0.1
-16.2
78.2
-0.1
12.5
6.2
2.7
-5.3
0.1
-6.2
7.2
-0.2
9.5
-1.2
2.7
-5.3
0.1
-16.2
-7.2
-0.2
how “big”
behaves in text
how “mistake”
behaves in text
how “big mistake”
behaves in text

Building Word Vectors by Prediction (vs. Co-occurrence Counting)
• on Wednesday ___?___
• on Friday ___?___
• - goal is to learn to make accurate predictions / assign high probs
• - how?
• - learn that “Wednesday” and “Friday” are both WEEKDAY
• - learn that “night” and “evening” are both TIME-OF-DAY
• - learn that “on WEEKDAY” often followed by TIME-OF-DAY
• - so that … “on Friday night” will now appear probable
Maybe we observe in text:
“on Wednesday night”
“on Wednesday evening”
“on Friday evening”
But do not observe:
“on Friday night”

played
the musician the guitar
+ + + +
word2vec [Mikolov et al 13]
hierarchical softmax
noise-contrastive estimation
negative sampling
? ?
neural network [Bengio et al 03]
saw
released 1.4m
vectors trained
on >100b words
“word vector”
the musician played the
guitar
+
σ
+
σ
+
σ
+
σ
piano
Train on 5-word
sequences from
English text

• How to evaluate?
• Similar words get similar vectors (e.g., apple and banana)
• Same relation gets similar vectors
• Analogy task:
•New_York : New_York_Times :: Baltimore : ??? (Baltimore_Sun)
•Steve_Ballmer : Microsoft :: Larry_Page : ??? (Google)
•Result: 72% accuracy for word2vec vectors

Co-occurrence Methods Versus Predictive Methods https://www.aclweb.org/anthology/P14-1023
Analogy task
Many tasks: word synonyms, categories, etc.

Simply take average of all vectors of words in phrase/sentence
Yields a fixed-length vector, no matter how long the phrase/sentence
Loses word-order information (why cats paint = why paint cats)
this movie is super good
average
CLASSIFIER
+ (positive sentiment)
CLASSIFIER TRAINING
positive
negative
positive
positive
negative
positive
…
SENTENCE
REPRESENTATION
this movie is super good à
I liked it à
a snooze-fest à
pretty decent date movie à
input output
e.g. word2vec download
SENTENCE
REPRESENTATIONS
…
Phrase Representation

Multilingual Word Vectors
English word vectors,
trained on English data
Italian word vectors,
trained on Italian data
Linear transformation of
English vectors into Italian space
https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Word_translation_without_parallel_data
Learn W via “seed” word pairs
from bilingual dictionary.
Putting word vectors into a shared, cross-lingual space may
help us pool data or succeed at multilingual tasks.
For example…

Multilingual Word Vectors
average average
SENTENCE
REPRESENTATION
SENTENCE
REPRESENTATION
CLASSIFIER
+ (same meaning) CLASSIFIER TRAINING
+ (same meaning)
- (different meaning)
- (different meaning)
input output
SENTENCE
REPRESENTATIONS
à
à
downloaded multilingual
word vectors
Far fewer labeled examples required
now, to build classifier.
Pre-trained word vectors already
“know” a lot.

Recurrent Neural Network (RNN) Language Model
guitar
Simple averaging of word vectors is not enough:
◦ Word order is important (“this movie is good” vs. “is this movie good”)
◦ Context is important (“plot of land” vs. “plot of movie”)
◦ Composition is complex (“not terrible”, “super terrible”)
the musician played the
guitar
+
σ
+
σ
+
σ
+
σ
piano
the
+
σ
+
σ
+
σ
+
σ
musician
musician
+
σ
+
σ
+
σ
+
σ
played
+
σ
+
σ
+
σ
+
σ
the
+
σ
+
σ
+
σ
+
σ
RNN
guitar
(previously)

Recurrent Neural Network (RNN) Language Model
A more common way to draw the same RNN:
In an unsupervised way (still),
develops word representations (vectors)
that are helpful for predicting the next word.
Evolving hidden vector models word order
and long-distance context.
the musician went home
musician went home <END>
What’s in these hidden vectors?
What do they track?

Sentiment Neuron (OpenAI)
• RNN/LSTM built on 82m Amazon reviews (unlabeled, raw text)
• 1. Matched prior state-of-art on Stanford Sentiment Treebank task
• with 30x-100x fewer labeled examples
• 2. Isolated a single sentiment neuron that was doing most of the work
https://openai.com/blog/unsupervised-sentiment-neuron/
(star rating)

Sentiment Neuron (OpenAI)
• RNN/LSTM built on 82m Amazon reviews (unlabeled, raw text)
• 1. Matched prior state-of-art on Stanford Sentiment Treebank task
• with 30x-100x fewer labeled examples
• 2. Isolated a single sentiment neuron that was doing most of the work
https://openai.com/blog/unsupervised-sentiment-
neuron/
I couldn’t figure out… why this movie
had been discontinued! Now I can
enjoy it anytime I like. So glad to have
found it again.
I couldn’t figure out… how to set it up
being that there was no warning on
the box. I wouldn’t recommend this to
anyone.
Text generated by LM when sentiment
neuron is forced “on”
Text generated by LM when sentiment
neuron is forced “off”

Sequence to Sequence RNN
Now we can read in an arbitrary-length sequence
◦ Build representation of it, use in classifier
How about generating arbitrary-length sequences?
◦ Recall: Many NLP applications convert one string to another
Sequence-to-sequence, aka seq2seq, aka Encoder-Decoder model:
ENCODER DECODER
Train on sentence pairs
Maximize P(word | context)
for each target-side word.
Develops word vectors, and
also sentence vectors.
I saw her <END>
Yo vi a ella <END>
INPUT SENTENCE
REPRESENTATION

What’s in a Sentence Vector?
https://arxiv.org/abs/1409.3215
https://www.aclweb.org/anthology/D16-1248
INPUT SENTENCE
REPRESENTATION

Better Word Vectors?
• Encoder-decoder is limited:
• needs input/output data sets, which are small, compared to plain English
• learns task-specific representations (e.g., counting neurons)
• no pre-trained, generic model others can use
• What can we do with just a zillion of words of English?

ELMo Contextualized Word Vectors (AI2 & UW) https://arxiv.org/pdf/1802.05365.pdf
they were actors in a successful play
she made an excellent play during the game
word2vec
These two vectors will be the same.
ELMo
These two vectors will be different.
In fact, every instance of “play” will be
assigned a different vector, based on
context.
So, ELMo is not a downloadable list of vectors, it
is a downloadable program that turns a sentence into
a sequence of vectors.

ELMo Contextualized Word Vectors
word2vec
ELMo
play
playing game
football
play (in “Kieffer was commended for
his ability to hit in the clutch, as
well as his all-round excellent play”)
play (in “Chico Ruiz made a spectacular
play on Alusik’s grounder”)
play (in “Olivia De Havilland signed
to do a Broadway play for Garson”)
play (in “they were actors who had been
handed fat roles in a successful play”)
…baseball…
…theater…
(reminiscent of multiple senses
in “representation by linguist”)
https://arxiv.org/pdf/1802.05365.pdf

How to Represent Words in a Useful Way?
1970s 1990s 2010s
60714
Representation by
Linguist
Representation by
Computer Scientist
Representation by
word2vec
“tank” “tank” “tank”
multiple, finite senses
12.5
16.2
2.7
-5.3
0.1
-16.2
78.2
-0.2
2018
Representation by
ELMo
“tank”
2.5
16.2
2.7
-5.3
0.1
-16.2
78.2
-0.1
12.5
6.2
2.7
-5.3
0.1
-6.2
7.2
-0.2
12.5
16.2
-2.7
0.3
0.1
-1.2
78.2
-0.9
9.5
-1.2
2.7
-5.3
0.1
-16.2
-7.2
-0.2
tank, as in vehicle
tank, as in storage
… tank in
battle …
… a tank
captain …
… a tank
stores …
multiple, infinite senses
… stocks
tank today …

ELMo Contextualized Word Vectors https://arxiv.org/pdf/1802.05365.pdf
We saw the play
Forward
RNN
Backward
RNN
Weighted
sum
your classifier, tagger, etc
Context-dependent
word vectors
ELMo
Trained on 30 million
English sentences
Plus lots of other tricks!
Multiple layers,
task-based weights for sum,
etc.

ELMo Contextualized Word Vectors
• Results:
Question answering
Natural language inference
Semantic role labeling
Co-reference resolution
Entity tagging
Sentiment
Single method outperforms previous,
problem-specific methods!
Swap out word2vec, swap in ELMo

BERT (Google) https://arxiv.org/abs/1810.04805
•
•
•
•
•

Using BERT Diagram: Jacob Devlin
Huge computational resources A few hours on one GPU

BERT can also give word and sentence representations
• Per word:
• 12 layers x 768 numbers/layer
• Convert to single vector per word by concatenating last four layers
• Per sentence:
• Average all word vectors

GLUE Benchmark Results (BERT paper) https://arxiv.org/abs/1810.04805
GLUE (Wang et al 2018). Collection of NLP tasks with standard train/dev/test data & evaluation metrics.
Natural language inference
(“does sentence A imply sentence B?”)
Paraphrase
(“does sentence A mean the same as sentence B?”)
Sentiment
(pos/neg)

ChineseNLP.xyz
20+ fields and
applications of NLP
Shared
evaluations
Standard
evaluation
test sets
State-of-
the-art
results

Multilingual BERT
• Single model trained on concatenation of Wikipedia in 104 languages
• BERT breaks words into Word Pieces, so vocabulary sharing happens
• Cross-lingual transfer [Pires et al 19]:
• Fine-tune entity tagger on annotated English (91%)
• Apply same tagger to un-annotated German, no training (74%)
• Not bad compared to fine-tuning on German annotations (82%)
• Transfer even happens when languages don’t share any vocabulary

Using Multilingual BERT
(MULTILINGUAL) BERT
CONTEXTUAL
WORD
EMBEDDINGS
12 LAYERS
Convolution Net (CNN)
same meaning?
pre-trained
at Google
fine-tune on
our labeled
task data
[CLS] [SEP]
Japanese Chinese

Multilingual Sentence Representations?
Sentence representations from seq2seq paper:
Le di una carta en el jardín.
En el jardín, ella me dio una carta.

Social Bias
man : woman :: king : ? queen
man : woman :: doctor : ? nurse
man : woman :: professor : ? assistant_professor
wow!
uhhh…
something’s wrong…
Reflects statistics of the text corpora we train on
… and possibly the world we live in

Social Bias Effects
• Web query for “computer scientist” may return a man’s page over a woman’s page.
• Or, machine translation may give wrong results, as in this famous headline:
• French input: Amy Winehouse retrouvée morte dans son appartement
• English output: Amy Winehouse found dead in his apartment
https://languagelog.ldc.upenn.edu/nll/?p=3527

Social Bias
https://languagelog.ldc.upenn.edu/nll/?p=3527

Social Bias
Smarter AI à Less Sexist AI ?https://languagelog.ldc.upenn.edu/nll/?p=3527

Summary So Far
Topic Where to Go
Distributional word vectors Collection of vectors for words and phrases.
Word2vec word vectors Collection of vectors for words and phrases. github.com/idio/wiki2vec/
Multilingual word vectors Collection of multilingual word vectors
projected into common space.
github.com/facebookresearch/MUSE
RNN / LSTM Builds representations of running text.
Sequence-to-sequence RNN Trainable on input/output sentence pairs. github.com/google/seq2seq
ELMo Pre-trained program for turning sentences into
sequences of contextualized word vectors.
allennlp.org/elmo
BERT Pre-trained program for turning sentences into
sequences of contextualized word vectors.
github.com/google-research/bert
GLUE benchmark Set of 9 English NLP standard tasks. https://gluebenchmark.com/
ChineseNLP.xyz Web page describing 20+ Chinese NLP tasks chinesenlp.xyz

nPart I: Text Embedding Applications
nPart II: Graph Embedding Techniques and Applications
nPart III: Text/Graph Embedding Applications in Customer
Service Scenarios

Part I: Text Embedding
Applications

NLP Applications
Annotation Platform
Machine learning Algorithms
K-means, SVM, GBDT
Tokenization, NER
Sentiment analysis Emotion recognition Text Summarization Semantic similarity Text correction
Machine translation Content review
Keyword extraction Text classification Entity linking
NLP PlatformSolutions
NLP
Components
NLP Functions
Algorithm
Libraries
Hadoop/Hive/Spark Cluster GPU cluster
Deep Learning Algorithms
CNN, RNN, LSTM, Attention
NLP Algorithms
Transformer, BERT, ELMO
Comments mining Content Moderation
Coref. resolution
Intent understanding
Word embeddings,
Sentence embeddings
Automatic Speech Recognition
Chatbot
Platform
Translation
Platform
Search/
Recommendations
Infrastructure
Customer
Services AI
Topic mining
Comments/Opinion
Analysis Platform

Applications
• Sentence Classification
• emotion recognition
• opinion/comment classification
• intent understanding
• Paraphrasing Detection
• Semantic Textual Similarity
• question/topic mining
• Information Retrieval
• semantic search
• Recommendation / suggestion
• machine translation
• Sentence Alignment
• ...
Applications of Text Embeddings

Text Classification
•
•
•
•
•
•
Task: assign tags or categories to text according to its content

Semantic Similarity
• Unsupervised Model
Task: measure the degree to which two pieces of text carry the same meaning
•
•
text embeddings by self-supervision learning

Semantic Similarity
• Supervised Classification Model
Sentence 1 Sentence 2
Sentence Vector u Sentence Vector v
(u, v, |u − v|, u ∗ v)
Classifier
Encoder(f) Encoder(g)
concat
Encoder(f) Encoder(g)
concat

Sentence Alignment
Task: the same text in two (or more) languages, align the different
language versions on a sentence level
Aligned:
•
• Last week, the broadcast of period drama “Beauty Private
Kitchen”was temporarily halted, and accidentally triggered heated
debate about faked ratings of locally produced dramas.
Not aligned:
•
• It was a really special time and one we will always cherish.
• The model needs a pre-trained translation model, and the alignment
model is affected by the translation model
• ** the evaluation was based on a manually aligned 200 subtitles as a test
set;

Information Retrieval
• “finding material (usually documents) of an unstructured nature (usually text) that satisfies an
information need from within large collections (usually stored on computers).”
Search & RankingAuto Completion Next Question Suggestion
Manning, Raghavan, Sch¨utze: Introduction to Information Retrieval (MRS), chapter 1

• Lexical search engine looks for literal matches of the query words
• semantic match (not only matching keywords) using embeddings:
• map documents and queries to a embedding space
• search the k nearest neighbors of x in terms of L2 distance
Information Retrieval using Text Embeddings
matching using term embeddings
Items close to query embedding are
retrieved as results

• Use embedding to do IR:
• No need hand-crafted features
• Deal with vocabulary mismatch
• Can capture different notions of similarity based on the data it is trained o
Information Retrieval using Text Embeddings
Sample training data Useful for
<“things to do in seattle”, “seattle tourist attractions”> Document ranking
<“things to do in”, “seattle”> Auto-completion
<“things to do in seattle”, “space needle”> Next query suggestion
Online
Query
Encoder
Embeddings
Indices
Ranker
Encoder
Mined
Questions
Suggestions
Offline

Part II: Graph Embedding
Techniques and Applications

nEmbedding Space
l Latent space such that the properties and the relationships between items are preserved
l Less number of dimensions
l Less sparseness
nGraph Embedding
l Represent a graph as low dimensional vectors
l Preserve graph structures
n Embeddings and Transfer Learning
l Learn embeddings from large dataset / graph (e.g., corpus with 1-100B words), (or Download pre-trained embeddings)
l Transfer embeddings to new task with small training set
l Optional: Continue to fine-tune the embeddings with new data.
Graph Embedding

Why Is It Hard
Jonathan Long et al 2015.
Fully Convolutional Networks for Semantic Segmentation
Fixed 2D structure
Jacob Devlin et al 2018.
BERT: Pre-training of Deep Bidirectional Transformers
for Language Understanding
Linear structure
Regular Euclidean data

Why Is It Hard
lNon-Euclidean and complex topographical
structure
lNo fixed node ordering or reference point(i.e., the
isomorphism problem)
lOften dynamic and have multimodal features.
lSuper large graphs in real world

Why Important
Graphs as denotation of a large number of systems across various areas
lE.g., social network, protein-protein interaction networks, knowledge graphs
Workday 12:00 PM
Workday 6:00PM
Workday 8:00 AM
Usage:
• node classification
• link prediction
• Clustering
• Transfer learning (node/edge representations)

How it works
William L. Hamilton et. al
Inductive Representation Learning on Large Graphs
Perozzi et. al
DeepWalk: Online Learning of Social Representations (2014
1. "Linearizing" the graph
• Create a sentence samples using random
walks
• Training (e.g., SkipGram), prediction
• Node2vec, deepwalk
2. Graph neural networks
• Propage info between nodes
• Sampling, propagation, aggregation, prediction
• GraphSage

Graph Embeddings Models
DeepWalk
Short random walks = sentences
use a stream of short random walks as the basic
tool for extracting information from a network
• easy to capture community information
• local exploration is easy to parallelize
• possible to accommodate small changes
without global recomputation

Translating Embeddings
represented as translations in the embedding space

GNN: Pass messages along edges of graph, agglomerate & transform
• Existing models: train individual embeddings for each node
• GraphSage:
• learn a function that generates embeddings by sampling and
aggregating features from a node's local neighborhood
• leverage node features (e.g., text attributes, node profile
information, node degrees)
GraphSage (GNN)

•
•

Graph Embeddings in Airbnb
Real-time Personalization using Embeddings for Search Ranking
• low-dimensional representations of home listings and users
• based on contextual co-occurrence in click/booking sessions
• Word/sentence -> actions/session
geographical similarity is well
encoded
listing characteristics, such as architecture, style and feel are
captured by embeddings
Similar Listing—CTR +21%
Real-time Personalization using Embeddings for Search Ranking at Airbnb, KDD 2018, best paper award

Graph Embeddings in Alibaba
Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba
Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba, KDD 2018

Embedding Applications in DiDi - Ride-Hailing
r1
r2
r3r4
r2
r5
r1
r4
1. Build traveling graphs from Ride-Hailing dataset
C
r2
! D
r3
! E
r4
! F
r4
! B<latexit sha1_base64="MzN78tmKlzkK38rkjOjBOT0gRvM=">AAAC7nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQt0OivEFJckJmcXpeZUF8Ub1VbH5OTnpRdlpmeUJBYV5ZfXKrigqDDGosIVRYUJFhVuBFU4xQsoG+gZgIECJsMQylBmgIKAfIHlDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzWbgAkaAIXpwYzLCjPQMDfQMA02UHZygUcHBIM2gxKABDG9zBgcGD4YAhlCgve8ZRRnlGOWZCpimMc1lmg9RysQI1SPMgAKYVgAAGMDBUg==</latexit><latexit sha1_base64="MzN78tmKlzkK38rkjOjBOT0gRvM=">AAAC7nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQt0OivEFJckJmcXpeZUF8Ub1VbH5OTnpRdlpmeUJBYV5ZfXKrigqDDGosIVRYUJFhVuBFU4xQsoG+gZgIECJsMQylBmgIKAfIHlDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzWbgAkaAIXpwYzLCjPQMDfQMA02UHZygUcHBIM2gxKABDG9zBgcGD4YAhlCgve8ZRRnlGOWZCpimMc1lmg9RysQI1SPMgAKYVgAAGMDBUg==</latexit><latexit sha1_base64="MzN78tmKlzkK38rkjOjBOT0gRvM=">AAAC7nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQt0OivEFJckJmcXpeZUF8Ub1VbH5OTnpRdlpmeUJBYV5ZfXKrigqDDGosIVRYUJFhVuBFU4xQsoG+gZgIECJsMQylBmgIKAfIHlDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzWbgAkaAIXpwYzLCjPQMDfQMA02UHZygUcHBIM2gxKABDG9zBgcGD4YAhlCgve8ZRRnlGOWZCpimMc1lmg9RysQI1SPMgAKYVgAAGMDBUg==</latexit><latexit sha1_base64="MzN78tmKlzkK38rkjOjBOT0gRvM=">AAAC7nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQt0OivEFJckJmcXpeZUF8Ub1VbH5OTnpRdlpmeUJBYV5ZfXKrigqDDGosIVRYUJFhVuBFU4xQsoG+gZgIECJsMQylBmgIKAfIHlDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzWbgAkaAIXpwYzLCjPQMDfQMA02UHZygUcHBIM2gxKABDG9zBgcGD4YAhlCgve8ZRRnlGOWZCpimMc1lmg9RysQI1SPMgAKYVgAAGMDBUg==</latexit>
C, r2, D<latexit sha1_base64="hbnDFHJjhY2iLBSKK/2wYepR3mY=">AAACbXichVHLSsNAFD2Nr1ofrYogKBIsPhZSboqguBLrwmVbrYoPShJHDaZJSNKCFn/AteBCFBRExM9w4w+48BPEhQsFNy68TQOiot5hZs6cuefOmRnNMQ3PJ3qISA2NTc0t0dZYW3tHZzzR1b3k2WVXFwXdNm13RVM9YRqWKPiGb4oVxxVqSTPFsrabqe0vV4TrGba16O85YqOkblvGlqGrPlOrmXHZLabH5Tm5mEhSioKQfwIlBEmEkbUTV1jHJmzoKKMEAQs+YxMqPG5rUEBwmNtAlTmXkRHsCxwgxtoyZwnOUJnd5XGbV2sha/G6VtML1DqfYnJ3WSljmO7pml7ojm7okd5/rVUNatS87PGs1bXCKcYP+xbe/lWVePax86n607OPLUwFXg327gRM7RZ6XV/ZP35ZmM4PV0fogp7Y/zk90C3fwKq86pc5kT9BjD9A+f7cP8FSOqVQSslNJGdmw6+Ioh9DGOP3nsQM5pFFgc+1cIRTnEWepV5pQBqsp0qRUNODLyGNfgDmMYwc</latexit><latexit sha1_base64="hbnDFHJjhY2iLBSKK/2wYepR3mY=">AAACbXichVHLSsNAFD2Nr1ofrYogKBIsPhZSboqguBLrwmVbrYoPShJHDaZJSNKCFn/AteBCFBRExM9w4w+48BPEhQsFNy68TQOiot5hZs6cuefOmRnNMQ3PJ3qISA2NTc0t0dZYW3tHZzzR1b3k2WVXFwXdNm13RVM9YRqWKPiGb4oVxxVqSTPFsrabqe0vV4TrGba16O85YqOkblvGlqGrPlOrmXHZLabH5Tm5mEhSioKQfwIlBEmEkbUTV1jHJmzoKKMEAQs+YxMqPG5rUEBwmNtAlTmXkRHsCxwgxtoyZwnOUJnd5XGbV2sha/G6VtML1DqfYnJ3WSljmO7pml7ojm7okd5/rVUNatS87PGs1bXCKcYP+xbe/lWVePax86n607OPLUwFXg327gRM7RZ6XV/ZP35ZmM4PV0fogp7Y/zk90C3fwKq86pc5kT9BjD9A+f7cP8FSOqVQSslNJGdmw6+Ioh9DGOP3nsQM5pFFgc+1cIRTnEWepV5pQBqsp0qRUNODLyGNfgDmMYwc</latexit><latexit sha1_base64="hbnDFHJjhY2iLBSKK/2wYepR3mY=">AAACbXichVHLSsNAFD2Nr1ofrYogKBIsPhZSboqguBLrwmVbrYoPShJHDaZJSNKCFn/AteBCFBRExM9w4w+48BPEhQsFNy68TQOiot5hZs6cuefOmRnNMQ3PJ3qISA2NTc0t0dZYW3tHZzzR1b3k2WVXFwXdNm13RVM9YRqWKPiGb4oVxxVqSTPFsrabqe0vV4TrGba16O85YqOkblvGlqGrPlOrmXHZLabH5Tm5mEhSioKQfwIlBEmEkbUTV1jHJmzoKKMEAQs+YxMqPG5rUEBwmNtAlTmXkRHsCxwgxtoyZwnOUJnd5XGbV2sha/G6VtML1DqfYnJ3WSljmO7pml7ojm7okd5/rVUNatS87PGs1bXCKcYP+xbe/lWVePax86n607OPLUwFXg327gRM7RZ6XV/ZP35ZmM4PV0fogp7Y/zk90C3fwKq86pc5kT9BjD9A+f7cP8FSOqVQSslNJGdmw6+Ioh9DGOP3nsQM5pFFgc+1cIRTnEWepV5pQBqsp0qRUNODLyGNfgDmMYwc</latexit><latexit sha1_base64="hbnDFHJjhY2iLBSKK/2wYepR3mY=">AAACbXichVHLSsNAFD2Nr1ofrYogKBIsPhZSboqguBLrwmVbrYoPShJHDaZJSNKCFn/AteBCFBRExM9w4w+48BPEhQsFNy68TQOiot5hZs6cuefOmRnNMQ3PJ3qISA2NTc0t0dZYW3tHZzzR1b3k2WVXFwXdNm13RVM9YRqWKPiGb4oVxxVqSTPFsrabqe0vV4TrGba16O85YqOkblvGlqGrPlOrmXHZLabH5Tm5mEhSioKQfwIlBEmEkbUTV1jHJmzoKKMEAQs+YxMqPG5rUEBwmNtAlTmXkRHsCxwgxtoyZwnOUJnd5XGbV2sha/G6VtML1DqfYnJ3WSljmO7pml7ojm7okd5/rVUNatS87PGs1bXCKcYP+xbe/lWVePax86n607OPLUwFXg327gRM7RZ6XV/ZP35ZmM4PV0fogp7Y/zk90C3fwKq86pc5kT9BjD9A+f7cP8FSOqVQSslNJGdmw6+Ioh9DGOP3nsQM5pFFgc+1cIRTnEWepV5pQBqsp0qRUNODLyGNfgDmMYwc</latexit>
D, r3, E<latexit sha1_base64="6hSYwpklL3Fgb70OoOSHhNG3eJI=">AAACbHichVG7SgNBFD1Z3/GR+CgEEYJBsZBwVwXFKvgAS195iErYXUdd3Be7m0AM/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazICrqHWbmzJl77pyZUR1D93yix4jU0NjU3NLaFm3v6OyKxbt7sp5ddDWR0WzDdvOq4glDt0TG131D5B1XKKZqiJy6P1fbz5WE6+m2teaXHbFlKruWvqNris/U+vxYwi1MjCUWCvEkpSiIxE8ghyCJMJbs+DU2sQ0bGoowIWDBZ2xAgcdtAzIIDnNbqDDnMtKDfYFDRFlb5CzBGQqz+zzu8mojZC1e12p6gVrjUwzuLisTGKYHuqEq3dMtPdH7r7UqQY2alzLPal0rnELsqH/17V+VybOPvU/Vn5597GA68KqzdydgarfQ6vrSwUl1dWZluDJCl/TM/i/oke74BlbpVbtaFiuniPIHyN+f+yfIjqdkSsnLk8n0bPgVrRjAEEb5vaeQxiKWkOFzTRzjDOeRF6lPGpAG66lSJNT04ktIIx+Eqov1</latexit><latexit sha1_base64="6hSYwpklL3Fgb70OoOSHhNG3eJI=">AAACbHichVG7SgNBFD1Z3/GR+CgEEYJBsZBwVwXFKvgAS195iErYXUdd3Be7m0AM/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazICrqHWbmzJl77pyZUR1D93yix4jU0NjU3NLaFm3v6OyKxbt7sp5ddDWR0WzDdvOq4glDt0TG131D5B1XKKZqiJy6P1fbz5WE6+m2teaXHbFlKruWvqNris/U+vxYwi1MjCUWCvEkpSiIxE8ghyCJMJbs+DU2sQ0bGoowIWDBZ2xAgcdtAzIIDnNbqDDnMtKDfYFDRFlb5CzBGQqz+zzu8mojZC1e12p6gVrjUwzuLisTGKYHuqEq3dMtPdH7r7UqQY2alzLPal0rnELsqH/17V+VybOPvU/Vn5597GA68KqzdydgarfQ6vrSwUl1dWZluDJCl/TM/i/oke74BlbpVbtaFiuniPIHyN+f+yfIjqdkSsnLk8n0bPgVrRjAEEb5vaeQxiKWkOFzTRzjDOeRF6lPGpAG66lSJNT04ktIIx+Eqov1</latexit><latexit sha1_base64="6hSYwpklL3Fgb70OoOSHhNG3eJI=">AAACbHichVG7SgNBFD1Z3/GR+CgEEYJBsZBwVwXFKvgAS195iErYXUdd3Be7m0AM/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazICrqHWbmzJl77pyZUR1D93yix4jU0NjU3NLaFm3v6OyKxbt7sp5ddDWR0WzDdvOq4glDt0TG131D5B1XKKZqiJy6P1fbz5WE6+m2teaXHbFlKruWvqNris/U+vxYwi1MjCUWCvEkpSiIxE8ghyCJMJbs+DU2sQ0bGoowIWDBZ2xAgcdtAzIIDnNbqDDnMtKDfYFDRFlb5CzBGQqz+zzu8mojZC1e12p6gVrjUwzuLisTGKYHuqEq3dMtPdH7r7UqQY2alzLPal0rnELsqH/17V+VybOPvU/Vn5597GA68KqzdydgarfQ6vrSwUl1dWZluDJCl/TM/i/oke74BlbpVbtaFiuniPIHyN+f+yfIjqdkSsnLk8n0bPgVrRjAEEb5vaeQxiKWkOFzTRzjDOeRF6lPGpAG66lSJNT04ktIIx+Eqov1</latexit><latexit sha1_base64="6hSYwpklL3Fgb70OoOSHhNG3eJI=">AAACbHichVG7SgNBFD1Z3/GR+CgEEYJBsZBwVwXFKvgAS195iErYXUdd3Be7m0AM/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazICrqHWbmzJl77pyZUR1D93yix4jU0NjU3NLaFm3v6OyKxbt7sp5ddDWR0WzDdvOq4glDt0TG131D5B1XKKZqiJy6P1fbz5WE6+m2teaXHbFlKruWvqNris/U+vxYwi1MjCUWCvEkpSiIxE8ghyCJMJbs+DU2sQ0bGoowIWDBZ2xAgcdtAzIIDnNbqDDnMtKDfYFDRFlb5CzBGQqz+zzu8mojZC1e12p6gVrjUwzuLisTGKYHuqEq3dMtPdH7r7UqQY2alzLPal0rnELsqH/17V+VybOPvU/Vn5597GA68KqzdydgarfQ6vrSwUl1dWZluDJCl/TM/i/oke74BlbpVbtaFiuniPIHyN+f+yfIjqdkSsnLk8n0bPgVrRjAEEb5vaeQxiKWkOFzTRzjDOeRF6lPGpAG66lSJNT04ktIIx+Eqov1</latexit>
F, r4, B<latexit sha1_base64="R2A9RG5yskesH8jrwHlFyjZcRwk=">AAACbHichVG7SgNBFD1Z3/EVH4UgQjAkWIRwVwTFShTE0leiEkPYXSe6uC92NwEN/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazIBrUO8zMmTP33DkzozqG7vlETxGppbWtvaOzK9rd09vXHxsYzHl22dVEVrMN291UFU8YuiWyvu4bYtNxhWKqhthQ9xfq+xsV4Xq6ba37B44omMqupZd0TfGZ2lpMx93iVDo+X4wlKENBxJuBHIIEwli2YzfYxg5saCjDhIAFn7EBBR63PGQQHOYKqDLnMtKDfYEjRFlb5izBGQqz+zzu8iofshav6zW9QK3xKQZ3l5VxJOmRbqlGD3RHz/Txa61qUKPu5YBntaEVTrH/eGTt/V+VybOPvS/Vn559lDATeNXZuxMw9VtoDX3l8LS2NruarKboil7Y/yU90T3fwKq8adcrYvUMUf4A+edzN4PcZEamjLwylZibD7+iE6MYxwS/9zTmsIRlZPlcEyc4x0XkVRqWRqWxRqoUCTVD+BZS6hOEvYv1</latexit><latexit sha1_base64="R2A9RG5yskesH8jrwHlFyjZcRwk=">AAACbHichVG7SgNBFD1Z3/EVH4UgQjAkWIRwVwTFShTE0leiEkPYXSe6uC92NwEN/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazIBrUO8zMmTP33DkzozqG7vlETxGppbWtvaOzK9rd09vXHxsYzHl22dVEVrMN291UFU8YuiWyvu4bYtNxhWKqhthQ9xfq+xsV4Xq6ba37B44omMqupZd0TfGZ2lpMx93iVDo+X4wlKENBxJuBHIIEwli2YzfYxg5saCjDhIAFn7EBBR63PGQQHOYKqDLnMtKDfYEjRFlb5izBGQqz+zzu8iofshav6zW9QK3xKQZ3l5VxJOmRbqlGD3RHz/Txa61qUKPu5YBntaEVTrH/eGTt/V+VybOPvS/Vn559lDATeNXZuxMw9VtoDX3l8LS2NruarKboil7Y/yU90T3fwKq8adcrYvUMUf4A+edzN4PcZEamjLwylZibD7+iE6MYxwS/9zTmsIRlZPlcEyc4x0XkVRqWRqWxRqoUCTVD+BZS6hOEvYv1</latexit><latexit sha1_base64="R2A9RG5yskesH8jrwHlFyjZcRwk=">AAACbHichVG7SgNBFD1Z3/EVH4UgQjAkWIRwVwTFShTE0leiEkPYXSe6uC92NwEN/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazIBrUO8zMmTP33DkzozqG7vlETxGppbWtvaOzK9rd09vXHxsYzHl22dVEVrMN291UFU8YuiWyvu4bYtNxhWKqhthQ9xfq+xsV4Xq6ba37B44omMqupZd0TfGZ2lpMx93iVDo+X4wlKENBxJuBHIIEwli2YzfYxg5saCjDhIAFn7EBBR63PGQQHOYKqDLnMtKDfYEjRFlb5izBGQqz+zzu8iofshav6zW9QK3xKQZ3l5VxJOmRbqlGD3RHz/Txa61qUKPu5YBntaEVTrH/eGTt/V+VybOPvS/Vn559lDATeNXZuxMw9VtoDX3l8LS2NruarKboil7Y/yU90T3fwKq8adcrYvUMUf4A+edzN4PcZEamjLwylZibD7+iE6MYxwS/9zTmsIRlZPlcEyc4x0XkVRqWRqWxRqoUCTVD+BZS6hOEvYv1</latexit><latexit sha1_base64="R2A9RG5yskesH8jrwHlFyjZcRwk=">AAACbHichVG7SgNBFD1Z3/EVH4UgQjAkWIRwVwTFShTE0leiEkPYXSe6uC92NwEN/oCthYVaKIiIn2HjD1j4CSLYRLCx8GazIBrUO8zMmTP33DkzozqG7vlETxGppbWtvaOzK9rd09vXHxsYzHl22dVEVrMN291UFU8YuiWyvu4bYtNxhWKqhthQ9xfq+xsV4Xq6ba37B44omMqupZd0TfGZ2lpMx93iVDo+X4wlKENBxJuBHIIEwli2YzfYxg5saCjDhIAFn7EBBR63PGQQHOYKqDLnMtKDfYEjRFlb5izBGQqz+zzu8iofshav6zW9QK3xKQZ3l5VxJOmRbqlGD3RHz/Txa61qUKPu5YBntaEVTrH/eGTt/V+VybOPvS/Vn559lDATeNXZuxMw9VtoDX3l8LS2NruarKboil7Y/yU90T3fwKq8adcrYvUMUf4A+edzN4PcZEamjLwylZibD7+iE6MYxwS/9zTmsIRlZPlcEyc4x0XkVRqWRqWxRqoUCTVD+BZS6hOEvYv1</latexit>
E, r4, F<latexit sha1_base64="WtdNDOUdoqQKOswMZPhHet7cP4w=">AAACbHichVG7SgNBFD1Z3/EVH4UQhGAwWIRwVwTFShTFMj5ilChhd53ExX2xuwlo8AdsLSzUQkFE/Awbf8DCTxDBRsHGwpvNgqiod5iZM2fuuXNmRnUM3fOJHiJSU3NLa1t7R7Szq7unN9bXv+bZFVcTOc02bHddVTxh6JbI+bpviHXHFYqpGiKv7s7V9/NV4Xq6ba36e47YMpWypZd0TfGZ2phPJ9ziRDqxUIwlKUNBJH4COQRJhJG1Y1fYxDZsaKjAhIAFn7EBBR63AmQQHOa2UGPOZaQH+wIHiLK2wlmCMxRmd3ks86oQshav6zW9QK3xKQZ3l5UJjNI9XdML3dENPdL7r7VqQY26lz2e1YZWOMXew6GVt39VJs8+dj5Vf3r2UcJU4FVn707A1G+hNfTV/eOXlenl0VqKLuiJ/Z/TA93yDazqq3a5JJZPEOUPkL8/90+wNp6RKSMvTSRnZsOvaEccIxjj957EDBaRRY7PNXGEU5xFnqVBKS4NN1KlSKgZwJeQUh+KtYv4</latexit><latexit sha1_base64="WtdNDOUdoqQKOswMZPhHet7cP4w=">AAACbHichVG7SgNBFD1Z3/EVH4UQhGAwWIRwVwTFShTFMj5ilChhd53ExX2xuwlo8AdsLSzUQkFE/Awbf8DCTxDBRsHGwpvNgqiod5iZM2fuuXNmRnUM3fOJHiJSU3NLa1t7R7Szq7unN9bXv+bZFVcTOc02bHddVTxh6JbI+bpviHXHFYqpGiKv7s7V9/NV4Xq6ba36e47YMpWypZd0TfGZ2phPJ9ziRDqxUIwlKUNBJH4COQRJhJG1Y1fYxDZsaKjAhIAFn7EBBR63AmQQHOa2UGPOZaQH+wIHiLK2wlmCMxRmd3ks86oQshav6zW9QK3xKQZ3l5UJjNI9XdML3dENPdL7r7VqQY26lz2e1YZWOMXew6GVt39VJs8+dj5Vf3r2UcJU4FVn707A1G+hNfTV/eOXlenl0VqKLuiJ/Z/TA93yDazqq3a5JJZPEOUPkL8/90+wNp6RKSMvTSRnZsOvaEccIxjj957EDBaRRY7PNXGEU5xFnqVBKS4NN1KlSKgZwJeQUh+KtYv4</latexit><latexit sha1_base64="WtdNDOUdoqQKOswMZPhHet7cP4w=">AAACbHichVG7SgNBFD1Z3/EVH4UQhGAwWIRwVwTFShTFMj5ilChhd53ExX2xuwlo8AdsLSzUQkFE/Awbf8DCTxDBRsHGwpvNgqiod5iZM2fuuXNmRnUM3fOJHiJSU3NLa1t7R7Szq7unN9bXv+bZFVcTOc02bHddVTxh6JbI+bpviHXHFYqpGiKv7s7V9/NV4Xq6ba36e47YMpWypZd0TfGZ2phPJ9ziRDqxUIwlKUNBJH4COQRJhJG1Y1fYxDZsaKjAhIAFn7EBBR63AmQQHOa2UGPOZaQH+wIHiLK2wlmCMxRmd3ks86oQshav6zW9QK3xKQZ3l5UJjNI9XdML3dENPdL7r7VqQY26lz2e1YZWOMXew6GVt39VJs8+dj5Vf3r2UcJU4FVn707A1G+hNfTV/eOXlenl0VqKLuiJ/Z/TA93yDazqq3a5JJZPEOUPkL8/90+wNp6RKSMvTSRnZsOvaEccIxjj957EDBaRRY7PNXGEU5xFnqVBKS4NN1KlSKgZwJeQUh+KtYv4</latexit><latexit sha1_base64="WtdNDOUdoqQKOswMZPhHet7cP4w=">AAACbHichVG7SgNBFD1Z3/EVH4UQhGAwWIRwVwTFShTFMj5ilChhd53ExX2xuwlo8AdsLSzUQkFE/Awbf8DCTxDBRsHGwpvNgqiod5iZM2fuuXNmRnUM3fOJHiJSU3NLa1t7R7Szq7unN9bXv+bZFVcTOc02bHddVTxh6JbI+bpviHXHFYqpGiKv7s7V9/NV4Xq6ba36e47YMpWypZd0TfGZ2phPJ9ziRDqxUIwlKUNBJH4COQRJhJG1Y1fYxDZsaKjAhIAFn7EBBR63AmQQHOa2UGPOZaQH+wIHiLK2wlmCMxRmd3ks86oQshav6zW9QK3xKQZ3l5UJjNI9XdML3dENPdL7r7VqQY26lz2e1YZWOMXew6GVt39VJs8+dj5Vf3r2UcJU4FVn707A1G+hNfTV/eOXlenl0VqKLuiJ/Z/TA93yDazqq3a5JJZPEOUPkL8/90+wNp6RKSMvTSRnZsOvaEccIxjj957EDBaRRY7PNXGEU5xFnqVBKS4NN1KlSKgZwJeQUh+KtYv4</latexit>
C, r2, r3, E<latexit sha1_base64="TOQxlBvSsKscsKhRpfGQ78IUDng=">AAACcXichVHLSsNAFD2Nr1ofrbpR3ASLIijlpgqKK0EEl1qtCj5KEkcNTZOQpIVa/AF/QMFVCyLiZ7jxB1z0E8RlBTcuvEkDoqLeYWbOnLnnzpkZzTENzydqxqSOzq7unnhvoq9/YDCZGhre9uyyq4u8bpu2u6upnjANS+R9wzfFruMKtaSZYkcrrgT7OxXheoZtbflVRxyU1BPLODZ01WfqcGVWdgvZYJiblVcLqTRlKAz5J1AikEYU63bqFvs4gg0dZZQgYMFnbEKFx20PCggOcweoMecyMsJ9gXMkWFvmLMEZKrNFHk94tRexFq+Dml6o1vkUk7vLShmT9ER31KJHuqdnev+1Vi2sEXip8qy1tcIpJC9GN9/+VZV49nH6qfrTs49jLIZeDfbuhExwC72tr5xdtjaXcpO1KWrQC/uvU5Me+AZW5VW/2RC5ayT4A5Tvz/0TbGczCmWUjfn0cjb6ijjGMYFpfu8FLGMN68jzuS6uUEcj1pLGJFmaaKdKsUgzgi8hzXwA5V2NZQ==</latexit><latexit sha1_base64="TOQxlBvSsKscsKhRpfGQ78IUDng=">AAACcXichVHLSsNAFD2Nr1ofrbpR3ASLIijlpgqKK0EEl1qtCj5KEkcNTZOQpIVa/AF/QMFVCyLiZ7jxB1z0E8RlBTcuvEkDoqLeYWbOnLnnzpkZzTENzydqxqSOzq7unnhvoq9/YDCZGhre9uyyq4u8bpu2u6upnjANS+R9wzfFruMKtaSZYkcrrgT7OxXheoZtbflVRxyU1BPLODZ01WfqcGVWdgvZYJiblVcLqTRlKAz5J1AikEYU63bqFvs4gg0dZZQgYMFnbEKFx20PCggOcweoMecyMsJ9gXMkWFvmLMEZKrNFHk94tRexFq+Dml6o1vkUk7vLShmT9ER31KJHuqdnev+1Vi2sEXip8qy1tcIpJC9GN9/+VZV49nH6qfrTs49jLIZeDfbuhExwC72tr5xdtjaXcpO1KWrQC/uvU5Me+AZW5VW/2RC5ayT4A5Tvz/0TbGczCmWUjfn0cjb6ijjGMYFpfu8FLGMN68jzuS6uUEcj1pLGJFmaaKdKsUgzgi8hzXwA5V2NZQ==</latexit><latexit sha1_base64="TOQxlBvSsKscsKhRpfGQ78IUDng=">AAACcXichVHLSsNAFD2Nr1ofrbpR3ASLIijlpgqKK0EEl1qtCj5KEkcNTZOQpIVa/AF/QMFVCyLiZ7jxB1z0E8RlBTcuvEkDoqLeYWbOnLnnzpkZzTENzydqxqSOzq7unnhvoq9/YDCZGhre9uyyq4u8bpu2u6upnjANS+R9wzfFruMKtaSZYkcrrgT7OxXheoZtbflVRxyU1BPLODZ01WfqcGVWdgvZYJiblVcLqTRlKAz5J1AikEYU63bqFvs4gg0dZZQgYMFnbEKFx20PCggOcweoMecyMsJ9gXMkWFvmLMEZKrNFHk94tRexFq+Dml6o1vkUk7vLShmT9ER31KJHuqdnev+1Vi2sEXip8qy1tcIpJC9GN9/+VZV49nH6qfrTs49jLIZeDfbuhExwC72tr5xdtjaXcpO1KWrQC/uvU5Me+AZW5VW/2RC5ayT4A5Tvz/0TbGczCmWUjfn0cjb6ijjGMYFpfu8FLGMN68jzuS6uUEcj1pLGJFmaaKdKsUgzgi8hzXwA5V2NZQ==</latexit><latexit sha1_base64="TOQxlBvSsKscsKhRpfGQ78IUDng=">AAACcXichVHLSsNAFD2Nr1ofrbpR3ASLIijlpgqKK0EEl1qtCj5KEkcNTZOQpIVa/AF/QMFVCyLiZ7jxB1z0E8RlBTcuvEkDoqLeYWbOnLnnzpkZzTENzydqxqSOzq7unnhvoq9/YDCZGhre9uyyq4u8bpu2u6upnjANS+R9wzfFruMKtaSZYkcrrgT7OxXheoZtbflVRxyU1BPLODZ01WfqcGVWdgvZYJiblVcLqTRlKAz5J1AikEYU63bqFvs4gg0dZZQgYMFnbEKFx20PCggOcweoMecyMsJ9gXMkWFvmLMEZKrNFHk94tRexFq+Dml6o1vkUk7vLShmT9ER31KJHuqdnev+1Vi2sEXip8qy1tcIpJC9GN9/+VZV49nH6qfrTs49jLIZeDfbuhExwC72tr5xdtjaXcpO1KWrQC/uvU5Me+AZW5VW/2RC5ayT4A5Tvz/0TbGczCmWUjfn0cjb6ijjGMYFpfu8FLGMN68jzuS6uUEcj1pLGJFmaaKdKsUgzgi8hzXwA5V2NZQ==</latexit>
E, r4, r4, B<latexit sha1_base64="f69O9Fx0RrRSqeVUDaUGre3Eed8=">AAACcXichVHLSsNAFD2N7/qqulHclBZFUMpNKSiuRBFcttaqoLUkcaqheZGkhVr8AX9AwZWCiPgZbvwBF36CuKzgxoW3aUC0qHeYmTNn7rlzZkZ1DN3ziZ4jUld3T29f/0B0cGh4ZDQ2Nr7t2VVXEwXNNmx3V1U8YeiWKPi6b4hdxxWKqRpiR62stfZ3asL1dNva8uuOKJrKkaWXdU3xmTpYX4i7pUw4rJZiSUpREPFOIIcgiTCyduwW+ziEDQ1VmBCw4DM2oMDjtgcZBIe5IhrMuYz0YF/gFFHWVjlLcIbCbIXHI17thazF61ZNL1BrfIrB3WVlHDP0RHfUpEe6pxf6+LVWI6jR8lLnWW1rhVMaPZvMv/+rMnn2cfyl+tOzjzKWAq86e3cCpnULra2vnZw388ubM41ZuqZX9n9Fz/TAN7Bqb9pNTmxeIsofIP987k6wnU7JlJJzmeRKOvyKfkwjgTl+70WsYANZFPhcFxe4wnWkKU1JcSnRTpUioWYC30Ka/wTpio1n</latexit><latexit sha1_base64="f69O9Fx0RrRSqeVUDaUGre3Eed8=">AAACcXichVHLSsNAFD2N7/qqulHclBZFUMpNKSiuRBFcttaqoLUkcaqheZGkhVr8AX9AwZWCiPgZbvwBF36CuKzgxoW3aUC0qHeYmTNn7rlzZkZ1DN3ziZ4jUld3T29f/0B0cGh4ZDQ2Nr7t2VVXEwXNNmx3V1U8YeiWKPi6b4hdxxWKqRpiR62stfZ3asL1dNva8uuOKJrKkaWXdU3xmTpYX4i7pUw4rJZiSUpREPFOIIcgiTCyduwW+ziEDQ1VmBCw4DM2oMDjtgcZBIe5IhrMuYz0YF/gFFHWVjlLcIbCbIXHI17thazF61ZNL1BrfIrB3WVlHDP0RHfUpEe6pxf6+LVWI6jR8lLnWW1rhVMaPZvMv/+rMnn2cfyl+tOzjzKWAq86e3cCpnULra2vnZw388ubM41ZuqZX9n9Fz/TAN7Bqb9pNTmxeIsofIP987k6wnU7JlJJzmeRKOvyKfkwjgTl+70WsYANZFPhcFxe4wnWkKU1JcSnRTpUioWYC30Ka/wTpio1n</latexit><latexit sha1_base64="f69O9Fx0RrRSqeVUDaUGre3Eed8=">AAACcXichVHLSsNAFD2N7/qqulHclBZFUMpNKSiuRBFcttaqoLUkcaqheZGkhVr8AX9AwZWCiPgZbvwBF36CuKzgxoW3aUC0qHeYmTNn7rlzZkZ1DN3ziZ4jUld3T29f/0B0cGh4ZDQ2Nr7t2VVXEwXNNmx3V1U8YeiWKPi6b4hdxxWKqRpiR62stfZ3asL1dNva8uuOKJrKkaWXdU3xmTpYX4i7pUw4rJZiSUpREPFOIIcgiTCyduwW+ziEDQ1VmBCw4DM2oMDjtgcZBIe5IhrMuYz0YF/gFFHWVjlLcIbCbIXHI17thazF61ZNL1BrfIrB3WVlHDP0RHfUpEe6pxf6+LVWI6jR8lLnWW1rhVMaPZvMv/+rMnn2cfyl+tOzjzKWAq86e3cCpnULra2vnZw388ubM41ZuqZX9n9Fz/TAN7Bqb9pNTmxeIsofIP987k6wnU7JlJJzmeRKOvyKfkwjgTl+70WsYANZFPhcFxe4wnWkKU1JcSnRTpUioWYC30Ka/wTpio1n</latexit><latexit sha1_base64="f69O9Fx0RrRSqeVUDaUGre3Eed8=">AAACcXichVHLSsNAFD2N7/qqulHclBZFUMpNKSiuRBFcttaqoLUkcaqheZGkhVr8AX9AwZWCiPgZbvwBF36CuKzgxoW3aUC0qHeYmTNn7rlzZkZ1DN3ziZ4jUld3T29f/0B0cGh4ZDQ2Nr7t2VVXEwXNNmx3V1U8YeiWKPi6b4hdxxWKqRpiR62stfZ3asL1dNva8uuOKJrKkaWXdU3xmTpYX4i7pUw4rJZiSUpREPFOIIcgiTCyduwW+ziEDQ1VmBCw4DM2oMDjtgcZBIe5IhrMuYz0YF/gFFHWVjlLcIbCbIXHI17thazF61ZNL1BrfIrB3WVlHDP0RHfUpEe6pxf6+LVWI6jR8lLnWW1rhVMaPZvMv/+rMnn2cfyl+tOzjzKWAq86e3cCpnULra2vnZw388ubM41ZuqZX9n9Fz/TAN7Bqb9pNTmxeIsofIP987k6wnU7JlJJzmeRKOvyKfkwjgTl+70WsYANZFPhcFxe4wnWkKU1JcSnRTpUioWYC30Ka/wTpio1n</latexit>
D, r3, r4, F<latexit sha1_base64="EYzUCnncTU6pM+18fa1Q8gLZKys=">AAACcXichVHLSsNAFD2N7/qqulHchBZFsJQbFRRXgiIuW7UP0FqSOGowTUKSFrT4A/6AgisFEfEz3PgDLvwEcVnBjQtv0oCoqHeYmTNn7rlzZkZzTMPziZ5iUlt7R2dXd0+8t69/YDAxNFzw7Jqri7xum7Zb0lRPmIYl8r7hm6LkuEKtaqYoaofLwX6xLlzPsK1N/8gR5aq6bxl7hq76TO2spGW3MhsMc2l5tZJIUYbCkH8CJQIpRJG1EzfYxi5s6KihCgELPmMTKjxuW1BAcJgro8Gcy8gI9wVOEGdtjbMEZ6jMHvK4z6utiLV4HdT0QrXOp5jcXVbKmKBHuqUmPdAdPdP7r7UaYY3AyxHPWksrnMrg6ejG27+qKs8+Dj5Vf3r2sYeF0KvB3p2QCW6ht/T147PmxuL6RGOSruiF/V/SE93zDaz6q36dE+sXiPMHKN+f+ycozGQUyii5udTSTPQV3RhHElP83vNYwhqyyPO5Ls5xiatYUxqTZCnZSpVikWYEX0Ka/gDtdY1p</latexit><latexit sha1_base64="EYzUCnncTU6pM+18fa1Q8gLZKys=">AAACcXichVHLSsNAFD2N7/qqulHchBZFsJQbFRRXgiIuW7UP0FqSOGowTUKSFrT4A/6AgisFEfEz3PgDLvwEcVnBjQtv0oCoqHeYmTNn7rlzZkZzTMPziZ5iUlt7R2dXd0+8t69/YDAxNFzw7Jqri7xum7Zb0lRPmIYl8r7hm6LkuEKtaqYoaofLwX6xLlzPsK1N/8gR5aq6bxl7hq76TO2spGW3MhsMc2l5tZJIUYbCkH8CJQIpRJG1EzfYxi5s6KihCgELPmMTKjxuW1BAcJgro8Gcy8gI9wVOEGdtjbMEZ6jMHvK4z6utiLV4HdT0QrXOp5jcXVbKmKBHuqUmPdAdPdP7r7UaYY3AyxHPWksrnMrg6ejG27+qKs8+Dj5Vf3r2sYeF0KvB3p2QCW6ht/T147PmxuL6RGOSruiF/V/SE93zDaz6q36dE+sXiPMHKN+f+ycozGQUyii5udTSTPQV3RhHElP83vNYwhqyyPO5Ls5xiatYUxqTZCnZSpVikWYEX0Ka/gDtdY1p</latexit><latexit sha1_base64="EYzUCnncTU6pM+18fa1Q8gLZKys=">AAACcXichVHLSsNAFD2N7/qqulHchBZFsJQbFRRXgiIuW7UP0FqSOGowTUKSFrT4A/6AgisFEfEz3PgDLvwEcVnBjQtv0oCoqHeYmTNn7rlzZkZzTMPziZ5iUlt7R2dXd0+8t69/YDAxNFzw7Jqri7xum7Zb0lRPmIYl8r7hm6LkuEKtaqYoaofLwX6xLlzPsK1N/8gR5aq6bxl7hq76TO2spGW3MhsMc2l5tZJIUYbCkH8CJQIpRJG1EzfYxi5s6KihCgELPmMTKjxuW1BAcJgro8Gcy8gI9wVOEGdtjbMEZ6jMHvK4z6utiLV4HdT0QrXOp5jcXVbKmKBHuqUmPdAdPdP7r7UaYY3AyxHPWksrnMrg6ejG27+qKs8+Dj5Vf3r2sYeF0KvB3p2QCW6ht/T147PmxuL6RGOSruiF/V/SE93zDaz6q36dE+sXiPMHKN+f+ycozGQUyii5udTSTPQV3RhHElP83vNYwhqyyPO5Ls5xiatYUxqTZCnZSpVikWYEX0Ka/gDtdY1p</latexit><latexit sha1_base64="EYzUCnncTU6pM+18fa1Q8gLZKys=">AAACcXichVHLSsNAFD2N7/qqulHchBZFsJQbFRRXgiIuW7UP0FqSOGowTUKSFrT4A/6AgisFEfEz3PgDLvwEcVnBjQtv0oCoqHeYmTNn7rlzZkZzTMPziZ5iUlt7R2dXd0+8t69/YDAxNFzw7Jqri7xum7Zb0lRPmIYl8r7hm6LkuEKtaqYoaofLwX6xLlzPsK1N/8gR5aq6bxl7hq76TO2spGW3MhsMc2l5tZJIUYbCkH8CJQIpRJG1EzfYxi5s6KihCgELPmMTKjxuW1BAcJgro8Gcy8gI9wVOEGdtjbMEZ6jMHvK4z6utiLV4HdT0QrXOp5jcXVbKmKBHuqUmPdAdPdP7r7UaYY3AyxHPWksrnMrg6ejG27+qKs8+Dj5Vf3r2sYeF0KvB3p2QCW6ht/T147PmxuL6RGOSruiF/V/SE93zDaz6q36dE+sXiPMHKN+f+ycozGQUyii5udTSTPQV3RhHElP83vNYwhqyyPO5Ls5xiatYUxqTZCnZSpVikWYEX0Ka/gDtdY1p</latexit>
D,r3,r4,r4,B<latexit sha1_base64="LCMDZmLw/H5ARxmfLVwHgsPLgOI=">AAACeHichVHLSsNAFD2N7/qKuhHcFIsvkHJTBcWVqAuX2loVVEoSpzWYJiFJC7X4A/6ACxeiIFr8DDf+gIt+grhUEMSFN2lAVNQbcufMmXvunJnRHNPwfKJGTGppbWvv6OyKd/f09vXLA4Obnl12dZHTbdN2tzXVE6ZhiZxv+KbYdlyhljRTbGmHy8H6VkW4nmFbG37VEXsltWgZBUNXfaby8tDKdMLNzwRpNkpLeTlJKQoj8RMoEUgiijVbvsYu9mFDRxklCFjwGZtQ4fG3AwUEh7k91JhzGRnhusAx4qwtc5XgCpXZQ85Fnu1ErMXzoKcXqnXexeTfZWUCY/RAdXqme7qlR3r/tVct7BF4qfKoNbXCyfefDGdf/1WVePRx8Kn607OPAuZDrwZ7d0ImOIXe1FeOTp+zC5mx2jhd0hP7v6AG3fEJrMqLfrUuMmeI8wMo36/7J9hMpxRKKeuzycV09BSdGMEoJvm+57CIVawhx/tWcY4b1GNvUkKakKaapVIs0gzhS0jpD98bjxk=</latexit><latexit sha1_base64="LCMDZmLw/H5ARxmfLVwHgsPLgOI=">AAACeHichVHLSsNAFD2N7/qKuhHcFIsvkHJTBcWVqAuX2loVVEoSpzWYJiFJC7X4A/6ACxeiIFr8DDf+gIt+grhUEMSFN2lAVNQbcufMmXvunJnRHNPwfKJGTGppbWvv6OyKd/f09vXLA4Obnl12dZHTbdN2tzXVE6ZhiZxv+KbYdlyhljRTbGmHy8H6VkW4nmFbG37VEXsltWgZBUNXfaby8tDKdMLNzwRpNkpLeTlJKQoj8RMoEUgiijVbvsYu9mFDRxklCFjwGZtQ4fG3AwUEh7k91JhzGRnhusAx4qwtc5XgCpXZQ85Fnu1ErMXzoKcXqnXexeTfZWUCY/RAdXqme7qlR3r/tVct7BF4qfKoNbXCyfefDGdf/1WVePRx8Kn607OPAuZDrwZ7d0ImOIXe1FeOTp+zC5mx2jhd0hP7v6AG3fEJrMqLfrUuMmeI8wMo36/7J9hMpxRKKeuzycV09BSdGMEoJvm+57CIVawhx/tWcY4b1GNvUkKakKaapVIs0gzhS0jpD98bjxk=</latexit><latexit sha1_base64="LCMDZmLw/H5ARxmfLVwHgsPLgOI=">AAACeHichVHLSsNAFD2N7/qKuhHcFIsvkHJTBcWVqAuX2loVVEoSpzWYJiFJC7X4A/6ACxeiIFr8DDf+gIt+grhUEMSFN2lAVNQbcufMmXvunJnRHNPwfKJGTGppbWvv6OyKd/f09vXLA4Obnl12dZHTbdN2tzXVE6ZhiZxv+KbYdlyhljRTbGmHy8H6VkW4nmFbG37VEXsltWgZBUNXfaby8tDKdMLNzwRpNkpLeTlJKQoj8RMoEUgiijVbvsYu9mFDRxklCFjwGZtQ4fG3AwUEh7k91JhzGRnhusAx4qwtc5XgCpXZQ85Fnu1ErMXzoKcXqnXexeTfZWUCY/RAdXqme7qlR3r/tVct7BF4qfKoNbXCyfefDGdf/1WVePRx8Kn607OPAuZDrwZ7d0ImOIXe1FeOTp+zC5mx2jhd0hP7v6AG3fEJrMqLfrUuMmeI8wMo36/7J9hMpxRKKeuzycV09BSdGMEoJvm+57CIVawhx/tWcY4b1GNvUkKakKaapVIs0gzhS0jpD98bjxk=</latexit><latexit sha1_base64="LCMDZmLw/H5ARxmfLVwHgsPLgOI=">AAACeHichVHLSsNAFD2N7/qKuhHcFIsvkHJTBcWVqAuX2loVVEoSpzWYJiFJC7X4A/6ACxeiIFr8DDf+gIt+grhUEMSFN2lAVNQbcufMmXvunJnRHNPwfKJGTGppbWvv6OyKd/f09vXLA4Obnl12dZHTbdN2tzXVE6ZhiZxv+KbYdlyhljRTbGmHy8H6VkW4nmFbG37VEXsltWgZBUNXfaby8tDKdMLNzwRpNkpLeTlJKQoj8RMoEUgiijVbvsYu9mFDRxklCFjwGZtQ4fG3AwUEh7k91JhzGRnhusAx4qwtc5XgCpXZQ85Fnu1ErMXzoKcXqnXexeTfZWUCY/RAdXqme7qlR3r/tVct7BF4qfKoNbXCyfefDGdf/1WVePRx8Kn607OPAuZDrwZ7d0ImOIXe1FeOTp+zC5mx2jhd0hP7v6AG3fEJrMqLfrUuMmeI8wMo36/7J9hMpxRKKeuzycV09BSdGMEoJvm+57CIVawhx/tWcY4b1GNvUkKakKaapVIs0gzhS0jpD98bjxk=</latexit>
C,r2,r3,r4,F<latexit sha1_base64="V/wmOoD7UxfOA4xsf5+pnQeBojA=">AAACeHichVHLSsNAFD2Nr1pfVTeCm2DxBVJuakFxJRTEZbVWBZWSxLEG0yQkaaEWf8AfcOFCFETFz3DjD7jwE8RlBUFceJMGREW9IXfOnLnnzpkZzTENzyd6jElt7R2dXfHuRE9vX/9AcnBo3bOrri6Kum3a7qamesI0LFH0Dd8Um44r1Ipmig3tIBesb9SE6xm2tebXHbFTUcuWsWfoqs9UKTmcm5HdUiZIs0HKzshLpWSK0hSG/BMoEUghirydvMI2dmFDRxUVCFjwGZtQ4fG3BQUEh7kdNJhzGRnhusAREqytcpXgCpXZA85lnm1FrMXzoKcXqnXexeTfZaWMcXqgG2rSPd3SE73/2qsR9gi81HnUWlrhlAaORwqv/6oqPPrY/1T96dnHHuZDrwZ7d0ImOIXe0tcOT5qFhdXxxgRd0DP7P6dHuuMTWLUX/XJFrJ4iwQ+gfL/un2A9k1YoraxkU4uZ6CniGMUYpvi+57CIZeRR5H3rOMM1bmJvkixNStOtUikWaYbxJaTMB+D0jxo=</latexit><latexit sha1_base64="V/wmOoD7UxfOA4xsf5+pnQeBojA=">AAACeHichVHLSsNAFD2Nr1pfVTeCm2DxBVJuakFxJRTEZbVWBZWSxLEG0yQkaaEWf8AfcOFCFETFz3DjD7jwE8RlBUFceJMGREW9IXfOnLnnzpkZzTENzyd6jElt7R2dXfHuRE9vX/9AcnBo3bOrri6Kum3a7qamesI0LFH0Dd8Um44r1Ipmig3tIBesb9SE6xm2tebXHbFTUcuWsWfoqs9UKTmcm5HdUiZIs0HKzshLpWSK0hSG/BMoEUghirydvMI2dmFDRxUVCFjwGZtQ4fG3BQUEh7kdNJhzGRnhusAREqytcpXgCpXZA85lnm1FrMXzoKcXqnXexeTfZaWMcXqgG2rSPd3SE73/2qsR9gi81HnUWlrhlAaORwqv/6oqPPrY/1T96dnHHuZDrwZ7d0ImOIXe0tcOT5qFhdXxxgRd0DP7P6dHuuMTWLUX/XJFrJ4iwQ+gfL/un2A9k1YoraxkU4uZ6CniGMUYpvi+57CIZeRR5H3rOMM1bmJvkixNStOtUikWaYbxJaTMB+D0jxo=</latexit><latexit sha1_base64="V/wmOoD7UxfOA4xsf5+pnQeBojA=">AAACeHichVHLSsNAFD2Nr1pfVTeCm2DxBVJuakFxJRTEZbVWBZWSxLEG0yQkaaEWf8AfcOFCFETFz3DjD7jwE8RlBUFceJMGREW9IXfOnLnnzpkZzTENzyd6jElt7R2dXfHuRE9vX/9AcnBo3bOrri6Kum3a7qamesI0LFH0Dd8Um44r1Ipmig3tIBesb9SE6xm2tebXHbFTUcuWsWfoqs9UKTmcm5HdUiZIs0HKzshLpWSK0hSG/BMoEUghirydvMI2dmFDRxUVCFjwGZtQ4fG3BQUEh7kdNJhzGRnhusAREqytcpXgCpXZA85lnm1FrMXzoKcXqnXexeTfZaWMcXqgG2rSPd3SE73/2qsR9gi81HnUWlrhlAaORwqv/6oqPPrY/1T96dnHHuZDrwZ7d0ImOIXe0tcOT5qFhdXxxgRd0DP7P6dHuuMTWLUX/XJFrJ4iwQ+gfL/un2A9k1YoraxkU4uZ6CniGMUYpvi+57CIZeRR5H3rOMM1bmJvkixNStOtUikWaYbxJaTMB+D0jxo=</latexit><latexit sha1_base64="V/wmOoD7UxfOA4xsf5+pnQeBojA=">AAACeHichVHLSsNAFD2Nr1pfVTeCm2DxBVJuakFxJRTEZbVWBZWSxLEG0yQkaaEWf8AfcOFCFETFz3DjD7jwE8RlBUFceJMGREW9IXfOnLnnzpkZzTENzyd6jElt7R2dXfHuRE9vX/9AcnBo3bOrri6Kum3a7qamesI0LFH0Dd8Um44r1Ipmig3tIBesb9SE6xm2tebXHbFTUcuWsWfoqs9UKTmcm5HdUiZIs0HKzshLpWSK0hSG/BMoEUghirydvMI2dmFDRxUVCFjwGZtQ4fG3BQUEh7kdNJhzGRnhusAREqytcpXgCpXZA85lnm1FrMXzoKcXqnXexeTfZaWMcXqgG2rSPd3SE73/2qsR9gi81HnUWlrhlAaORwqv/6oqPPrY/1T96dnHHuZDrwZ7d0ImOIXe0tcOT5qFhdXxxgRd0DP7P6dHuuMTWLUX/XJFrJ4iwQ+gfL/un2A9k1YoraxkU4uZ6CniGMUYpvi+57CIZeRR5H3rOMM1bmJvkixNStOtUikWaYbxJaTMB+D0jxo=</latexit>
chain
len=3
2. generates walks by sequentially connecting
the POIs appeared in the graph
chain
len=2
chain
len=1
3. learn embedded representations
4. apply POI embeddings to different models
Traffic
PerditionService
Suggestion
anomaly
detection

Embedding Applications in DiDi - Ride-Hailing
AHINE: Adaptive Heterogeneous Information Network Embedding

Part III: Text/Graph
Embedding Applications in
Customer Service Scenarios

Intelligent Customer Services
Recommendation
technology
3. Chatbots
4. Smart ReplyIntelligent routing
Work Order System
Auto Call
Intelligent Speech
Interaction
1. Knowledge
Mining
5. Smart Summary
Knowledge Graph
2.
Recommendatio
ns
Product
Layer
Voice Technology
Basic Layer
NLPDeep Learning
Dynamic IVR
Persona
Full Quality Inspection
Subscriber
Access
Intelligent
Processing
Intelligent
Routing
Intelligent
Assist
Service
Process
User Customer
Service
AI Improve user experience and
reduce manual entry
HI Improve efficiency, reduce cost and guarantee quality
CRM System
Telephone Traffic
System
Knowledge Base
System
Data System Operation System
Intelligent
Operation
Technical
Layer

Automatic Question-Answering
9. Clustering &
Annotating
User
Knowledge
Graph
Knowledge
Mining
1. Access 4. KG query
5. Answer
8. Unclicked answer
2.Answer
10. Knowledge update
Recommend
ation
3. Ask Question
Classifier
6. Deep matching
7. Answer4. Answer
Chatbot

Pain points
• Many questions are not covered by KB
• The granularity of some issues needs to
be refined
Solution
• Mine question items from the user-agent
and user-chatbot dialogues, then analyze
mappings between mined items and the
existing items in KB
• add new items if not covered by KB
• split an item if mapped to multiple
mined items
• Learn from human agent
QA
Dialogues
Question
Extractor
Question
Embeddings
Question
Classifier
Sentence
Similarity
Nonparametric
Clustering
Knowledge
Mining
Suggestions
Knowledge Mining

Prediction & Recommendations
Step 2: Graph Embeddings
Step 1: Construct Graph
based on historical data
Historical
interactive
Historical
recommendation
information
……
User Embedding
Session
Embedding
Online
q
q
1
q
2
t
1
t
2
Step 3: Session based Graph Neural Network (GNN)

Chatbot-Intent Understanding
Question Why I haven’t got my surcharge fee back?
• Existing business units have plenty of training data and
achieve good results
• New business units are lack of annotated data
Multi-task Shared EncoderSeparate Encoder
Task: understand the intentions of humans and extraction of
relevant information

Smart Reply
Smart Reply
• help customer service agents quickly reply to incoming inquiries

Smart Reply
D5 A1 5
5 1 35 A5 1 5 C 4
5 1 35
5 1 4 1 4 C 3 1AA 1 1
5 1 A : . 5 31 A 51 C: 1HC
5 1 35
A5 5 5A C 1D5 3 13 5A 31 5 ?
5 ? 13 35 4 C
5 1 35 ! 5 3 C A5
5 1 35 A5 1 1D5 1 55 A 5AA
5A? A5 1 4 41 5A
C 5 1D5 C 5D5 ? 1 54 4 C 25 5
1 5AA A 4 C 1
Key task:
Given a conversation, choose the best
matching answer from the candidate
answers
Multi-turn conversation modeling

Smart Reply
Multi-view Response Selection for Human-Computer Conversation
Word Sequence Model: utterances of context are connected as a
sequence of words
context embeddings response embeddings
Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
Utterance sequence model: regards the context as a hierarchical structure.

Dialog Summary
Hello, Is there anything I can do for you?
Please help me check whether my Didi account is activated.
You may need to wait a little longer. The platform is updating, and will
gradually serve after the upgrade. Please notice the Didi announcement
or short messages.
Okay, thanks.
I have finished the registration and passed the verification.
You are welcome, and it is my pleasure to help you.
The driver consulted the activation of
the new driver, and I explained that the
platform was updating, and advised the
driver to pay attention to the
notification. The driver approved.

Dialog Summary – Methods
Three years ago, U.S. health officials warned
hundreds of thousands of clinicians in hospitals
around the country to be on the lookout for
a new, quickly spreading and highly drug-
resistant type of yeast that was causing
potentially fatal infections in hospitalized
patients around the world.
Three years ago, health officials warned
clinicians.
U.S health officials warned clinicians about
a yeast which is fatal.
Extractive
Abstractive
Dialog summary
• Speakers of utterances
• Interactions between speakers
1. Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue
summarization with sentence-gated modeling optimized by dialogue
acts. arXiv preprint arXiv:1809.05715 (2018).
Hierarchical text encoder

Dialog Summary
• Take the key points as auxiliary labels to ensure the
integrity.
• A key point sequence to ensure logic.
• Merge the important facts into the short auxiliary
sequence to ensure correctness.
• Each point guides an independent sub-summary to
get the final summaries.

Dialog Summary
-
!̇#,%
-
& -
'#̇
!̇#,( !̇#,) !̇#,*
!#,%+,- !#,( !#,) !#,*
∑
!0#,% !0#,( !0#,) !0#,*Att Att AttAtt
-
'̇( '̇) '̇* '̇1
-
234 235 236 237
-
-
8̇9+9 8̇( 8̇) 8̇*
89+9 8( 8) 8*
-
80*809+9 80( 80)
:*
;
8*
;
-
-
-
<̇#,; <̇#,( <̇#,) <̇#,*
<#,( <#,) <#,*
-
<̂#,*<̂#,; <̂#,( <̂#,)
:*
9
<#,*
;
!0(,(
!0(,)
!0(,*
!0(,1
>?@+A-B
>?C@@
<̅#,*
∑>?#,E
F
:*
G
× × ∑
8I-B 1 − 8I-B
-
Token-level Encoder Utterance-level Encoder Leader Net Pointer Writer Net
-
- -
• Token-level Encoder->context-free representations
• Transformer encoder
• Utterance-level representation via attention
• Utterance-level Encoder->contextual representations
• Transformer encoder
• Dialogue position encoding: normalize the dialogue length
• Leader Net: Decode the key point sequence
• Standard Transformer decoder
• The decoded representation is used as the initial state of
the Writer net.
• Writer Net: Given a key point, decoding the corresponding sub-
summaries.
Key Ideas:
• Hierarchical Encoder: word-token level and utterance
level;
• Leader-Writer Decoder: hierarchical decoder, key point
level and sub-summary level;

Speech Recognition Outline
• Speech recognition: classic methods
• Speech recognition: deep learning approaches
• From GMM to DNN
• From HMM to CTC
• Speech recognition: attentional approaches
• Attention based approaches
• Recent trends: transformer
• Related topics
• Noise and far-field

• Speech signal -> Transcripts
• Three main parts:
• Acoustic Model: ! " #
• Language Model: ! #
• Decoder: arg max (⋅)
Speech recognition: basic concepts
ASR “Hello World”
,# = arg max
.
! # " = arg max
.
! # ! " #
! #
= arg max
.
! # ! " #

Speech recognition: basic concepts
Speech
Features
Decoder
Language Model Acoustic Model
Dictionary
Dataset
Word Sequence

• Language Model
• ! "
•
• Decoder:
•
•
Speech recognition: classic methods
! " = ! $%, $', … , $) = *
+,%
)
!($+|ℎ+)
= ! $% ! $' $% ! $1 $%, $' … ! $) $%, $', … , $)
= *
+,%
)
! $+ $%, $', … , $+2%
≈ ! $4 $42 )2% , $42 )2' , … , $42%

Acoustic Models: Variability
Context
Variability
Style
Variability
Speaker
Variability
Environment
Variability

• GMM-HMM based Acoustic Model
•
•
Acoustic Models: GMM-HMM
GMMs
HMMs
Transition probability
Output probability

• Acoustic model: mapping the speech feature into acoustic unit
• The choice of acoustic modeling units
•
•
•
•
•

• Context dependency
•
•
•

• The introduce of DNN in speech recognition
Speech recognition: deep learning approaches
CI-DNN-HMM(RBM pretrained):

• DNN replace GMM: still using HMM
Acoustic Models: DNN-HMM
HMMs
Transition probability
Output probability
…
…
…
DNN
Hidden
Hidden
Hidden

• DNN replace GMM: still using HMM
• DNN output the posterior probability
!"#
$ = & '( = )* +(
• Using a pseudo likelihood in the HMM framework
& +( '( = )* =
& '( = )* +( & +(
& )*
≅
!"#
$
& )*
Acoustic Models: DNN-HMM

• The input feature:
• Trying to remove the hand-crafted features: MFCC -> FBANK
• Maybe: waveform
• Various neural network structures
• Feedforward, Convolutions, Recurrent
DNN-HMM ASR

• The rise of end-to-end learning
• Replacing pipeline systems with a single learning algorithm
• Go directly from the input to the desired output
The rise of end-to-end learning
outputinput
Neural Network
TranscriptsAudio
Neural Network

• Hybrid: LSTM-HMM
• Connectionist Temporal Classification (CTC)
• Introduce the blank label
a b c = blank a a b blank c c c blank
= blank a blank b b blank c blank
= blank a a blank b b c c blank
= …
CTC based speech recognition
! !
! !

• Objective function of CTC is defined as the negative log probability of
correctly labelling the entire training set:
!"#" = − ln (
),+ ∈-
. + ) = − /
),+ ∈-
ln . + )
• Forward and backward variables used for accelerated the calculating
the objective function
• Similar to the forward-backward algorithm of DNN-HMM, but using different
topology
Connectionist Temporal Classification (CTC)

• Map input feat to output symbol (maybe blank)
• Do not need pre-alignment
• Conditional independent assumption
• Possible output peak delay
• Main difference
• Topology
Connectionist Temporal Classification (CTC) vs. HMM

•Modeling units in CTC ASR:
• Some systems use One-state tied tri-phone
• Trying to perform end-to-end
• For English: using Grapheme,
• For Mandarin: Characters or Syllables
•Input features in CTC ASR:
• Still using FBank
• 3-fold down-sampling, so 30 ms each frame
Connectionist Temporal Classification (CTC) vs. HMM

Speech recognition: from GMM to end-to-end
Input
audio
Output
text
aeiou……bpmf
Weather
HMM
• GMM-HMM ASR
• Acoustic Model GMM-HMM
• Trained using acoustic training set
• Using GMM to model the
distributions of HMM states
• Dictionary Word->Phoneme
• Language Model N-gram
• Trained using text corpus
L.R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of
the IEEE, 1989

• DNN-HMM ASR
• Acoustic Model DNN-HMM
• Trained using acoustic training set
• Using deep learning
• Output HMM states
• Dictionary Word->Phoneme
• Language Model N-gram
• Trained using text corpus
George Dahl, Dong Yu, Li Deng, Alex Acero, Context-dependent pre-trained deep neural networks for large
vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing. 2012
Weather

CTC ASR
◦ Acoustic Model RNN-CTC
◦ Trained using acoustic training set
◦ Using deep sequence learning
◦ Output grapheme for English, syllables
for Mandarin
◦ Dictionary Word->Phoneme
◦ Language Model N-gram
◦ Trained using text corpus
Weather

Input
audio
Output
text
…………
Attention Decoder
Attentional ASR
◦ Acoustic Model RNN-
Attention
◦ Trained using acoustic training set
◦ Using deep sequence learning
◦ Output characters / phonemes
◦ Dictionary
◦ Language Model
Weather

Attentional ASR
• Dictionary
• The modeling units for Mandarin Chinese ASR
• Characters are usually selected as the basic modeling units
• Language Model
• How to benefit from the large text corpus without N-gram ?
• We pre-train RNN-LM and then merged into acoustic neural network

• End-to-end is a relative concept
End-to-end speech recognition
phoneme syllable/character
DNN-HMM
We need decision-tree based
state clustering, dictionary,
language model
RNN-CTC
We need dictionary, language
model,
(If we use the cd-phone as
modeling units, we still need
decision-tree based state
clustering)
The N-gram based language
models would improve the
performance
RNN-Attention We do not need extra models

• Sequence-to-sequence model from translation
Attentional ASR

• Same structure with
Bahdanau’s neural translation
model
First Attention in Speech

• Encoder
• Listen, map the input feature
sequence to embedding
• Decoder
• Spell, map the embedding based on
the attention information to the
output symbols
Listen-Attend-Spell

• Advantages
• There is no conditional independence assumptions
• Joint learning of acoustic information and language information
• Speech recognition system is more simple
• Disadvantages
• Not easy to converge, We need more tricks to train attention model
• Cannot be used for “streaming” speech recognition, during inference, the
model can produce the first output token only after all input speech frames
have been consumed.
Attention vs. CTC

• Hard to train – many “tricks”
• Schedule sampling
• Label smoothing (2016)
Listen-Attend-Spell

• Multi-Task Learning (2017)
• Joint CTC-attention based end-to-
end framework
• The shared encoder is trained by
both CTC and attention model
objectives simultaneously.
Listen-Attend-Spell

• Multi-headed Attention (2018)
• Inspired by transformer
• Replacing single head attention
Listen-Attend-Spell

• Multi-headed Attention (2018)
• SpecAugment (2019)
• Data augmentation to LAS
• Achieved sota results on Librispeech and
SWBD
Listen-Attend-Spell

• A limited sequence
streaming attention-based
model
• Consumes a fixed number
of input frames (a chunk)
• Outputs a variable number
of labels before it
consumes the next chunk
Online neural transducer

• Speech Transformer
• Transformer applied to ASR
• With Conv layers as inputs
Speech-Transformer

• Speech Transformer
• Transformer applied to ASR
• With Conv layers as inputs
• Time-restricted self-attention
• Left & Right Contexts restricting the attention mechanism
Speech-Transformer

• Pre-training:
• Like BERT in NLP, e.g. Mask Predictive
Coding
• Fine-tuning:
• Plug in a decoder
Unsupervised pre-training for speech-transformer

• Mask Predictive Coding:
• mask 15% of all frames in each sequence at
random, and only predict the masked frame rather
than reconstructing the entire input
• Dynamic Masking:
• Like RoBERTA, masking strategies are not decided in
advance
• Down-sampling:
• Local smoothness of speech makes learning too
easy without down-sampling. Eight-fold down-
sampling is used, like LAS.

Related topics: signal processing for noise and far-field
AEC De-reverb BSS
BeamformingNSAGC
Fixed filter
Fixed filter
……
……
BM
( )0x k
( )1x k
( )1Nx k-
Adap Filter
Fixed filter
Adap Filter
Adap Filter
Å
ÅÅ Z-L +
-
( )0u k
( )1u k
( )Mu k
( )y k

-- Acoustic Echo Cancellation
-- Noise suppression
-- Beamforming / Blind source separation
-- Auto Gain Control
original speech
processed speech
Note:
The reasons for this dive seemed foolish now. His captain was thin
and haggard and his beautiful boots were worn and shabby.
Production may fall far below expectations.

Single channel farfield ASR Multi-channel farfield ASR
with ULA
Multi-channel farfield ASR
with UCA
reflection
noise & interference
beamforming
UCAULA
direct arrival mainlobe
sidelobe

* =
near-field speech simulated
farfield speech
room impulse response
&
noise and interference
real
farfield speech

Outline
• Conversational understanding
• Overview: from a dialogue system
• Intent recognition
• Slot filling
• Remarks
• Multimodality for conversation: speech + text
• Emotion recognition
• End-to-end conversational understanding
• DELTA: a deep learning based language technology platform

What is conversational understanding
Task-oriented dialogue system
- -
-
• - / - -
• -
• /
• -
-
• Conversational /natural language understanding
• Extract “meaningful” information from natural language in the context of
conversations.

What is conversational understanding
•
•
•
•
•
à

Intent recognition: rule-based method
• Hand-crafted rules based on keywords or regex. For example,
• Data:
• Intent: play_music_by_artist
Regular expression:
Can also do slot filling: artist music_title

Regular expression
• Error analysis:
• Less/more constraints à False positive / missing
• Conflict
• Intent1: play_music_by_artist
• Intent2: play_movie_by_title
• Sentence:
• Regular expressions needs a lot of human effort, but it play a important
role in real application
• Cold start / bootstrapping
• Used as features

Intent recognition: from a classification standpoint
• Conversational text à category
• Input: a piece of text X of length N, X=<x1, …, xN>
• Output: one of k labels y
“Play The Sound of
Music movie”
Representation Model
play_movie_by_title: 0.6
play_movie_by_genre: 0.2
play_movie_by_director: 0.1
resume_playback: 0.05
pause: 0.05
• Bag of words
• Embedding
• Naïve Bayes
• Logistic regression
• SVM
• Neural network
…

Classification: Naïve Bayes
• Naïve Bayes relies on bag of words
• Other models: logistic regression, SVM, decision tree, …
“Naïve” assumption: P(xi|c) are independent given the class c
Prior probabilitylikelihood
• For a document d and a class c, the predicted class is:

Classification with embedding
……
vocabulary
Embedding
dimension
Book a flight ticket from New York to Beijing
Look-up table
sentence vector:
end2end model: CNN/RNN/…
pretrained: doc2vec/skip-thought/BERT
statistics: average pooling
Classification
Embedding can be trained/updated too
transformationa
aachen
aaron
aaronic
aarp
aave
ab
abac
abaci
……

A further step: N-gram embedding
• Word embedding + N-gram embedding
• N-gram: O(|V|N) à hashing trick
• FastText: average of N-gram embedding (Joulin et al., ACL 2017)
1 2 3 … V-1 V
Embedding
dimension
N-gram
N-gram embeddingword embedding
hash
It is recommended as a baseline before considering more complex approaches

Neural networks methods: Convolutional networks
• CNN for classification (Kim, 2014)
• Each column represents a token (embedding vector)
• 1-D convolution
• CNN filters à N-grams
multiple filters with trigram window
Several window sizes (e.g, 2, 3, 4),
Each has multiple filters (e.g, 100)
sliding
windows

Neural networks methods: Recurrent networks
Lai et.al. AAAI 2015
https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9745/9552
RNN: a natural choice for sequence modeling. Bidirectional RNN
models two directions. (Lai et al, 2015)
RNN
w1 w2 w3 w4 w5 w6 w7
Concat
NN y

Pooling
Neural networks methods: Recurrent networks
Lai et.al. AAAI 2015
https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9745/9552
RNN
w1 w2 w3 w4 w5 w6 w7
Pooling
NN y
RNN: a natural choice for sequence modeling. Bidirectional RNN
models two directions. (Lai et al, 2015)

Neural networks methods: CNN + RNN
Zhou et.al. arxiv 2015
CNN: phrase-level features
RNN: sequences of higher level representations
CNN+RNN: (Zhou et al., 2015; Lee and Dernoncourt, 2016; Xiao and
Cho, 2016)

Aggregate sequence information
• Max pooling / Average
• Attention for sequence classification
(Raffel and Ellis, ICLR workshop, 2016)
• Attention: helps models handle very long and
widely variable-length sequences.

Dialogue modeling
Word
Encoder
Word
Attention
Sentence
Encoder
Sentence
Attention
Prediction
Attentive
Parameter
sentence
!" !# !$
!%
…sentenc
e
sentence sentence
Attentive
Parameter&" &# &$
&%
Document
vector
softmax• Dialogue: understanding in the
context of dialogue
• From single sentence to multiple
sentences
• Multi-turn: intent recognition with
historic information
• Hierarchical attention networks
• Tang et al., 2015; Yang et al., 2016

Semantic matching
• Why need semantic matching?
• What if the target labels have semantic meaning? For example, QA
• What if the target labels change frequently?
Transformation
Transformation
Metric
query
intent1
intent2
intent3
intent2
intent3
intent1
Ranking scores
e.g. cosine similarity
0.8
0.7
0.5

Practices: Which model should I use?
• Which model?
• Little labeled data à Keywords/Regex (automatic labeling)
• Small amount of labeled data (< 1000) àsimple classifier: Naïve Bayes, LR, …
• Large amount of labeled data (>1000)àEmbedding + NN
• FastText / CNN/ RNN
• Utilize huge amount of unlabeled data
• Pretrained model: ELMO/BERT/…

Slot filling
• Sequence labeling problem: given a sentence find tag sequences.
• BIO labeling
• “Book a flight ticket from New York to Beijing on November 20”
O O O O O B-orig I-orig O B-dest O B-date I-date
Slot filling v.s. Named entity recognition (NER):
B-ORIG à B-LOCATION

Slot filling: CRF
• Hidden Markov models: a generative model for sequence data
• Conditional Random Field: a discriminative model for sequence data
(Lafferty et al., ICML, 2001)
http://homepages.inf.ed.ac.uk/csutton/publications/crftut-fnt.pdf
directly model the conditional
distribution
model the joint probability

Slot filling: Linear-chain CRF
• Linear-chain CRF: the label at time step t depends on the observation
sequence and the label in the previous time step t-1
• Maximize the log probability log p(y|x) w.r.t. parameters
http://homepages.inf.ed.ac.uk/csutton/publications/crftut-fnt.pdf
fi: feature function, e.g., fi =1 if yi-1 is a noun and yi is a verb

Slot filling: Neural networks
• Neural networks
• LSTM: Yao et al., 2014; Mesnil, 2015; Liu and Lane, 2015
• LSTM+CNN: Chio and Nicols, TACL 2016
RNN
w1 w2 w3 w4 w5 w6 w7
y1 y2 y3 y4 y5 y6 y7

Slot filling
• Neural networks + CRF
• Huang et al., 2015; Lample, NAACL, 2016; LSTM+CNN+CRF, Ma and Hovy, ACL,
2016

Multitask: intent recognition + slot filling
• The slots often highly depend on the intent
• LSTM + max-pooling/attention (Hakkani-Tur et al., Interspeech, 2016; Zhang
and Wang, IJCAI, 2016; Li et al., EMNLP, 2018; Goo et al., NAACL, 2018)
• Encoder-decoder, Liu and Lane, Interspeech, 2016

Multimodality: speech + text
Speech TextASR model NLP model Intent/slot…
Can we utilize speech information or even build an end-to-end model?

Multimodal speech emotion recognition
• Emotion recognition: identify the emotional state of a human being
from his or her voice.
• audio signals à speech emotion recognition
• transcribed text à text emotion recognition
• Multimodal methods:
https://www.cs.cmu.edu/~morency/MMML-Tutorial-ACL2017.pdf
That is great. You look radiant!
But it ignores the temporal relationship between speech and text

Learning alignment between speech and text
• Utilize an attention network to learn
the alignment between speech and text
(Xu et al., Interspeech, 2019)
• Concatenate the aligned feature to
multimodal feature

CU from speech
• How to deal with ASR errors?
• Hakkani-Tur et al, 2006; Schumann and
Angkititrakul, ICASSP 2018; Zhu et al, ICASSP
2018; Huang and Chen, 2019
• Do we really need ASR?
• Serdyuk et al., ICASSP, 2018; Haghani et al., ICASSP, 2018; Chen et al., ICASSP 2018;
Qian et al., ASRU 2017; Chen et al., ICASSP 2018; Lugosch et al., Interspeech, 2019

Conquer ASR errors: learn the errors
• Train a LM (ELMO/BERT) based on ASR-generated text helps (Huang and Chen,
2019)
• Confusion-aware ELMO using word confusion network
• Better robustness to ASR errors

End-to-end CU: initialization with speech
• Direct end-to-end
• Serdyuk et al., ICASSP, 2018; Haghani
et al., ICASSP, 2018
• Initialization with a speech model
• Pretrain ASR model (Qian et al., ASRU
2017; Chen et al., ICASSP 2018;
Lugosch et al., Interspeech, 2019)
pretraining

DELTA: a deep learning based language technology platform
A uniform platform for modeling speech
and text data
https://github.com/didi/delta
01
02
0304
05
Support NLP tasks
Support speech tasks
Multimodal and
numerical features
Easy and fast
deployment
Speed up research and
development cycle
DELTA

DELTA: Configurable pipeline
Easy to use Easy to deploy Easy to develop
• Many NLP and speech tasks
• Off-the-shelf models
• Multimodal features
• Configurable tasks
• Identical interface for
training and inference
• Flexible platform for
deployment on different
environments
• Simple pipeline for modeling
• Modularized components for
easy development
• Fully tested modules
Training DeploymentData Model
configuration configuration
Serving

DELTA: Training and deployment
Graph Adapter
CPU GPU
TF Graph TF Lite TensorRT
Executable file
Distillation Quantization
Model C API
iOS
Android
Epoch
Mini Batch
Adagrad
Adadelta
SGD
AUC
ROUGE
Acc
CNN
RNN
MLP
Attention
Learning
rate
ModelData
Adam
Transformer
MSE
Cross
Entropy
Hinge Loss
Text Classification
Sequence
Labeling
Sentence Matching Seq2seq
Speaker
Verification
Keyword
Spotting
Speech
Recognition
Multitask Learning
Emotion
Recognition
Multimodal
Learning
BLUE
Model training
Model deployment
User-defined
Configuration

DELTA example: Hierarchical attention networks
Configurable model training
# complained in 7
days
word BiLSTM
Attention
word BiLSTM
word BiLSTM
word BiLSTM
word BiLSTM
Text file
# cancel in 7 days
pickup time
…
S
O
F
T
M
A
X
Dense feature file
User-prepared files
C: Hello
A: Hello, this is DiDi customer servcie
C: I lost my stuffs in the car. …..
A: Of course. Would you let me know …
C: Sure. I took DiDi yesterday afternoon …
...

DELTA: benchmarks
You are welcome to contribute to https://github.com/didi/delta

Open Datasets
KDD Cup 2017
Highway Tollgates Traffic
Flow Prediction
GAIA Open Dataset
Trajectory and OD data
Uber Movement
Federal Highway Administration
Next Generation Simulation (NGSIM) Program
Public Data

GAIADiDi Academia
GAIA Open Dataset

Route Planning
Smart Transportation
Traffic Status Prediction
Analysis of Traffic Condition
Analysis of Driving Behavior
……
Civil
Engineering
Transportation
Resources and
Environmental
Communications Engineering
Others include
psychology journalism
etc.
Mathematics
Computer
Science
Economics and
Management
GAIA Review

Trajectory
Data
Point of Interest
(POI) Retrieval Data
D2-City Large-Scale Driving Video Data
GAIA Open Dataset GAIA.didichuxing.com/en
3000+ application from
660+ universities and
research institutes from
30 countries.
Travel Time Index Data
GAIA.didichuxing.com/en

Data
DiDi Academia
Scenarios
To Redefine the Future of Mobility

Together, we can transform transportation.

ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications

Recommended

Recommended

More Related Content

Similar to ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications

Similar to ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications (20)

Recently uploaded

Recently uploaded (20)

ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications