Download to read offline
![Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions
for Punctuation Restoration
Seokhwan Kim
Adobe Research, San Jose, CA, USA
Punctuation Restoration
Aims to generate the punctuation marks from the
unpunctuated ASR outputs
A sequence labelling task
to predict the punctuation label yt for the t-th timestep
in a given word sequence X = {x1 · · · xt · · · xT }
where C ∈ {‘, COMMA , ‘.PERIOD , ‘?QMARK }
yt =
c ∈ C if a punctuation symbol c is located
between xt−1 and xt,
O otherwise,
Examples
Input: so if we make no changes today what does tomorrow
look like well i think you probably already have the picture
Output: so , if we make no changes today , what does
tomorrow look like ? well , i think you probably already have
the picture .
t xt yt
1 so O
2 if ,COMMA
3 we O
4 make O
5 no O
6 changes O
7 today O
8 what ,COMMA
9 does O
10 tomorrow O
11 look O
t xt yt
12 like O
13 well ?QMARK
14 i ,COMMA
15 think O
16 you O
17 probably O
18 already O
19 have O
20 the O
21 picture O
22 </s> .PERIOD
Related Work
Multiple fully-connected hidden layers [1]
Convolutional neural networks (CNNs) [2]
Recurrent neural networks (RNNs) [4]
RNNs with neural attention mechanisms [3]
Main Contributions
Context encoding from the stacked recurrent layers to
learn more hierarchical aspects
Neural attentions applied on every intermediate
hidden layer to capture the layer-wise features
Multi-head attentions instead of a single attention
function to diversify the layer-wise attentions further
Methods
Model Architecture
yt
st-1st-1
stst
∙∙∙∙∙∙
n-bidirectional recurrent layers
AttentionAttention AttentionAttention AttentionAttention
m-heads mm mm
Attention Attention Attention
m-heads m m
x1 ∙∙∙h1
1h1
1 h2
1h2
1 hn
1hn
1x1 ∙∙∙h1
1 h2
1 hn
1
xT ∙∙∙h1
Th1
T h2
Th2
T hn
Thn
TxT ∙∙∙h1
T h2
T hn
T
h2
th2
txt ∙∙∙h1
th1
t hn
thn
th2
txt ∙∙∙h1
t hn
t
∙∙∙
∙∙∙∙∙∙
∙∙∙∙∙∙
∙∙∙∙∙∙
∙∙∙
∙∙∙
∙∙∙
∙∙∙
∙∙∙
∙∙∙∙∙∙
∙∙∙∙∙∙
∙∙∙∙∙∙
∙∙∙
∙∙∙
∙∙∙
∙∙∙
Word Embedding
Each word xt in a given input sequence X is represented as a
d-dimensional distributed vector xt ∈ Rd
Word embeddings can be learned from scratch with random
initialization or fine-tuned from pre-trained word vectors
Bi-directional Recurrent Layers
Stacking multiple recurrent layers on top of each other
−→
h i
t =
GRU xt,
−→
h 1
t−1 if i = 1,
GRU hi−1
t ,
−→
h i
t−1 if i = 2, · · · n,
The backward state
←−
h i
t is computed also with the same way
but in the reverse order from the end of the sequence T to t.
Both directional states are concatenated into the output state
hi
t =
−→
h i
t,
←−
h i
t ∈ R2d
to represent both contexts together
Uni-directional Recurrent Layer
The top-layer outputs [hn
1, · · · , hn
T ] are forwarded to a
uni-directional recurrent layer
st = GRU (hn
t , st−1) ∈ Rd
,
Multi-head Attentions
The hidden state st constitutes a query to neural attentions
based on scaled dot-product attention
Attn (Q, K, V) = softmax
QKT
√
d
V,
Multi-head attentions are applied for every layer separately to
compute the weighted contexts from different subspaces
fi,j
= Attn S · Wi,j
Q , Hi
· Wi,j
K , Hi
· Wi,j
V ,
where S = [s1; s2; · · · ; sT ] and Hi
= hi
1; hi
2; · · · ; hi
T
Softmax Classification
yt = softmax st, f1,1
t , · · · , fn,m
t Wy + by
Experimental Setup
IWSLT dataset
English reference transcripts of TED Talks
2.1M, 296k, and 13k words for training,
development, and test on reference, respectively
Model Configurations
Number of recurrent layers: n ∈ {1, 2, 3, 4}
Number of attention heads: m ∈ {1, 2, 3, 4, 5}
Attended contexts: layer-wise or only on top
Implementation Details
Word embeddings: 50d GloVe on Wikipedia
Optimizer: Adam optimizer
Loss: Negative log likelihood loss
Hidden dimension d = 256
The same dimension across all the components
Except the word embedding layer
Mini-batch size: 128
Dropout rate: 0.5
Early stopping within 100 epochs
Results
Comparisons of the performances with
different network configurations
Single-head
Top-layer
Single-head
Layer-wise
Multi-head
Top-layer
Multi-head
Layer-wise
0.62
0.63
0.64
0.65
0.66
0.67
0.68
F-measure
1 layer 2 layers 3 layers 4 layers
Results
COMMA PERIOD QUESTION OVERALL
Models P R F P R F P R F P R F SER
DNN-A [1] 48.6 42.4 45.3 59.7 68.3 63.7 - - - 54.8 53.6 54.2 66.9
CNN-2A [1] 48.1 44.5 46.2 57.6 69.0 62.8 - - - 53.4 55.0 54.2 68.0
T-LSTM [2] 49.6 41.4 45.1 60.2 53.4 56.6 57.1 43.5 49.4 55.0 47.2 50.8 74.0
T-BRNN [3] 64.4 45.2 53.1 72.3 71.5 71.9 67.5 58.7 62.8 68.9 58.1 63.1 51.3
T-BRNN-pre [3] 65.5 47.1 54.8 73.3 72.5 72.9 70.7 63.0 66.7 70.0 59.7 64.4 49.7
Single-BiRNN [4] 62.2 47.7 54.0 74.6 72.1 73.4 67.5 52.9 59.3 69.2 59.8 64.2 51.1
Corr-BiRNN [4] 60.9 52.4 56.4 75.3 70.8 73.0 70.7 56.9 63.0 68.6 61.6 64.9 50.8
DRNN-LWMA 63.4 55.7 59.3 76.0 73.5 74.7 75.0 71.7 73.3 70.0 64.6 67.2 47.3
DRNN-LWMA-pre 62.9 60.8 61.9 77.3 73.7 75.5 69.6 69.6 69.6 69.9 67.2 68.6 46.1
so
if
wemake
nochangestodaywhatdoestomorrow
look
likewell
ithink
youprobablyalreadyhave
thepicture
so
if
we
make
no
changes
today
what
does
tomorrow
look
like
well
i
think
you
probably
already
have
the
picture
so
if
wemake
nochangestodaywhatdoestomorrow
look
likewell
ithink
youprobablyalreadyhave
thepicture
so
if
we
make
no
changes
today
what
does
tomorrow
look
like
well
i
think
you
probably
already
have
the
picture
so
if
wemake
nochangestodaywhatdoestomorrow
look
likewell
ithink
youprobablyalreadyhave
thepicture
so
if
we
make
no
changes
today
what
does
tomorrow
look
like
well
i
think
you
probably
already
have
the
picture
so
if
wemake
nochangestodaywhatdoestomorrow
look
likewell
ithink
youprobablyalreadyhave
thepicture
so
if
we
make
no
changes
today
what
does
tomorrow
look
like
well
i
think
you
probably
already
have
the
picture
References
[1] Xiaoyin Che, ChengWang, Haojin Yang, and Christoph Meinel, ‘Punctuation prediction for unsegmented transcript based on word vector’, LREC 2016.
[2] Ottokar Tilk and Tanel Alumae, ‘LSTM for punctuation restoration in speech transcripts’, INTERSPEECH 2015.
[3] Ottokar Tilk and Tanel Alumae, ‘Bidirectional recurrent neural network with attention mechanism for punctuation restoration’, INTERSPEECH 2016.
[4] Vardaan Pahuja et al., ‘Joint learning of correlated sequence labelling tasks using bidirectional recurrent neural networks’, INTERSPEECH 2017.
345 Park Avenue, San Jose, CA 95110, USA Email: seokim@adobe.com](https://image.slidesharecdn.com/icassp2019kimposter-190514212129/75/Deep-Recurrent-Neural-Networks-with-Layer-wise-Multi-head-Attentions-for-Punctuation-Restoration-1-2048.jpg)
Uploaded bySeokhwan Kim
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punctuation Restoration
The document describes a deep recurrent neural network model with multi-head attention mechanisms for punctuation restoration. The model stacks multiple bidirectional recurrent layers to encode context and applies multi-head attention to each layer to capture hierarchical features. Evaluated on an English speech transcription dataset, the proposed model outperforms previous methods that use convolutional or recurrent neural networks alone or with single-head attention.
More Related Content
Similar to Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punctuation Restoration
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punctuation Restoration
- 1. Deep Recurrent NeuralNetworks with Layer-wise Multi-head Attentions for Punctuation Restoration Seokhwan Kim Adobe Research, San Jose, CA, USA Punctuation Restoration Aims to generate the punctuation marks from the unpunctuated ASR outputs A sequence labelling task to predict the punctuation label yt for the t-th timestep in a given word sequence X = {x1 · · · xt · · · xT } where C ∈ {‘, COMMA , ‘.PERIOD , ‘?QMARK } yt = c ∈ C if a punctuation symbol c is located between xt−1 and xt, O otherwise, Examples Input: so if we make no changes today what does tomorrow look like well i think you probably already have the picture Output: so , if we make no changes today , what does tomorrow look like ? well , i think you probably already have the picture . t xt yt 1 so O 2 if ,COMMA 3 we O 4 make O 5 no O 6 changes O 7 today O 8 what ,COMMA 9 does O 10 tomorrow O 11 look O t xt yt 12 like O 13 well ?QMARK 14 i ,COMMA 15 think O 16 you O 17 probably O 18 already O 19 have O 20 the O 21 picture O 22 </s> .PERIOD Related Work Multiple fully-connected hidden layers [1] Convolutional neural networks (CNNs) [2] Recurrent neural networks (RNNs) [4] RNNs with neural attention mechanisms [3] Main Contributions Context encoding from the stacked recurrent layers to learn more hierarchical aspects Neural attentions applied on every intermediate hidden layer to capture the layer-wise features Multi-head attentions instead of a single attention function to diversify the layer-wise attentions further Methods Model Architecture yt st-1st-1 stst ∙∙∙∙∙∙ n-bidirectional recurrent layers AttentionAttention AttentionAttention AttentionAttention m-heads mm mm Attention Attention Attention m-heads m m x1 ∙∙∙h1 1h1 1 h2 1h2 1 hn 1hn 1x1 ∙∙∙h1 1 h2 1 hn 1 xT ∙∙∙h1 Th1 T h2 Th2 T hn Thn TxT ∙∙∙h1 T h2 T hn T h2 th2 txt ∙∙∙h1 th1 t hn thn th2 txt ∙∙∙h1 t hn t ∙∙∙ ∙∙∙∙∙∙ ∙∙∙∙∙∙ ∙∙∙∙∙∙ ∙∙∙ ∙∙∙ ∙∙∙ ∙∙∙ ∙∙∙ ∙∙∙∙∙∙ ∙∙∙∙∙∙ ∙∙∙∙∙∙ ∙∙∙ ∙∙∙ ∙∙∙ ∙∙∙ Word Embedding Each word xt in a given input sequence X is represented as a d-dimensional distributed vector xt ∈ Rd Word embeddings can be learned from scratch with random initialization or fine-tuned from pre-trained word vectors Bi-directional Recurrent Layers Stacking multiple recurrent layers on top of each other −→ h i t = GRU xt, −→ h 1 t−1 if i = 1, GRU hi−1 t , −→ h i t−1 if i = 2, · · · n, The backward state ←− h i t is computed also with the same way but in the reverse order from the end of the sequence T to t. Both directional states are concatenated into the output state hi t = −→ h i t, ←− h i t ∈ R2d to represent both contexts together Uni-directional Recurrent Layer The top-layer outputs [hn 1, · · · , hn T ] are forwarded to a uni-directional recurrent layer st = GRU (hn t , st−1) ∈ Rd , Multi-head Attentions The hidden state st constitutes a query to neural attentions based on scaled dot-product attention Attn (Q, K, V) = softmax QKT √ d V, Multi-head attentions are applied for every layer separately to compute the weighted contexts from different subspaces fi,j = Attn S · Wi,j Q , Hi · Wi,j K , Hi · Wi,j V , where S = [s1; s2; · · · ; sT ] and Hi = hi 1; hi 2; · · · ; hi T Softmax Classification yt = softmax st, f1,1 t , · · · , fn,m t Wy + by Experimental Setup IWSLT dataset English reference transcripts of TED Talks 2.1M, 296k, and 13k words for training, development, and test on reference, respectively Model Configurations Number of recurrent layers: n ∈ {1, 2, 3, 4} Number of attention heads: m ∈ {1, 2, 3, 4, 5} Attended contexts: layer-wise or only on top Implementation Details Word embeddings: 50d GloVe on Wikipedia Optimizer: Adam optimizer Loss: Negative log likelihood loss Hidden dimension d = 256 The same dimension across all the components Except the word embedding layer Mini-batch size: 128 Dropout rate: 0.5 Early stopping within 100 epochs Results Comparisons of the performances with different network configurations Single-head Top-layer Single-head Layer-wise Multi-head Top-layer Multi-head Layer-wise 0.62 0.63 0.64 0.65 0.66 0.67 0.68 F-measure 1 layer 2 layers 3 layers 4 layers Results COMMA PERIOD QUESTION OVERALL Models P R F P R F P R F P R F SER DNN-A [1] 48.6 42.4 45.3 59.7 68.3 63.7 - - - 54.8 53.6 54.2 66.9 CNN-2A [1] 48.1 44.5 46.2 57.6 69.0 62.8 - - - 53.4 55.0 54.2 68.0 T-LSTM [2] 49.6 41.4 45.1 60.2 53.4 56.6 57.1 43.5 49.4 55.0 47.2 50.8 74.0 T-BRNN [3] 64.4 45.2 53.1 72.3 71.5 71.9 67.5 58.7 62.8 68.9 58.1 63.1 51.3 T-BRNN-pre [3] 65.5 47.1 54.8 73.3 72.5 72.9 70.7 63.0 66.7 70.0 59.7 64.4 49.7 Single-BiRNN [4] 62.2 47.7 54.0 74.6 72.1 73.4 67.5 52.9 59.3 69.2 59.8 64.2 51.1 Corr-BiRNN [4] 60.9 52.4 56.4 75.3 70.8 73.0 70.7 56.9 63.0 68.6 61.6 64.9 50.8 DRNN-LWMA 63.4 55.7 59.3 76.0 73.5 74.7 75.0 71.7 73.3 70.0 64.6 67.2 47.3 DRNN-LWMA-pre 62.9 60.8 61.9 77.3 73.7 75.5 69.6 69.6 69.6 69.9 67.2 68.6 46.1 so if wemake nochangestodaywhatdoestomorrow look likewell ithink youprobablyalreadyhave thepicture so if we make no changes today what does tomorrow look like well i think you probably already have the picture so if wemake nochangestodaywhatdoestomorrow look likewell ithink youprobablyalreadyhave thepicture so if we make no changes today what does tomorrow look like well i think you probably already have the picture so if wemake nochangestodaywhatdoestomorrow look likewell ithink youprobablyalreadyhave thepicture so if we make no changes today what does tomorrow look like well i think you probably already have the picture so if wemake nochangestodaywhatdoestomorrow look likewell ithink youprobablyalreadyhave thepicture so if we make no changes today what does tomorrow look like well i think you probably already have the picture References [1] Xiaoyin Che, ChengWang, Haojin Yang, and Christoph Meinel, ‘Punctuation prediction for unsegmented transcript based on word vector’, LREC 2016. [2] Ottokar Tilk and Tanel Alumae, ‘LSTM for punctuation restoration in speech transcripts’, INTERSPEECH 2015. [3] Ottokar Tilk and Tanel Alumae, ‘Bidirectional recurrent neural network with attention mechanism for punctuation restoration’, INTERSPEECH 2016. [4] Vardaan Pahuja et al., ‘Joint learning of correlated sequence labelling tasks using bidirectional recurrent neural networks’, INTERSPEECH 2017. 345 Park Avenue, San Jose, CA 95110, USA Email: seokim@adobe.com