[Paper Review] @Modulabs Title : DRAGNN: A Transition-based Framework for Dynamically Connected Neural Networks url : https://arxiv.org/abs/1703.04474

1. DRAGNN: A Transition­based Framework for
Dynamically Connected Neural Networks
Lingpeng Kong et al. [2017]
Review editted by PyunginPaek@modulabs

2. Table of Contents
● Introduction of DRAGNN
● Transition system & TBRU
● Examples of TBRU
● Combinations of TBRUs
● Experiments and Discussions

3. Intro. of DRAGONDRAGNN*
● General Model for Structured Prediction
– By connecting multiple TBRUs, we can extend to
architectures such as seq2seq, attention mechanism,
recursive tree models..
DRAGNN
TBRU
TBRU
TBRU TBRU
* Kong et al. [2017] A Transition­based Framework for Dynamically Connected Neural Networks , https://arxiv.org/abs/1703.04474

4. SyntaxNet­II (from Google)
● Major upgrade to SyntaxNet (Mar/15, 2017)
– Multilingual language understanding
– Joint modeling of multiple levels of linguistic structure
– Neural­network architectures to be created dynamically during processing of a
sentence or document
● Implementation*
: combination of below 2.
– Recurrent multi­task parsing model (with DRAGNN)
– Character­based representation (with LSTM) as input to DRAGNN
● DRAGNN as the new core
– Architecture as a series of modular unit(TBRU)s
– Connections between modules are unfolded dynamically.
→ Dynamic Recurrent Acyclic Graphcal Neural Networks
* Alberti et al. [2017] SyntaxNet Models for the CoNLL 2017 Shared Task ,
(paper) https://arxiv.org/abs/1703.04929 (code) https://github.com/tensorflow/models/tree/master/syntaxnet

5. (Roughly Speaking) Generalization??
RNN module
LSTM, GRU, ...
Structured Models
Seq2seq, recursive tree,
bi­LSTM tagger,
Encoder/Decoder...
Generalize TBRU
Generalize DRAGNN

6. Table of Contents
● Introduction of DRAGNN
● Transition system & TBRU
● Examples of TBRU
● Combinations of TBRUs
● Experiments and Discussions

7. Transition System
● Transition System : T
– T = {S, A, t}
– S(x) : Set of States ( s+
∈ S(x) : Start State )
– x : input (eg. sentence)
– A(s, x) : Set of allowed decision for any s ∈ S
– transition function t(s, d, x)
● s’ = t(s, d, x) : new state s’ for any decision d ∈ A(s, x)
● s’ = t(s, d) for brevity
● Complete structure : sequence of state/decision pairs
– (s1
, d1
)...(sn
, dn
)
– s1 = s+
, di
∈ A(s, x)
– si+1
= t(si
, di
)

8. Transition Based Recurrent Unit (TBRU)
● m(s) : Input embedding function
– eg. lookup op m(s) : S R→ k
● r(s) : recurrence function (Connection to previous states)
– mapping : states set of previous time steps→
– r(si
) : S P{1, …, i­1} ( P : power set, so n(P) variable)→
● (RNN) Network Cell : computes new hidden representation
– hs
= RNN( m(s), {hi
| i ∈ r(s)} )
( Logically : r,m h d )→ →
di
argmax← d A(si)∈
wd
T
hi

9. Inference with TBRUs
1. Initialize s1
= s+
2. For i = 1, …, n:
(a) Update the hidden state:
hi
RNN(m(s← i
), {hj
| j r(s∈ i
)})
(b) Update the transition state:
di
argmax← d A(si)∈
wd
T
hi
si+1
t(s← i
, di
)

10. Table of Contents
● Introduction of DRAGNN
● Transition system & TBRU
● Examples of TBRU
● Combinations of TBRUs
● Experiments and Discussions

11. TBRU Ex.(1) Simple LSTM Tagger
input x sequence of word embedding
x = {x1
, …, xn
}
output d sequence of tags
d1
, …, dn
discrete state s state = previous sequence of tags
si
= {d1
, …, di­1
}
input embedding
m(s)
word embedding for the next token
m(si
) = xi
recurrence function
r(s)
network connected only to the previous state
r(si
) = {i­1}
RNN and hidden
state h
Simple LSTM cell
hi
= RNN(xi
, hi­1
)
Transition
t(s, d)
si+1
t(s← i
, di
) as LSTM cell state

12. TBRU Ex.(2) SyntaxNet­I*
input x sequence of character­level multi­scale representation
x = {x1
, …, xn
}
output d if action = {Shift}: d = input word
elif action = {Left/Right Arc}: d = partial­built tree
discrete state s State = stack squence of {d1
, …, di­1
}
It contains all words and partial­built trees on the stack
input embedding
m(s)
Sentence structure projected to 52 feature embedding
recurrence function
r(s)
Empty
r(si
) = {}
RNN and hidden
state h
Feed­forward MLP (Not RNN!!)
di
= MLP(m(si
))
Transition
t(s, d)
si+1
t(s← i
, di
) as next stack sequence
* Andor et al. [2016] Globally Normalized Transition­based Neural Networks, https://arxiv.org/abs/1603.06042
Arc­standard
Transition System
Refer Andor et al. [2016]
Global Normalization

13. Arc­Standard Transition System*
Input Word Top Tree node On the Stack
Bob Bob
gave gave
Alice gave
a gave a
pretty gave a pretty
flower gave flower
on gave flower on
Monday gave flower on
. gave
* Nivre [2006] Inductive dependency parsing.

14. Table of Contents
● Introduction of DRAGNN
● Transition system & TBRU
● Examples of TBRU
● Combinations of TBRUs
● Experiments and Discussions

15. Now, let’s combine TBRUs !!
● Multiple TBRUs, having different recurrence r(s).
– So, they act as different transition task.
● Multiple TBRUs, but run one at a time, after another.
– Multiple TBRUs have common global step­counter.
● Then, if a hidden state hA
of TBRU A is connected to
recurrence r(sB
) of TBRU B??
● Possibly, might we reach to multi­task, hierarchical learning?

16. TBRU Combination Examples (1)
Important Note:
A TBRU can be both
a decoder of it’s own task
and
an encoder of other TBRU’s of another TBRU
at the same time.

17. Stack Propagation
● Using explicit structure improves encoder­decoder.
– For newswire text, TBRU exceeds pure encoder­decoder seq2seq.
● Utilizing parse representations improves summarization.
– Multi­task learning setup
● Deep stacked bi­directional parsing
– Bi­directional approach gives significant advantages.
– Final model (5 TBRUs) : 1 L­R pos­tagging 2 shift only 2 arc­parser bi­directionally→ →

18. TBRU Combination Examples (2)

19. Table of Contents
● Introduction of DRAGNN
● Transition system & TBRU
● Examples of TBRU
● Combinations of TBRUs
● Experiments and Discussions

20. How to train a DRAGNN
● Assumption
– there should be at least one TBRU for which example x along with gold decision(d*
)
sequence.
● Log­likelihood objective
– For TBRU which has gold
decision sequence,
– Θ are combined parameters across all TBRUs.
● Where decision d1:N
from?
– 1) case when they are part of gold annotations
– 2) prediction from unrolling previous TBRU
● In practice, task sampling is usually preceded by pre­training steps.

21. Experiment Result (1)
● Using explicit structure improves encoder­decoder.
– For newswire text, TBRU exceeds pure encoder­decoder seq2seq.
● Utilizing parse representations improves summarization.
– Multi­task learning setup

22. Experiment Result (2)
● Deep stacked bi­directional parsing
– Bi­directional approach gives significant advantages.
– Final model (5 TBRUs) : 1 L­R pos­tagging 2 shift →
only 2 arc­parser bi­directionally→

23. Conclusion & Discussions
● Primarily focused on syntactic parsing, but can it
provide general means of sharing representations
between tasks?
● Can this approach be globally normalized?
● Multi­task learning possible with a single model?
– By combining translation, summarization, and even
abstraction and reasoning.
– And beyond language processing