How Transformers Generate and Translate Sequences: The Nuts and Bolts Brandon Rohrer

1. How Transformers
Generate and
Translate Sequences:
The Nuts and Bolts
Brandon Rohrer

2. 1
0
0
0
1
0
0
0
1
f
i
l
e
s
f
i
n
d
m
y

3. 1
0
0
0
1
0
0
0
1
f
i
l
e
s
f
i
n
d
m
y

4. x
x
x
x
0
1
1
2
1
0
1
2
=
=
=
=
0
1
0
4
+
5

5. +
x
x
x
x
0
1
0
0
0
1
0
0
=
=
=
=
1
0
0
0
1

6. +
x
x
x
x
0
1
0
0
0
0
0
1
=
=
=
=
0
0
0
0
0

7. +
x
x
x
x
0
0
1
0
.2
.7
.8
.1
=
=
=
=
0
.8
0
0
.8

8. .
0
0
1
0
.2
.7
.8
.1
= .8
0 0 1 0
.2
.7
.8
.1
A B
=

9. Dot product of
the first row of A with
B
Dot product of
the second row of A
with B
.2
.8
=
0 0 1 0
.2
.7
.8
.1
A B
=
1 0 0 0

10. Dot
product
of A
with the
first
column
of B
Dot
product
of A
with the
second
column
of B
.3
.8
=
0 0 1 0
.9
0
.3
.4
A B
=
.2
.7
.8
.1

11. row 1 of A,
col 1 of B
row 1 of A,
col 2 of B
0 0 1 0
.9
0
.3
.4
A B
=
.2
.7
.8
.1
0 0 0 1
1 0 0 0
Dot products of
row 2 of A,
col 1 of B
row 2 of A,
col 2 of B
row 3 of A,
col 1 of B
row 3 of A,
col 2 of B
=
.9
.4
.3
.2
.1
.8

12. show me my files
directories
photos
please
.5
.3
.2 1
1
1
1
1

13. 0 0 0 1
directories
0 0 0 0
0 0 0 0
0 0 0
0 1 0
0 1 0
0 0 0 0
0 0 0 0
.2 .3 0 0
0 0 0
0 1 0
.5 0 0
0 0 1 0 0 0 0
files
me
my
photos
please
show
d
i
r
e
c
t
o
r
i
e
s
f
i
l
e
s
m
e
m
y
p
h
o
t
o
s
p
l
e
a
s
e
s
h
o
w

14. 0 0 0 1
0 0 0 0
0 0 0 0
0 0 0
0 1 0
0 1 0
0 0 0 0
0 0 0 0
.2 .3 0 0
0 0 0
0 1 0
.5 0 0
0 0 1 0 0 0 0
d
i
r
e
c
t
o
r
i
e
s
f
i
l
e
s
m
e
m
y
p
h
o
t
o
s
p
l
e
a
s
e
s
h
o
w
0 0 0 1 0 0 0 .2 .3 0 0 .5 0 0
=
d
i
r
e
c
t
o
r
i
e
s
f
i
l
e
s
m
e
m
y
p
h
o
t
o
s
p
l
e
a
s
e
s
h
o
w
d
i
r
e
c
t
o
r
i
e
s
f
i
l
e
s
m
e
m
y
p
h
o
t
o
s
p
l
e
a
s
e
s
h
o
w

15. check whether the
battery
program
ran
down
please
1
.4
.6 .6
1
1
1
1
.4

16. 0 0 0 1
battery
0 0 0 0
0 0 0 0
0 0 0
0 0 0
0 1 0
0 0 .4 .6
0 0 0 0
0 0
0 0 0
0 1 0
0 0
.4 0 0 0 .6 0 0
check
down
please
program
ran
the
b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
r
a
n
t
h
e
whether
w
h
e
t
h
e
r
0 0 0 0 0 0 1
0
1
0
0
0
0
0
0
0
0
0

17. down
please
1
battery ran
program ran
1

18. 0 0 0 1
battery ran
0 0 0 0
0 0 1 0
0 0 0
0 0 1
0 0 0
0 0 0 1
0 0 0 0
0 0
0 0 0
0 1 0
1 0
.4 0 0 0 .6 0 0
check whether
program ran
ran down
the battery
b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
r
a
n
t
h
e
whether the
w
h
e
t
h
e
r
0 0 0 0 0 0 0
0
0
0
0
0
0
0
0
0
0
0
the program
.
.
.

19. down
please
and, ran
battery, ran
check, ran
find, ran
it, ran
log ran
out, ran
program, ran
the, ran
whether, ran

20. .5
and, ran
1
0
battery, ran
check, ran
it, ran
down, ran
b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
r
a
n
t
h
e
log, ran
w
h
e
t
h
e
r
find, ran
l
o
g
a
n
d
f
i
n
d
o
u
t
i
t
0
0 0 0
0 0 0
0 0 0
0 0 0
0 0
0 0
0 0 0
0
0
0
0
0
0
0
0
0
out, ran
please, ran
program, ran
whether, ran
ran, ran
the, ran
0
0 0 0
0 0 0
0 0
0 0
0 0 0
0
0
0
0
0
0
1
.5
.5
.5
.5
.5
.5
.5
.5
.5
.5
.5
.5
.5
.5
.5
0

21. 0
0
b
a
t
t
e
r
y
,
r
a
n
c
h
e
c
k
,
r
a
n
d
o
w
n
,
r
a
n
p
l
e
a
s
e
,
r
a
n
0 0 0
p
r
o
g
r
a
m
,
r
a
n
l
o
g
,
r
a
n
f
i
n
d
,
r
a
n
o
u
t
,
r
a
n
i
t
,
r
a
n
a
n
d
,
r
a
n
1 1
r
a
n
,
r
a
n
t
h
e
,
r
a
n
w
h
e
t
h
e
r
,
r
a
n
1 1 0 0
1
1 0 1 1
1
... ...

22. 0
0
b
a
t
t
e
r
y
,
r
a
n
c
h
e
c
k
,
r
a
n
d
o
w
n
,
r
a
n
p
l
e
a
s
e
,
r
a
n
0 0 0
p
r
o
g
r
a
m
,
r
a
n
l
o
g
,
r
a
n
f
i
n
d
,
r
a
n
o
u
t
,
r
a
n
i
t
,
r
a
n
a
n
d
,
r
a
n
1 1
r
a
n
,
r
a
n
t
h
e
,
r
a
n
w
h
e
t
h
e
r
,
r
a
n
1 1 0 0
1
1 0 1 1
1
0
0 0 0 0
0 0 0 0 0 1
0
1 0 0 0
0
mask
feature
activities
x
=
0
0 0 0 0
0 0 0 0 0 0
0
1 0 0 0
0
masked
feature
activities

23. and, ran
1
0
battery, ran
check, ran
it, ran
down, ran
b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
r
a
n
t
h
e
log, ran
w
h
e
t
h
e
r
find, ran
l
o
g
a
n
d
f
i
n
d
o
u
t
i
t
0
0 0 0
0 0 0
0 0 0
0 0 0
0 0
0 0
0 0 0
0
0
0
0
0
0
0
0
0
out, ran
please, ran
program, ran
whether, ran
ran, ran
the, ran
0
0 0 0
0 0 0
0 0
0 0
0 0 0
0
0
0
0
0
0
1
0

24. feature-specific mask
0 0 0 0 0 0 0 0 0 1 0 0 0 0
0 1 0 0 0 0 0 0 0 0 1 0 0 0
0 1 0 0 0 0 0 0 0 0 1 0 0 0
=
one-hot feature vector (query: Q )
all the masks (keys: K )
T

26. b
a
t
t
e
r
y
p
r
o
g
r
a
m
r
a
n
b
a
t
t
e
r
y
,
r
a
n
p
r
o
g
r
a
m
,
r
a
n
1
-1
1
-1
-1 1
1
1 -1
battery
program
run
bias
b
a
t
t
e
r
y
,
r
u
n
p
r
o
g
r
a
m
,
r
u
n
-1 -1
-1
-1
1 1
1
1

27. b
a
t
t
e
r
y
,
r
u
n
p
r
o
g
r
a
m
,
r
u
n
-1 -1
-1
-1
1 1
1
1
b
a
t
t
e
r
y
p
r
o
g
r
a
m
r
u
n
b
i
a
s
=
1 1 1 -1
1
0

28. b
a
t
t
e
r
y
,
r
u
n
p
r
o
g
r
a
m
,
r
u
n
-1 -1
-1
-1
1 1
1
1
b
a
t
t
e
r
y
p
r
o
g
r
a
m
r
u
n
b
i
a
s
=
0 1 1
-1
1
1

29. ReLU
masked word
activities
multi-word
feature creation
matrix
selective second
order transition
matrix

31. b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
l
o
g
f
i
n
d
o
u
t
i
t
a
n
d
r
a
n
t
h
e
w
h
e
t
h
e
r
embedding
in 2 dimensions

32. b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
l
o
g
f
i
n
d
o
u
t
i
t
a
n
d
r
a
n
w
h
e
t
h
e
r
t
h
e
1.1
-.3 program
1.0
.2 battery
-1.2
-.5 out
-.8
-.9 down
...

33. b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
l
o
g
f
i
n
d
o
u
t
i
t
a
n
d
r
a
n
t
h
e
w
h
e
t
h
e
r
.2
.2
1.0
1.0
1 =
embedding
projection
matrix
battery

35. ...
battery
before position
encoding
battery
at position 0
1
2
3
battery
in the last
position
.
.
.

37. b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
l
o
g
f
i
n
d
o
u
t
i
t
a
n
d
r
a
n
t
h
e
w
h
e
t
h
e
r
1
=
de-embedding
projection matrix
program
-.3 1.1
-.1
.9

38. b
a
t
t
e
r
y
c
h
e
c
k
d
o
w
n
p
l
e
a
s
e
p
r
o
g
r
a
m
l
o
g
f
i
n
d
o
u
t
i
t
a
n
d
r
a
n
t
h
e
w
h
e
t
h
e
r
1 .4 -.8 0
-.6
-.3
.8
-.7
0
-.2
.1
.9
-.7

40. N
n
one-hot
word
sequence
N
d_model
n
d_model
=
embedding
transform
embedded
word
sequence
[ n x N ] [ N x d_model ] [ n x d_model ]

41. [ n x N ]
[ N x d_model ]
[ n x d_model ]
[ n x d_model ]
[ n x d_model ]
[ d_model x N ]
[ n x N ]
[ n x N ]
[ n x d_model ]

42. [ n x d_model ]
[ d_model x d_k ]
[ d_model x d_v ]
[ n x d_k ]
[ n x d_v ]
[ n x d_v ]
[ n x h*d_v ]
[ n x d_model ]
[ h*d_v x d_model ]

43. [ n x d_k ]
[ n x d_v ]
[ n x n ]
[ n x n ]
[ n x n ]
[ n x n ]
[ n x d_v ]

45. Feed
Forward
Multi-Head
Attention
+ +
Feed
Forward
Multi-Head
Attention
+ +
Layer 1 Layer 2 ...