The document discusses the implementation and analysis of Adaptive Computation Time (ACT) in PyTorch and TensorFlow, focusing on its motivation, theoretical background, and related work. It compares ACT with a newly designed model called Repeat-RNN through various experiments, demonstrating that the simpler model performed better than ACT. Conclusions highlight the achievements and suggest future improvements for ACT.

1. Reproducing and Analyzing
Adaptive Computation Time
in PyTorch and TensorFlow
Víctor Campos Xavier Giró-i-NietoDani Fojo
06/02/2018

2. Outline
2
1. Motivation
2. Theoretical background
3. Related work
4. Adaptive Computation Time
5. Implementation
6. Experiments
7. Conclusions

3. Motivation

4. Motivation
4
Why do we need adaptive computation?

5. Motivation
5
The complexity of posing a problem is not
proportional to the complexity of solving it.

6. Motivation
6
The complexity of posing a problem is not
proportional to the complexity of solving it.
Year 1637

7. Motivation
7
The complexity of posing a problem is not
proportional to the complexity of solving it.
A. Wiles “Modular elliptic curves and Fermat’s last theorem”
Year 1637 Year 1995

8. Motivation
8
Problems may differ in complexity

9. Theoretical background

10. Neural Networks
Alternate linear and non-linear functions.
10

11. Loss function
11
Mean Squared Error:
Cross Entropy:

12. Recurrent Neural Networks (RNN’s)
12
Main idea: The network has a state.
Slide credit: Xavier Giró

13. Recurrent Neural Networks (RNN’s)
13
time
time
Unfold
(Rotation
90o
)
Front View Side View
Rotation
90o
Slide credit: Xavier Giró

14. Related work

15. Related work
15
Spatially-Adaptive Computation Time for
Residual Networks.
M. Figurnov et al. “Spatially Adaptive Computation Time for Residual Networks” CVPR 2017

16. Related work
16
LSTM-Jump and Skip-RNN.
A. Yu et al. “Learning to Skim Text” ACL 2017
V. Campos et al. “Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks” ICLR 2018

17. Adaptive Computation
Time

18. Adaptive Computation Time
18
Adaptive Computation Time for RNN’s (ACT).
A. Graves “Adaptive Computation Time for Recurrent Neural Networks”

19. Adaptive Computation Time (ACT)
19

20. Simple Recurrent Neural Network (RNN)
20

21. Model description
21

22. Model description
22
Halting probability

23. Model description
23
Each sample is processed until these probabilities
add up to one.

24. Model description
24
● Residual:
● Outputs:

25. Limiting computation
25
● Modified loss:
● Ponder cost:
Time penalty

26. Ponder cost: Intuition
26
Constant
Negated
probabilities

27. Limiting computation
27
Maximum steps

28. Implementation

29. Deep Learning Frameworks
29

30. Deep Learning Frameworks
30

31. Computation graph
31
Static graph Dynamic graph

32. Implementation
32
Pros:
● Dynamic graph
Cons:
● Version 0.2
● Lack of docs
● Too slow for custom
RNN’s

33. Implementation
33
Pros:
● Version 1.4
● Much better docs
● Much faster
Cons:
● Static graph
● Harder to use

34. Implementation
34
bit.ly/dani-tfg

35. Experiments

36. Experiments
36
We wanted to test ACT:
Comparing it with a simple RNN is unfair.
VS

37. New baseline: Repeat-RNN
37
Repetitions

38. Experiments: Parity
38

39. Experiments: Addition
39
Each digit is fed as a one-hot encoding

40. Experiments: Addition
40
Each digit is fed as a one-hot encoding

41. Experiments: Addition
41
Each digit is fed as a one-hot encoding

42. Experiments: Addition
42
Each digit is fed as a one-hot encoding

43. Experiments: Addition
43
Each digit is fed as a one-hot encoding

44. Experiments: Addition
44
Each digit is fed as a one-hot encoding

45. One-hot encoding
45

46. Experiments: Addition
46
Each digit is fed as a one-hot encoding

47. Parity: Comparison
47

48. Parity: Comparison
48

49. Parity: Comparison
49

50. Parity: Comparison
50

51. Parity: Comparison
51

52. Parity: Comparison
52

53. Addition: Comparison
53

54. Conclusions

55. Conclusions
55
1. We implemented ACT in two frameworks
2. Designed Repeat-RNN and compared it to ACT
3. We could achieve better performance than ACT
with a simpler more interpretable model

56. Conclusions
56
1. We implemented ACT in two frameworks
2. Designed Repeat-RNN and compared it to ACT
3. We could achieve better performance than ACT
with a simpler more interpretable model

57. Conclusions
57
1. We implemented ACT in two frameworks
2. Designed Repeat-RNN and compared it to ACT
3. We could achieve better performance than ACT
with a simpler more interpretable model
VS

58. Future work
58
● ICLR 2018 workshop
● Improve ACT
● Skipping samples of Repeat-RNN

59. 59

61. Backup slides

62. Parity: ACT-RNN

63. Parity: Repeat-RNN

64. Parity: Comparison

65. Addition: ACT-LSTM

66. Addition: Repeat-LSTM

67. Addition: Ponder distribution

68. Addition: Comparison