2. Motivation
• ML is ultimately about automating tasks, hoping
that machine can do everything for human
• For example, I want the machine to make a cup of
coffee for me
3. Motivation
• Ancient way: is to write full highly-detailed program
specifications to carry them out
• AI way: come up with a lot of training examples that
capture the variability in the real world, and then
train some general learning machine on this large
data set.
4. Motivation
• but sometimes the dataset is not big enough! and it
doesn’t generalize well..
• NPI is an attempt to use neural methods to train
machines to carry out simple tasks based on a
small amount of training data.
5. NPI Goals
• 1. Long-term prediction: Model potentially long sequences
of actions by exploiting compositional structure.
• 2. Continual learning: Learn new programs by composing
previously- learned programs, rather than from scratch.
• 3. Data efficiency: Learn generalizable programs from a
small number of example traces.
• 4. Interpretability: By looking at NPI’s generated
commands, we can understand what it is doing at multiple
levels of temporal abstraction.
6. Related Work
• Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le.
"Sequence to sequence learning with neural
networks." Advances in neural information
processing systems. 2014.
• Graves, Alex, Greg Wayne, and Ivo Danihelka.
"Neural turing machines." arXiv preprint arXiv:
1410.5401 (2014).
9. Outline
• NPI core module: how it works
• Demos
• Experiment
• Conclusion
10. Outline
• NPI core module: how it works
• Demos
• Experiment
• Conclusion
11. NPI core module
• The NPI core is a LSTM network that acts as a
router between programs conditioned on the
current state observation and previous hidden unit
states
• input: a learnable program embedding, program
arguments passed on by the calling program, and
a feature representation of the environment.
• output: a key indicating what program to call next,
arguments for the following program and a flag
indicating whether the program should terminate.
core: an LSTM-based
sequence model
14. Car Rendering
• Whatever the starting position, the program should generate a
trajectory of actions that delivers the camera to the target view, e.g.
frontal pose at a 15◦ elevation.
20. Outline
• NPI core module: how it works
• Demos
• Experiment
• Conclusion
21. Adding Numbers
• Environment:
• Scratch pad with the two numbers to be added, a carry row
and output row.
• 4 read/write pointers location
• Program:
• LEFT, RIGHT programs that can move a carry pointer left or
right, respectively.
• WRITE program that writes a specified value to the location of
a specified pointer
23. Adding Numbers
• all output actions (primitive atomic actions that can
be performed on the environment) are performed
with a single instruction – ACT.
all output actions (primitive atomic actions that can be
performed on the environment) are performed with a single
instruction – ACT.
28. Car Rendering
• Environment:
• Rendering of the car
(pixels). (use CNN as feature
encoder)
• The current car pose is NOT
provided
• Target angle and elevation
coordinates.
30. Car Rendering
• Whatever the starting position, the program should generate a
trajectory of actions that delivers the camera to the target view, e.g.
frontal pose at a 15◦ elevation.
41. Data Efficiency - Sorting
• Seq2Seq LSTM and NPI
used the same number of
layers
and hidden units.
• Trained on length 20 arrays
of single-digit numbers.
• NPI benefits from mining
multiple subprogram
examples per sorting
instance
accuracy v.s. training example
42. Generalization - Sorting
• For each length 2 up
to 20, we provided 64
example bubble sort
traces, for a total of
1216 examples.
• Then, we evaluated
whether the network
can learn to sort arrays
beyond length 20
44. Learning New Programs
with a Fixed NPI Core
• example task: find the Max in array
• RJMP: move all pointers to the right by repeatedly
calling RSHIFT program
• MAX: call BUBBLESORT and then RJMP
• Expand program memory by adding 2 slots.
Randomly initialize, then learn by
backpropagation with the NPI core and all other
parameters fixed.
45. • 1. Randomly initialize new program vectors in
memory
• 2. Freeze core and other program vectors
• 3. Backpropagate gradients to new program
vectors
46. • + Max: performance after addition of MAX program
to memory.
• “unseen” uses a test set with disjoint car models
from the training set.
47. Outline
• NPI core module: how it works
• Demos
• Experiment
• Conclusion
48. Conclusion(1/2)
• NPI is a RNN/LSTM-based sequence-to-sequence
translator with the ability to keep track of calling
programs while recurse into sub-program
• NPI generalizes well in comparison to sequence-to-
sequence LSTMs.
• A trained NPI with a fix core can learn new task
while not forgetting about the old task
49. Conclusion(2/2)
• provide far fewer examples, but where the labels
contains richer information allowing the model to
learn compositional structure(It’s like sending kids
to school)
50. Further Discussion
• Can each task help each other during training?
• Can we share environment encoder?
• Any comments?
project page: http://www-personal.umich.edu/~reedscot/
iclr_project.html
