Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Sebastian Ruder

PhD Candidate, Insight Centre

Research Scientist, AYLIEN
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dubli...
Agenda
1. NIPS overview
2. Generative Adversarial Networks
3. Building applications with Deep Learning
4. RNNs
5. Improvin...
NIPS Overview
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Source: http://www.tml.cs.uni-tuebingen.de/team/luxb...
Generative Adversarial Networks
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Ian Goodfellow’s GAN tutorial1
•...
Generative Adversarial Networks
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Generative Adversarial Networks
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Ian Goodfellow’s GAN tutorial1
•...
Generative Adversarial Networks
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Generative Adversarial Networks
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Ian Goodfellow’s GAN tutorial1
•...
Building applications with DL
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Building applications with DL
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• 20 year
anniversary of
LSTM…
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• 20 year
anniversary of
LSTM…

being rejected
from NIPS 1996...
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RNN symposium
• Many papers on improving RNNs:
• Handling d...
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RNN symposium
• Many papers on improving RNNs:
• Handling d...
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RNN symposium
• Many papers on improving RNNs:
• Handling d...
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RNN symposium
• Many papers on improving RNNs:
• Handling d...
RNNs
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RNN symposium
• Many papers on improving RNNs:
• Handling d...
Improving classic algorithms
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Matrix completion: Non-convex objec...
Improving classic algorithms
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Matrix completion: Non-convex objec...
Improving classic algorithms
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Matrix completion: Non-convex objec...
Reinforcement Learning
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RL tutorial by Pieter Abbeel and John Sch...
Reinforcement Learning
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Reinforcement Learning
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• RL tutorial by Pieter Abbeel and John Sch...
Learning-to-learn / Meta-learning
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Several papers, e.g.

Learning...
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
General AI
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Source: Drew Purves
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
General Artificial Intelligence
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Topic in keynote talks
• Yann LeC...
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Variations of Neural MT
• Dual learning for MT (with RL)1
• ...
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Variations of Neural MT
• Dual learning for MT (with RL)1
• ...
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Variations of Neural MT
• Dual learning for MT (with RL)1
• ...
Miscellaneous
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Schmidhuber is

everywhere!
Miscellaneous
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Schmidhuber is

everywhere!
• Robotics
Miscellaneous
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Schmidhuber is

everywhere!
• Robotics
• Apple sta...
Miscellaneous
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Schmidhuber is

everywhere!
• Robotics
• Apple sta...
Miscellaneous
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
• Schmidhuber is

everywhere!
• Robotics
• Apple sta...
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Thanks for your attention!
Questions?
Upcoming SlideShare
Loading in …5
×

NIPS 2016 Highlights - Sebastian Ruder

2,774 views

Published on

Talk at 4th NLP Dublin meetup (https://www.meetup.com/NLP-Dublin/events/233314527/) on highlights of NIPS 2016.

Published in: Science

NIPS 2016 Highlights - Sebastian Ruder

  1. 1. Sebastian Ruder
 PhD Candidate, Insight Centre
 Research Scientist, AYLIEN @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup NIPS 2016 Highlights
  2. 2. Agenda 1. NIPS overview 2. Generative Adversarial Networks 3. Building applications with Deep Learning 4. RNNs 5. Improving classic algorithms 6. Reinforcement Learning 7. Learning-to-learn / Meta-learning 8. General AI 9. NLP 10. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  3. 3. NIPS Overview @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup Source: http://www.tml.cs.uni-tuebingen.de/team/luxburg/misc/nips2016/index.php
  4. 4. Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks
  5. 5. Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  6. 6. Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 • Yann LeCun is very bullish on GANs; lots of work on GANs from FAIR, etc.3 • Cool extensions, e.g. GA what-were network4 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks 3https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view 4https://papers.nips.cc/paper/6111-learning-what-and-where-to-draw.pdf
  7. 7. Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  8. 8. Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 • Yann LeCun is very bullish on GANs; lots of work on GANs from FAIR, etc.3 • Cool extensions, e.g. GA what-were network4 • Mostly used in CV for image generation; discriminator can be used as feature extractor • Less work in NLP and other areas; promising directions at GAN workshop (1, 2, 3, 4) 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks 3https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view 4https://papers.nips.cc/paper/6111-learning-what-and-where-to-draw.pdf 5https://arxiv.org/abs/1611.01144
  9. 9. Building applications with DL @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  10. 10. Building applications with DL @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  11. 11. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • 20 year anniversary of LSTM…
  12. 12. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • 20 year anniversary of LSTM…
 being rejected from NIPS 1996
 — perseverance pays off
  13. 13. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf
  14. 14. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for- long-or-event-based-sequences.pdf
  15. 15. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf 3https://papers.nips.cc/paper/6039-sequential-neural-models-with-stochastic-layers.pdf
  16. 16. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3
  17. 17. RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3 •General DL improvements • Weight normalisation4 • Multinomial dropout5 •Edward Grefenstette’s talk at NAMPI workshop6 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf 3https://papers.nips.cc/paper/6039-sequential-neural-models-with-stochastic-layers.pdf 4https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural- networks.pdf 5https://papers.nips.cc/paper/6561-improved-dropout-for-shallow-and-deep-learning.pdf 6https://uclmr.github.io/nampi/talk_slides/grefenstette_limitations_of_rnns.pdf
  18. 18. Improving classic algorithms @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf
  19. 19. Improving classic algorithms @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 • Clustering: Using queries to convey domain information and reduce computational complexity 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf
  20. 20. Improving classic algorithms @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 • Clustering: Using queries to convey domain information and reduce computational complexity3 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf 3https://papers.nips.cc/paper/6449-clustering-with-same-cluster-queries.pdf
  21. 21. Reinforcement Learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RL tutorial by Pieter Abbeel and John Schulman:
 Slides1; practical advice2 • Best paper: Value Iteration Networks3 • Learns to plan; contribution: differentiable approximation to classic algorithm via CNN 1http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulman-Abbeel.pdf 2http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf 3https://papers.nips.cc/paper/6046-value-iteration-networks.pdf
  22. 22. Reinforcement Learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  23. 23. Reinforcement Learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RL tutorial by Pieter Abbeel and John Schulman:
 Slides1; practical advice2 • Best paper: Value Iteration Networks3 • Learns to plan; contribution: differentiable approximation to classic algorithm via CNN • New research environments: • OpenAI’s Universe • Deep Mind Lab • FAIR’s Torchcraft 1http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulman-Abbeel.pdf 2http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf 3https://papers.nips.cc/paper/6046-value-iteration-networks.pdf
  24. 24. Learning-to-learn / Meta-learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Several papers, e.g.
 Learning to learn by gradient descent by gradient descent1 • Meta-learning panel at RNN symposium • Sutskever: No good meta-learning models so far • Schmidhuber: Focus on universal model • Focus at Neural Abstract Machines workshop 1https://papers.nips.cc/paper/6461-learning-to-learn-by-gradient-descent-by-gradient-descent.pdf
  25. 25. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning
  26. 26. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning
  27. 27. General AI @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  28. 28. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI
  29. 29. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI
  30. 30. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup Source: Drew Purves
  31. 31. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain
  32. 32. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain
  33. 33. General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain • Machine Intelligence workshop: • FAIR’s CommAI-env • Hierarchical representations, identifying patterns • Panel: Unclear if focus on specific tasks brings us closer to General AI.
  34. 34. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf
  35. 35. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  36. 36. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  37. 37. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 • Modelling user reviews4 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf 4 https://papers.nips.cc/paper/6362-beyond-exchangeability-the-chinese-voting-process.pdf
  38. 38. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  39. 39. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  40. 40. NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 • Modelling user reviews4 • Dialogue modelling workshop: • End-to-end systems, linguistics, and ML methods 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf 4 https://papers.nips.cc/paper/6362-beyond-exchangeability-the-chinese-voting-process.pdf
  41. 41. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere!
  42. 42. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics
  43. 43. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing
  44. 44. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing • AI startups: • Geometric Intelligence
  45. 45. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing • AI startups: • Geometric Intelligence • RocketAI “launch” party
  46. 46. @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup Thanks for your attention! Questions?

×