SR
Uploaded bySebastian Ruder
3,628 views

NIPS 2016 Highlights - Sebastian Ruder

The 4th NLP Dublin meetup featured highlights from NIPS 2016, covering topics such as generative adversarial networks, recurrent neural networks, and advancements in reinforcement learning. Key discussions included building applications with deep learning, meta-learning, and the exploration of general AI. Various improvements in classic algorithms and innovations in natural language processing were also presented, along with insights from notable talks and tutorials.

Embed presentation

Downloaded 71 times
Sebastian Ruder
 PhD Candidate, Insight Centre
 Research Scientist, AYLIEN @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup NIPS 2016 Highlights
Agenda 1. NIPS overview 2. Generative Adversarial Networks 3. Building applications with Deep Learning 4. RNNs 5. Improving classic algorithms 6. Reinforcement Learning 7. Learning-to-learn / Meta-learning 8. General AI 9. NLP 10. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NIPS Overview @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup Source: http://www.tml.cs.uni-tuebingen.de/team/luxburg/misc/nips2016/index.php
Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks
Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 • Yann LeCun is very bullish on GANs; lots of work on GANs from FAIR, etc.3 • Cool extensions, e.g. GA what-were network4 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks 3https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view 4https://papers.nips.cc/paper/6111-learning-what-and-where-to-draw.pdf
Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Generative Adversarial Networks @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 • Yann LeCun is very bullish on GANs; lots of work on GANs from FAIR, etc.3 • Cool extensions, e.g. GA what-were network4 • Mostly used in CV for image generation; discriminator can be used as feature extractor • Less work in NLP and other areas; promising directions at GAN workshop (1, 2, 3, 4) 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks 3https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view 4https://papers.nips.cc/paper/6111-learning-what-and-where-to-draw.pdf 5https://arxiv.org/abs/1611.01144
Building applications with DL @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Building applications with DL @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • 20 year anniversary of LSTM…
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • 20 year anniversary of LSTM…
 being rejected from NIPS 1996
 — perseverance pays off
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for- long-or-event-based-sequences.pdf
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf 3https://papers.nips.cc/paper/6039-sequential-neural-models-with-stochastic-layers.pdf
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3
RNNs @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3 •General DL improvements • Weight normalisation4 • Multinomial dropout5 •Edward Grefenstette’s talk at NAMPI workshop6 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf 3https://papers.nips.cc/paper/6039-sequential-neural-models-with-stochastic-layers.pdf 4https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural- networks.pdf 5https://papers.nips.cc/paper/6561-improved-dropout-for-shallow-and-deep-learning.pdf 6https://uclmr.github.io/nampi/talk_slides/grefenstette_limitations_of_rnns.pdf
Improving classic algorithms @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf
Improving classic algorithms @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 • Clustering: Using queries to convey domain information and reduce computational complexity 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf
Improving classic algorithms @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 • Clustering: Using queries to convey domain information and reduce computational complexity3 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf 3https://papers.nips.cc/paper/6449-clustering-with-same-cluster-queries.pdf
Reinforcement Learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RL tutorial by Pieter Abbeel and John Schulman:
 Slides1; practical advice2 • Best paper: Value Iteration Networks3 • Learns to plan; contribution: differentiable approximation to classic algorithm via CNN 1http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulman-Abbeel.pdf 2http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf 3https://papers.nips.cc/paper/6046-value-iteration-networks.pdf
Reinforcement Learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
Reinforcement Learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • RL tutorial by Pieter Abbeel and John Schulman:
 Slides1; practical advice2 • Best paper: Value Iteration Networks3 • Learns to plan; contribution: differentiable approximation to classic algorithm via CNN • New research environments: • OpenAI’s Universe • Deep Mind Lab • FAIR’s Torchcraft 1http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulman-Abbeel.pdf 2http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf 3https://papers.nips.cc/paper/6046-value-iteration-networks.pdf
Learning-to-learn / Meta-learning @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Several papers, e.g.
 Learning to learn by gradient descent by gradient descent1 • Meta-learning panel at RNN symposium • Sutskever: No good meta-learning models so far • Schmidhuber: Focus on universal model • Focus at Neural Abstract Machines workshop 1https://papers.nips.cc/paper/6461-learning-to-learn-by-gradient-descent-by-gradient-descent.pdf
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning
General AI @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup Source: Drew Purves
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain
General Artificial Intelligence @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain • Machine Intelligence workshop: • FAIR’s CommAI-env • Hierarchical representations, identifying patterns • Panel: Unclear if focus on specific tasks brings us closer to General AI.
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 • Modelling user reviews4 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf 4 https://papers.nips.cc/paper/6362-beyond-exchangeability-the-chinese-voting-process.pdf
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
NLP @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 • Modelling user reviews4 • Dialogue modelling workshop: • End-to-end systems, linguistics, and ML methods 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf 4 https://papers.nips.cc/paper/6362-beyond-exchangeability-the-chinese-voting-process.pdf
Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere!
Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics
Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing
Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing • AI startups: • Geometric Intelligence
Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing • AI startups: • Geometric Intelligence • RocketAI “launch” party
@seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup Thanks for your attention! Questions?

More Related Content

PPTX
Transfer learning-presentation
byBushra Jbawi
 
PDF
Transfer Learning for Natural Language Processing
bySebastian Ruder
 
PDF
Successes and Frontiers of Deep Learning
bySebastian Ruder
 
PDF
Transfer Learning -- The Next Frontier for Machine Learning
bySebastian Ruder
 
PDF
Challenges in transfer learning in nlp
byLaraOlmosCamarena
 
PDF
Apply chinese radicals into neural machine translation: deeper than character...
byLifeng (Aaron) Han
 
PDF
Incorporating Chinese Radicals Into Neural Machine Translation: Deeper Than C...
byLifeng (Aaron) Han
 
PDF
ODSC East: Effective Transfer Learning for NLP
byindico data
 
Transfer learning-presentation
byBushra Jbawi
 
Transfer Learning for Natural Language Processing
bySebastian Ruder
 
Successes and Frontiers of Deep Learning
bySebastian Ruder
 
Transfer Learning -- The Next Frontier for Machine Learning
bySebastian Ruder
 
Challenges in transfer learning in nlp
byLaraOlmosCamarena
 
Apply chinese radicals into neural machine translation: deeper than character...
byLifeng (Aaron) Han
 
Incorporating Chinese Radicals Into Neural Machine Translation: Deeper Than C...
byLifeng (Aaron) Han
 
ODSC East: Effective Transfer Learning for NLP
byindico data
 

What's hot

PPTX
Document Classification with Neo4j
byKenny Bastani
 
PDF
Intro to Deep Learning for Question Answering
byTraian Rebedea
 
PPTX
From TREC to Watson: is open domain question answering a solved problem?
byConstantin Orasan
 
PDF
Presentation of Domain Specific Question Answering System Using N-gram Approach.
byTasnim Ara Islam
 
PDF
Practical machine learning - Part 1
byTraian Rebedea
 
PDF
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
byMatthew Lease
 
PDF
Data Tactics Data Science Brown Bag (April 2014)
byRich Heimann
 
PDF
Data Tactics Analytics Brown Bag (Aug 22, 2013)
byRich Heimann
 
PPT
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
byRich Heimann
 
PDF
Arabic Question Answering: Challenges, Tasks, Approaches, Test-sets, Tools, A...
byAhmed Magdy Ezzeldin, MSc.
 
PDF
Philippe Langlais - 2017 - Users and Data: The Two Neglected Children of Bili...
byAssociation for Computational Linguistics
 
PDF
[KDD 2018 tutorial] End to-end goal-oriented question answering systems
byQi He
 
PDF
Sentence representations and question answering (YerevaNN)
byYerevaNN research lab
 
PDF
Aspects of NLP Practice
byVsevolod Dyomkin
 
PDF
Practical NLP with Lisp
byVsevolod Dyomkin
 
PDF
[AAAI 2019 tutorial] End-to-end goal-oriented question answering systems
byQi He
 
PDF
SF Python Meetup: TextRank in Python
byPaco Nathan
 
PPTX
Recurrent networks and beyond by Tomas Mikolov
byBhaskar Mitra
 
PPTX
Text, Topics, and Turkers: A Consensus Measure for Statistical Topics
byFred Morstatter
 
PPTX
Microsoft PROSE SDK: A Framework for Inductive Program Synthesis
byAlex Polozov
 
Document Classification with Neo4j
byKenny Bastani
 
Intro to Deep Learning for Question Answering
byTraian Rebedea
 
From TREC to Watson: is open domain question answering a solved problem?
byConstantin Orasan
 
Presentation of Domain Specific Question Answering System Using N-gram Approach.
byTasnim Ara Islam
 
Practical machine learning - Part 1
byTraian Rebedea
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
byMatthew Lease
 
Data Tactics Data Science Brown Bag (April 2014)
byRich Heimann
 
Data Tactics Analytics Brown Bag (Aug 22, 2013)
byRich Heimann
 
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
byRich Heimann
 
Arabic Question Answering: Challenges, Tasks, Approaches, Test-sets, Tools, A...
byAhmed Magdy Ezzeldin, MSc.
 
Philippe Langlais - 2017 - Users and Data: The Two Neglected Children of Bili...
byAssociation for Computational Linguistics
 
[KDD 2018 tutorial] End to-end goal-oriented question answering systems
byQi He
 
Sentence representations and question answering (YerevaNN)
byYerevaNN research lab
 
Aspects of NLP Practice
byVsevolod Dyomkin
 
Practical NLP with Lisp
byVsevolod Dyomkin
 
[AAAI 2019 tutorial] End-to-end goal-oriented question answering systems
byQi He
 
SF Python Meetup: TextRank in Python
byPaco Nathan
 
Recurrent networks and beyond by Tomas Mikolov
byBhaskar Mitra
 
Text, Topics, and Turkers: A Consensus Measure for Statistical Topics
byFred Morstatter
 
Microsoft PROSE SDK: A Framework for Inductive Program Synthesis
byAlex Polozov
 

Similar to NIPS 2016 Highlights - Sebastian Ruder

PDF
Deep Learning Frameworks slides
bySheamus McGovern
 
PDF
Deep learning frameworks v0.40
byJessica Willis
 
PDF
Deep learning and reasoning: Recent advances
byDeakin University
 
PDF
Deep learning 1.0 and Beyond, Part 1
byDeakin University
 
PPTX
Deep Neural Methods for Retrieval
byBhaskar Mitra
 
PDF
[系列活動] 一日搞懂生成式對抗網路
by台灣資料科學年會
 
PDF
Deep learning 1.0 and Beyond, Part 2
byDeakin University
 
PPTX
BRV CTO Summit Deep Learning Talk
byDoug Chang
 
PDF
Neuromorphic computing for neural networks
byClaudio Gallicchio
 
PDF
Data Science, Machine Learning and Neural Networks
byBICA Labs
 
PDF
Deep Learning & NLP: Graphs to the Rescue!
byRoelof Pieters
 
PDF
retrieval augmentation generation presentation slide part2
byViswakarmaChakravart
 
PDF
unit4 Neural Networks and Deep Learning.pdf
byRathiya R
 
PDF
Introduction to Deep learning Models.pdf
bycse21216
 
PPTX
Deep learning
byRatnakar Pandey
 
PPTX
Deep Learning on Qubole Data Platform
byShivaji Dutta
 
PDF
Rnn presentation 2
byShubhangi Tandon
 
PDF
Deep analytics via learning to reason
byDeakin University
 
PDF
Artificial intelligence in the post-deep learning era
byDeakin University
 
PPTX
Journey of Generative AI
bythomasjvarghese49
 
Deep Learning Frameworks slides
bySheamus McGovern
 
Deep learning frameworks v0.40
byJessica Willis
 
Deep learning and reasoning: Recent advances
byDeakin University
 
Deep learning 1.0 and Beyond, Part 1
byDeakin University
 
Deep Neural Methods for Retrieval
byBhaskar Mitra
 
[系列活動] 一日搞懂生成式對抗網路
by台灣資料科學年會
 
Deep learning 1.0 and Beyond, Part 2
byDeakin University
 
BRV CTO Summit Deep Learning Talk
byDoug Chang
 
Neuromorphic computing for neural networks
byClaudio Gallicchio
 
Data Science, Machine Learning and Neural Networks
byBICA Labs
 
Deep Learning & NLP: Graphs to the Rescue!
byRoelof Pieters
 
retrieval augmentation generation presentation slide part2
byViswakarmaChakravart
 
unit4 Neural Networks and Deep Learning.pdf
byRathiya R
 
Introduction to Deep learning Models.pdf
bycse21216
 
Deep learning
byRatnakar Pandey
 
Deep Learning on Qubole Data Platform
byShivaji Dutta
 
Rnn presentation 2
byShubhangi Tandon
 
Deep analytics via learning to reason
byDeakin University
 
Artificial intelligence in the post-deep learning era
byDeakin University
 
Journey of Generative AI
bythomasjvarghese49
 

More from Sebastian Ruder

PDF
Frontiers of Natural Language Processing
bySebastian Ruder
 
PDF
Strong Baselines for Neural Semi-supervised Learning under Domain Shift
bySebastian Ruder
 
PDF
On the Limitations of Unsupervised Bilingual Dictionary Induction
bySebastian Ruder
 
PDF
Neural Semi-supervised Learning under Domain Shift
bySebastian Ruder
 
PDF
Optimization for Deep Learning
bySebastian Ruder
 
PPTX
Human Evaluation: Why do we need it? - Dr. Sheila Castilho
bySebastian Ruder
 
PDF
Machine intelligence in HR technology: resume analysis at scale - Adrian Mihai
bySebastian Ruder
 
PDF
Hashtagger+: Real-time Social Tagging of Streaming News - Dr. Georgiana Ifrim
bySebastian Ruder
 
PDF
Making sense of word senses: An introduction to word-sense disambiguation and...
bySebastian Ruder
 
PDF
Spoken Dialogue Systems and Social Talk - Emer Gilmartin
bySebastian Ruder
 
PDF
Modeling documents with Generative Adversarial Networks - John Glover
bySebastian Ruder
 
PDF
Multi-modal Neural Machine Translation - Iacer Calixto
bySebastian Ruder
 
PDF
Funded PhD/MSc. Opportunities at AYLIEN
bySebastian Ruder
 
PDF
FaDA: Fast document aligner with word embedding - Pintu Lohar, Debasis Gangul...
bySebastian Ruder
 
PPTX
Transformation Functions for Text Classification: A case study with StackOver...
bySebastian Ruder
 
PDF
Dynamic Topic Modeling via Non-negative Matrix Factorization (Dr. Derek Greene)
bySebastian Ruder
 
PDF
Idiom Token Classification using Sentential Distributed Semantics (Giancarlo ...
bySebastian Ruder
 
PDF
A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis
bySebastian Ruder
 
PDF
Topic Listener - Observing Key Topics from Multi-Channel Speech Audio Streams...
bySebastian Ruder
 
Frontiers of Natural Language Processing
bySebastian Ruder
 
Strong Baselines for Neural Semi-supervised Learning under Domain Shift
bySebastian Ruder
 
On the Limitations of Unsupervised Bilingual Dictionary Induction
bySebastian Ruder
 
Neural Semi-supervised Learning under Domain Shift
bySebastian Ruder
 
Optimization for Deep Learning
bySebastian Ruder
 
Human Evaluation: Why do we need it? - Dr. Sheila Castilho
bySebastian Ruder
 
Machine intelligence in HR technology: resume analysis at scale - Adrian Mihai
bySebastian Ruder
 
Hashtagger+: Real-time Social Tagging of Streaming News - Dr. Georgiana Ifrim
bySebastian Ruder
 
Making sense of word senses: An introduction to word-sense disambiguation and...
bySebastian Ruder
 
Spoken Dialogue Systems and Social Talk - Emer Gilmartin
bySebastian Ruder
 
Modeling documents with Generative Adversarial Networks - John Glover
bySebastian Ruder
 
Multi-modal Neural Machine Translation - Iacer Calixto
bySebastian Ruder
 
Funded PhD/MSc. Opportunities at AYLIEN
bySebastian Ruder
 
FaDA: Fast document aligner with word embedding - Pintu Lohar, Debasis Gangul...
bySebastian Ruder
 
Transformation Functions for Text Classification: A case study with StackOver...
bySebastian Ruder
 
Dynamic Topic Modeling via Non-negative Matrix Factorization (Dr. Derek Greene)
bySebastian Ruder
 
Idiom Token Classification using Sentential Distributed Semantics (Giancarlo ...
bySebastian Ruder
 
A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis
bySebastian Ruder
 
Topic Listener - Observing Key Topics from Multi-Channel Speech Audio Streams...
bySebastian Ruder
 

Recently uploaded

PPTX
Private blockchains (or Permissioned) are created and maintained by private o...
bycscprabh
 
PDF
Enhancing the Role of Knowledge Holders in the UN CBD: Pioneering Capacity-Bu...
bypensoftservices
 
PPT
Electron Impact Scattering r-matrix method.ppt
byVinayakSavarkar2
 
PDF
Titan’s strong tidal dissipation precludes a subsurface ocean
bySérgio Sacani
 
PPTX
the achaemenid history from Cyrus till Darius III.
bymaki maki
 
PDF
Cellular Communication: Signal Transduction Pathway
byNistarini College, Purulia (W.B) India
 
PPTX
Typical_Menstrual_Cycle introductiom .pptx
bysofiamansoor4
 
PDF
Class - 10 Science Book Full Concept Explanation With Last Five Years Solved ...
byDr. Juber Hawaldar
 
PPTX
Lateral Thinking in - Managerial Skills
bycscprabh
 
PDF
Campbell Biology 12th Edition PDF Textbook
bywosarix923
 
PPTX
Biodiversity and stability.pptx for grade 10
byROLANARIBATO3
 
PDF
Child Development 9th Edition PDF Textbook
bywosarix923
 
PPTX
Block chain Multi-chain is a process where projects deploy smart contracts.pptx
bycscprabh
 
PDF
ZTF25abjmnps (AT2025ulz) and S250818k: A Candidate Superkilonova from a Subth...
bySérgio Sacani
 
PDF
Detection of disk-jet coprecession in a tidaldisruption event
bySérgio Sacani
 
PPTX
Copy of quiz bee questions for ISKOWIZARD-2019.pptx
byMaryJoyGEvangelista
 
PPTX
Anatomy of long bone and it's blood supply
byHimanshu66467
 
PPTX
Cardiac muscle physiology 2025.pptx
byshebanatalie22
 
PDF
Cosmic-Ray Bath in a Past Supernova Gives Birth to Earth-Like Planets
bySérgio Sacani
 
PPTX
RESTRICTION ENDONUCLEASE ,TYPES , MECHANISM , APPLICATION
byRemya M S
 
Private blockchains (or Permissioned) are created and maintained by private o...
bycscprabh
 
Enhancing the Role of Knowledge Holders in the UN CBD: Pioneering Capacity-Bu...
bypensoftservices
 
Electron Impact Scattering r-matrix method.ppt
byVinayakSavarkar2
 
Titan’s strong tidal dissipation precludes a subsurface ocean
bySérgio Sacani
 
the achaemenid history from Cyrus till Darius III.
bymaki maki
 
Cellular Communication: Signal Transduction Pathway
byNistarini College, Purulia (W.B) India
 
Typical_Menstrual_Cycle introductiom .pptx
bysofiamansoor4
 
Class - 10 Science Book Full Concept Explanation With Last Five Years Solved ...
byDr. Juber Hawaldar
 
Lateral Thinking in - Managerial Skills
bycscprabh
 
Campbell Biology 12th Edition PDF Textbook
bywosarix923
 
Biodiversity and stability.pptx for grade 10
byROLANARIBATO3
 
Child Development 9th Edition PDF Textbook
bywosarix923
 
Block chain Multi-chain is a process where projects deploy smart contracts.pptx
bycscprabh
 
ZTF25abjmnps (AT2025ulz) and S250818k: A Candidate Superkilonova from a Subth...
bySérgio Sacani
 
Detection of disk-jet coprecession in a tidaldisruption event
bySérgio Sacani
 
Copy of quiz bee questions for ISKOWIZARD-2019.pptx
byMaryJoyGEvangelista
 
Anatomy of long bone and it's blood supply
byHimanshu66467
 
Cardiac muscle physiology 2025.pptx
byshebanatalie22
 
Cosmic-Ray Bath in a Past Supernova Gives Birth to Earth-Like Planets
bySérgio Sacani
 
RESTRICTION ENDONUCLEASE ,TYPES , MECHANISM , APPLICATION
byRemya M S
 

NIPS 2016 Highlights - Sebastian Ruder

  • 1.
    Sebastian Ruder
 PhD Candidate,Insight Centre
 Research Scientist, AYLIEN @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup NIPS 2016 Highlights
  • 2.
    Agenda 1. NIPS overview 2.Generative Adversarial Networks 3. Building applications with Deep Learning 4. RNNs 5. Improving classic algorithms 6. Reinforcement Learning 7. Learning-to-learn / Meta-learning 8. General AI 9. NLP 10. Miscellaneous @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 3.
    NIPS Overview @seb_ruder |@_aylien |13.12.16 | 4th NLP Dublin Meetup Source: http://www.tml.cs.uni-tuebingen.de/team/luxburg/misc/nips2016/index.php
  • 4.
    Generative Adversarial Networks @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks
  • 5.
    Generative Adversarial Networks @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 6.
    Generative Adversarial Networks @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 • Yann LeCun is very bullish on GANs; lots of work on GANs from FAIR, etc.3 • Cool extensions, e.g. GA what-were network4 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks 3https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view 4https://papers.nips.cc/paper/6111-learning-what-and-where-to-draw.pdf
  • 7.
    Generative Adversarial Networks @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 8.
    Generative Adversarial Networks @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Ian Goodfellow’s GAN tutorial1 • Soumith Chintala’s “How to train your GAN” talk2 • Yann LeCun is very bullish on GANs; lots of work on GANs from FAIR, etc.3 • Cool extensions, e.g. GA what-were network4 • Mostly used in CV for image generation; discriminator can be used as feature extractor • Less work in NLP and other areas; promising directions at GAN workshop (1, 2, 3, 4) 1http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf 2https://github.com/soumith/ganhacks 3https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view 4https://papers.nips.cc/paper/6111-learning-what-and-where-to-draw.pdf 5https://arxiv.org/abs/1611.01144
  • 9.
    Building applications withDL @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 10.
    Building applications withDL @seb_ruder | @_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 11.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • 20 year anniversary of LSTM…
  • 12.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • 20 year anniversary of LSTM…
 being rejected from NIPS 1996
 — perseverance pays off
  • 13.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf
  • 14.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for- long-or-event-based-sequences.pdf
  • 15.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf 3https://papers.nips.cc/paper/6039-sequential-neural-models-with-stochastic-layers.pdf
  • 16.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3
  • 17.
    RNNs @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • RNN symposium • Many papers on improving RNNs: • Handling different time scales1,2 • Modelling uncertainty3 •General DL improvements • Weight normalisation4 • Multinomial dropout5 •Edward Grefenstette’s talk at NAMPI workshop6 1https://papers.nips.cc/paper/6057-using-fast-weights-to-attend-to-the-recent-past.pdf 2https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf 3https://papers.nips.cc/paper/6039-sequential-neural-models-with-stochastic-layers.pdf 4https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural- networks.pdf 5https://papers.nips.cc/paper/6561-improved-dropout-for-shallow-and-deep-learning.pdf 6https://uclmr.github.io/nampi/talk_slides/grefenstette_limitations_of_rnns.pdf
  • 18.
    Improving classic algorithms @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf
  • 19.
    Improving classic algorithms @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 • Clustering: Using queries to convey domain information and reduce computational complexity 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf
  • 20.
    Improving classic algorithms @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Matrix completion: Non-convex objective for matrix completion has no spurious local minima1 • Clustering: Seedings for large-scale k-means++ clustering2 • Clustering: Using queries to convey domain information and reduce computational complexity3 1https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf 2https://papers.nips.cc/paper/6478-fast-and-provably-good-seedings-for-k-means.pdf 3https://papers.nips.cc/paper/6449-clustering-with-same-cluster-queries.pdf
  • 21.
    Reinforcement Learning @seb_ruder |@_aylien |13.12.16 | 4th NLP Dublin Meetup • RL tutorial by Pieter Abbeel and John Schulman:
 Slides1; practical advice2 • Best paper: Value Iteration Networks3 • Learns to plan; contribution: differentiable approximation to classic algorithm via CNN 1http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulman-Abbeel.pdf 2http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf 3https://papers.nips.cc/paper/6046-value-iteration-networks.pdf
  • 22.
    Reinforcement Learning @seb_ruder |@_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 23.
    Reinforcement Learning @seb_ruder |@_aylien |13.12.16 | 4th NLP Dublin Meetup • RL tutorial by Pieter Abbeel and John Schulman:
 Slides1; practical advice2 • Best paper: Value Iteration Networks3 • Learns to plan; contribution: differentiable approximation to classic algorithm via CNN • New research environments: • OpenAI’s Universe • Deep Mind Lab • FAIR’s Torchcraft 1http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulman-Abbeel.pdf 2http://rll.berkeley.edu/deeprlcourse/docs/nuts-and-bolts.pdf 3https://papers.nips.cc/paper/6046-value-iteration-networks.pdf
  • 24.
    Learning-to-learn / Meta-learning @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Several papers, e.g.
 Learning to learn by gradient descent by gradient descent1 • Meta-learning panel at RNN symposium • Sutskever: No good meta-learning models so far • Schmidhuber: Focus on universal model • Focus at Neural Abstract Machines workshop 1https://papers.nips.cc/paper/6461-learning-to-learn-by-gradient-descent-by-gradient-descent.pdf
  • 25.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning
  • 26.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning
  • 27.
    General AI @seb_ruder |@_aylien |13.12.16 | 4th NLP Dublin Meetup
  • 28.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI
  • 29.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI
  • 30.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup Source: Drew Purves
  • 31.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain
  • 32.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain
  • 33.
    General Artificial Intelligence @seb_ruder| @_aylien |13.12.16 | 4th NLP Dublin Meetup • Topic in keynote talks • Yann LeCun: Focus on unsupervised learning • Drew Purves: AI for environment, ground for General AI • Saket Navlakha: Borrowing engineering principles from the brain • Machine Intelligence workshop: • FAIR’s CommAI-env • Hierarchical representations, identifying patterns • Panel: Unclear if focus on specific tasks brings us closer to General AI.
  • 34.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf
  • 35.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup
  • 36.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup
  • 37.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 • Modelling user reviews4 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf 4 https://papers.nips.cc/paper/6362-beyond-exchangeability-the-chinese-voting-process.pdf
  • 38.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup
  • 39.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup
  • 40.
    NLP @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Variations of Neural MT • Dual learning for MT (with RL)1 • Structured prediction with bandit feedback2 • Improvements to existing methods • Supervised Word Mover’s Distance3 • Modelling user reviews4 • Dialogue modelling workshop: • End-to-end systems, linguistics, and ML methods 1https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf 2 https://papers.nips.cc/paper/6134-stochastic-structured-prediction-under-bandit-feedback.pdf 3 https://papers.nips.cc/paper/6139-supervised-word-movers-distance.pdf 4 https://papers.nips.cc/paper/6362-beyond-exchangeability-the-chinese-voting-process.pdf
  • 41.
    Miscellaneous @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere!
  • 42.
    Miscellaneous @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics
  • 43.
    Miscellaneous @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing
  • 44.
    Miscellaneous @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing • AI startups: • Geometric Intelligence
  • 45.
    Miscellaneous @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup • Schmidhuber is
 everywhere! • Robotics • Apple starts publishing • AI startups: • Geometric Intelligence • RocketAI “launch” party
  • 46.
    @seb_ruder | @_aylien|13.12.16 | 4th NLP Dublin Meetup Thanks for your attention! Questions?