SlideShare a Scribd company logo

Attention please! Attention Mechanism in Neural Networks

•
•
Patrick Michl
Patrick Michl

Talk about attention mechanisms in computer vision and natural language processing. Given at 4th PyData Heidelberg, Germany, 2019.

Science
1 of 27
Download to read offline
Attention Please! Attention Mechanisms in Neural Networks PyData Heidelberg #4 2019-11-21 Patrick Michl www.frootlab.org
Patrick Michl www.frootlab.org Slide 2 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Human Perception since 1990s Milestones in Attention Research
Patrick Michl www.frootlab.org Slide 3 PyData Heidelberg #4 2019-11-21 Attention Please! Image source: John Henderson and Taylor Hayes, UC Davis Visual attention in human perception is based on the dynamics between Recognition and Selection Visual Attention in Human Perception
Patrick Michl www.frootlab.org Slide 4 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception since 1990s since 2014
Patrick Michl www.frootlab.org Slide 5 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Decoder: RNN Architecture
Patrick Michl www.frootlab.org Slide 6 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Convolutional features entangle spatial with semantic information Decoder: RNN Encoder
Patrick Michl www.frootlab.org Slide 7 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Convolutional features entangle spatial with semantic information Decoder: RNN Visual Attention allows to recover spatial relations of current outputs Decoder
Patrick Michl www.frootlab.org Slide 8 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Convolutional features entangle spatial with semantic information Decoder: RNN Visual Attention allows to recover spatial relations of current outputs
Patrick Michl www.frootlab.org Slide 9 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception since 1990s since 2014 since 2015 Global Attention in Machine Translation
Patrick Michl www.frootlab.org Slide 10 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Encoder: RNN Decoder: RNN Architecture
Patrick Michl www.frootlab.org Slide 11 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Encoder: RNN Sequence of hidden states encodes the context building stack Decoder: RNN Encoder
Patrick Michl www.frootlab.org Slide 12 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Decoder Encoder: RNN Sequence of hidden states encodes the context building stack Decoder: RNN Global Attention allows to recover the context of current outputs
Patrick Michl www.frootlab.org Slide 13 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Context Encoder: RNN Sequence of hidden states encodes the context building stack Decoder: RNN Global Attention allows to recover the context of current outputs
Patrick Michl www.frootlab.org Slide 14 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception since 1990s since 2014 since 2015 since 2016 Global Attention in Machine Translation Multi-Step Attention in Natural Language Processing
Patrick Michl www.frootlab.org Slide 15 PyData Heidelberg #4 2019-11-21 Attention Please! Multi-Step Attention in Natural Language Processing Encoder: CNN Decoder: CNN Architecture
Patrick Michl www.frootlab.org Slide 16 PyData Heidelberg #4 2019-11-21 Attention Please! Multi-Step Attention in Natural Language Processing Encoder: CNN Sequence of convolutional features encodes the context building stack Decoder: CNN Encoder
Patrick Michl www.frootlab.org Slide 17 PyData Heidelberg #4 2019-11-21 Attention Please! Multi-Step Attention in Natural Language Processing Encoder: CNN Sequence of convolutional features encodes the context building stack Decoder: CNN Multi-Step Attention allows to recover the current context of current deconvolutional steps Decoder
Patrick Michl www.frootlab.org Slide 18 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception Self-Attention in Natural Language Processing since 1990s since 2014 since 2015 since 2016 since 2017 Global Attention in Machine Translation Multi-Step Attention in Natural Language Processing
Patrick Michl www.frootlab.org Slide 19 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Decoder: Stacked ANN Architecture
Patrick Michl www.frootlab.org Slide 20 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Sequence of hidden features captures the encoder context building stack Decoder: Stacked ANN Encoder Stack
Patrick Michl www.frootlab.org Slide 21 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Sequence of hidden features captures the encoder context building stack Decoder: Stacked ANN Multi-Step Attention unrolls the context stack to the decoder Decoder Stack
Patrick Michl www.frootlab.org Slide 22 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Sequence of hidden features captures the encoder context building stack Decoder: Stacked ANN Multi-Step Attention unrolls the context stack to the decoder ------ Self-attention aggregates context dependencies within the inputs Self-Attention
Patrick Michl www.frootlab.org Slide 23 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #1 Human Perception is based on the dynamics between selection and recognition
Patrick Michl www.frootlab.org Slide 24 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #2 Attention machanisms immitate this behaviour by using intermediate states, that entangle context with semantic information #1 Human Perception is based on the dynamics between selection and recognition
Patrick Michl www.frootlab.org Slide 25 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #2 Attention machanisms immitate this behaviour by using intermediate states, that entangle context with semantic information #1 Human Perception is based on the dynamics between selection and recognition #3 The incorporation of context provides dynamic features that are context specific and therefore improve the model perfomance
Patrick Michl www.frootlab.org Slide 26 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #2 Attention machanisms immitate this behaviour by using intermediate states, that entangle context with semantic information #1 Human Perception is based on the dynamics between selection and recognition #3 The incorporation of context provides dynamic features that are context specific and therefore improve the model perfomance #4 After all - attention mechanisms can also help to understand the decisions of deep networks
Patrick Michl www.frootlab.org Slide 27 PyData Heidelberg #4 2019-11-21 Attention Please! Thank you for your attention!

Recommended

Attention Please! 2022
Attention Please! 2022Attention Please! 2022
Attention Please! 2022Patrick Michl
 
Digitizing Local Collections
Digitizing Local CollectionsDigitizing Local Collections
Digitizing Local CollectionsBethany Davis
 
Requirements Conversations: A New Frontier in AI-for-RE
Requirements Conversations: A New Frontier in AI-for-RERequirements Conversations: A New Frontier in AI-for-RE
Requirements Conversations: A New Frontier in AI-for-REFabiano Dalpiaz
 
Scientix: Projectes europeus RRI i CSC Recursos per l’aula.
Scientix: Projectes europeus RRI i CSC Recursos per l’aula.Scientix: Projectes europeus RRI i CSC Recursos per l’aula.
Scientix: Projectes europeus RRI i CSC Recursos per l’aula.cienciaicontroversia
 
Socially-aware Traffic Management (Workshop Sozioinformatik)
Socially-aware Traffic Management (Workshop Sozioinformatik)Socially-aware Traffic Management (Workshop Sozioinformatik)
Socially-aware Traffic Management (Workshop Sozioinformatik)SmartenIT
 
An Introduction to digital preservation at the Library of Congress
An Introduction to digital preservation at the Library of CongressAn Introduction to digital preservation at the Library of Congress
An Introduction to digital preservation at the Library of Congresslljohnston
 
Training Week: Create a Knowledge Graph: A Simple ML Approach
Training Week: Create a Knowledge Graph: A Simple ML Approach Training Week: Create a Knowledge Graph: A Simple ML Approach
Training Week: Create a Knowledge Graph: A Simple ML Approach Neo4j
 
Idq summit2014 ronald damhof - it's all about the data
Idq summit2014 ronald damhof - it's all about the dataIdq summit2014 ronald damhof - it's all about the data
Idq summit2014 ronald damhof - it's all about the dataPrudenza B.V
 

Recommended

Attention Please! 2022
Attention Please! 2022Attention Please! 2022
Attention Please! 2022Patrick Michl
 
Digitizing Local Collections
Digitizing Local CollectionsDigitizing Local Collections
Digitizing Local CollectionsBethany Davis
 
Requirements Conversations: A New Frontier in AI-for-RE
Requirements Conversations: A New Frontier in AI-for-RERequirements Conversations: A New Frontier in AI-for-RE
Requirements Conversations: A New Frontier in AI-for-REFabiano Dalpiaz
 
Scientix: Projectes europeus RRI i CSC Recursos per l’aula.
Scientix: Projectes europeus RRI i CSC Recursos per l’aula.Scientix: Projectes europeus RRI i CSC Recursos per l’aula.
Scientix: Projectes europeus RRI i CSC Recursos per l’aula.cienciaicontroversia
 
Socially-aware Traffic Management (Workshop Sozioinformatik)
Socially-aware Traffic Management (Workshop Sozioinformatik)Socially-aware Traffic Management (Workshop Sozioinformatik)
Socially-aware Traffic Management (Workshop Sozioinformatik)SmartenIT
 
An Introduction to digital preservation at the Library of Congress
An Introduction to digital preservation at the Library of CongressAn Introduction to digital preservation at the Library of Congress
An Introduction to digital preservation at the Library of Congresslljohnston
 
Training Week: Create a Knowledge Graph: A Simple ML Approach
Training Week: Create a Knowledge Graph: A Simple ML Approach Training Week: Create a Knowledge Graph: A Simple ML Approach
Training Week: Create a Knowledge Graph: A Simple ML Approach Neo4j
 
Idq summit2014 ronald damhof - it's all about the data
Idq summit2014 ronald damhof - it's all about the dataIdq summit2014 ronald damhof - it's all about the data
Idq summit2014 ronald damhof - it's all about the dataPrudenza B.V
 
Puglia marac-file formats-20111020
Puglia marac-file formats-20111020Puglia marac-file formats-20111020
Puglia marac-file formats-20111020MARAC Bethlehem PC
 
Digital Preservation Best Practices: Lessons Learned From Across the Pond
Digital Preservation Best Practices: Lessons Learned From Across the PondDigital Preservation Best Practices: Lessons Learned From Across the Pond
Digital Preservation Best Practices: Lessons Learned From Across the PondBenoit Pauwels
 
Digital Presentation Best Practices: Lessons Learned From Across the Pond
Digital Presentation Best Practices: Lessons Learned From Across the PondDigital Presentation Best Practices: Lessons Learned From Across the Pond
Digital Presentation Best Practices: Lessons Learned From Across the PondULB - Bibliothèques
 
Humans & Machines Together
Humans & Machines TogetherHumans & Machines Together
Humans & Machines TogetherLora Aroyo
 
Presentation 17 may keynote lara aroyo
Presentation 17 may keynote lara aroyoPresentation 17 may keynote lara aroyo
Presentation 17 may keynote lara aroyoNederlands Instituut voor Beeld en Geluid
 
Personal Digital Archiving Initiatives at the Library of Congress
Personal Digital Archiving Initiatives at the Library of CongressPersonal Digital Archiving Initiatives at the Library of Congress
Personal Digital Archiving Initiatives at the Library of Congresslljohnston
 
Collecting and Coding Twitter Data in DiscoverText
Collecting and Coding Twitter Data in DiscoverTextCollecting and Coding Twitter Data in DiscoverText
Collecting and Coding Twitter Data in DiscoverTextJill Hopke
 
Semantics And Multimedia
Semantics And MultimediaSemantics And Multimedia
Semantics And MultimediaPeter Berger
 
From Essence to Assets. Making sense of an audiovisual archive
From Essence to Assets. Making sense of an audiovisual archiveFrom Essence to Assets. Making sense of an audiovisual archive
From Essence to Assets. Making sense of an audiovisual archiveBrecht Declercq
 
Digital Technology and Programming Combined Presentation
Digital Technology and Programming Combined PresentationDigital Technology and Programming Combined Presentation
Digital Technology and Programming Combined PresentationRobert Casto
 
Data Security and the Hard Outer Shell
Data Security and the Hard Outer ShellData Security and the Hard Outer Shell
Data Security and the Hard Outer ShellMatt Turner
 
Five Ways To Do Data Analytics "The Wrong Way"
Five Ways To Do Data Analytics "The Wrong Way"Five Ways To Do Data Analytics "The Wrong Way"
Five Ways To Do Data Analytics "The Wrong Way"Discover Pinterest
 
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdfAdvanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdfWiseNaeem
 
Is Talent the Missing Link in the Next Evolution of Supply Chain?
Is Talent the Missing Link in the Next Evolution of Supply Chain?Is Talent the Missing Link in the Next Evolution of Supply Chain?
Is Talent the Missing Link in the Next Evolution of Supply Chain?Lora Cecere
 
Memahamialirprosesproduksiprodukmultimedia3 (ind)
Memahamialirprosesproduksiprodukmultimedia3 (ind)Memahamialirprosesproduksiprodukmultimedia3 (ind)
Memahamialirprosesproduksiprodukmultimedia3 (ind)Sayugo
 
Mag pi18 Citation "PhotoReportage"
Mag pi18 Citation "PhotoReportage"Mag pi18 Citation "PhotoReportage"
Mag pi18 Citation "PhotoReportage"Arnaud VELTEN (BUSINESS COMMANDO)
 
Crowdsourcing video annotation
Crowdsourcing video annotationCrowdsourcing video annotation
Crowdsourcing video annotationMichiel Hildebrand
 
xAPI in Action
xAPI in ActionxAPI in Action
xAPI in Actionbbetts
 
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections Kara Van Malssen
 
A Hybrid Memory Data Cube Approach for High Dimension Relations
A Hybrid Memory Data Cube Approach for High Dimension RelationsA Hybrid Memory Data Cube Approach for High Dimension Relations
A Hybrid Memory Data Cube Approach for High Dimension RelationsRodrigo Rocha Silva
 
Spec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRSSpec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRSPatrick Michl
 
Structure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksStructure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksPatrick Michl
 

More Related Content

Similar to Attention please! Attention Mechanism in Neural Networks

Puglia marac-file formats-20111020
Puglia marac-file formats-20111020Puglia marac-file formats-20111020
Puglia marac-file formats-20111020MARAC Bethlehem PC
 
Digital Preservation Best Practices: Lessons Learned From Across the Pond
Digital Preservation Best Practices: Lessons Learned From Across the PondDigital Preservation Best Practices: Lessons Learned From Across the Pond
Digital Preservation Best Practices: Lessons Learned From Across the PondBenoit Pauwels
 
Digital Presentation Best Practices: Lessons Learned From Across the Pond
Digital Presentation Best Practices: Lessons Learned From Across the PondDigital Presentation Best Practices: Lessons Learned From Across the Pond
Digital Presentation Best Practices: Lessons Learned From Across the PondULB - Bibliothèques
 
Humans & Machines Together
Humans & Machines TogetherHumans & Machines Together
Humans & Machines TogetherLora Aroyo
 
Personal Digital Archiving Initiatives at the Library of Congress
Personal Digital Archiving Initiatives at the Library of CongressPersonal Digital Archiving Initiatives at the Library of Congress
Personal Digital Archiving Initiatives at the Library of Congresslljohnston
 
Collecting and Coding Twitter Data in DiscoverText
Collecting and Coding Twitter Data in DiscoverTextCollecting and Coding Twitter Data in DiscoverText
Collecting and Coding Twitter Data in DiscoverTextJill Hopke
 
Semantics And Multimedia
Semantics And MultimediaSemantics And Multimedia
Semantics And MultimediaPeter Berger
 
From Essence to Assets. Making sense of an audiovisual archive
From Essence to Assets. Making sense of an audiovisual archiveFrom Essence to Assets. Making sense of an audiovisual archive
From Essence to Assets. Making sense of an audiovisual archiveBrecht Declercq
 
Digital Technology and Programming Combined Presentation
Digital Technology and Programming Combined PresentationDigital Technology and Programming Combined Presentation
Digital Technology and Programming Combined PresentationRobert Casto
 
Data Security and the Hard Outer Shell
Data Security and the Hard Outer ShellData Security and the Hard Outer Shell
Data Security and the Hard Outer ShellMatt Turner
 
Five Ways To Do Data Analytics "The Wrong Way"
Five Ways To Do Data Analytics "The Wrong Way"Five Ways To Do Data Analytics "The Wrong Way"
Five Ways To Do Data Analytics "The Wrong Way"Discover Pinterest
 
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdfAdvanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdfWiseNaeem
 
Is Talent the Missing Link in the Next Evolution of Supply Chain?
Is Talent the Missing Link in the Next Evolution of Supply Chain?Is Talent the Missing Link in the Next Evolution of Supply Chain?
Is Talent the Missing Link in the Next Evolution of Supply Chain?Lora Cecere
 
Memahamialirprosesproduksiprodukmultimedia3 (ind)
Memahamialirprosesproduksiprodukmultimedia3 (ind)Memahamialirprosesproduksiprodukmultimedia3 (ind)
Memahamialirprosesproduksiprodukmultimedia3 (ind)Sayugo
 
Crowdsourcing video annotation
Crowdsourcing video annotationCrowdsourcing video annotation
Crowdsourcing video annotationMichiel Hildebrand
 
xAPI in Action
xAPI in ActionxAPI in Action
xAPI in Actionbbetts
 
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections Kara Van Malssen
 
A Hybrid Memory Data Cube Approach for High Dimension Relations
A Hybrid Memory Data Cube Approach for High Dimension RelationsA Hybrid Memory Data Cube Approach for High Dimension Relations
A Hybrid Memory Data Cube Approach for High Dimension RelationsRodrigo Rocha Silva
 

Similar to Attention please! Attention Mechanism in Neural Networks (20)

Puglia marac-file formats-20111020
Puglia marac-file formats-20111020Puglia marac-file formats-20111020
Puglia marac-file formats-20111020
 
Digital Preservation Best Practices: Lessons Learned From Across the Pond
Digital Preservation Best Practices: Lessons Learned From Across the PondDigital Preservation Best Practices: Lessons Learned From Across the Pond
Digital Preservation Best Practices: Lessons Learned From Across the Pond
 
Digital Presentation Best Practices: Lessons Learned From Across the Pond
Digital Presentation Best Practices: Lessons Learned From Across the PondDigital Presentation Best Practices: Lessons Learned From Across the Pond
Digital Presentation Best Practices: Lessons Learned From Across the Pond
 
Humans & Machines Together
Humans & Machines TogetherHumans & Machines Together
Humans & Machines Together
 
Presentation 17 may keynote lara aroyo
Presentation 17 may keynote lara aroyoPresentation 17 may keynote lara aroyo
Presentation 17 may keynote lara aroyo
 
Personal Digital Archiving Initiatives at the Library of Congress
Personal Digital Archiving Initiatives at the Library of CongressPersonal Digital Archiving Initiatives at the Library of Congress
Personal Digital Archiving Initiatives at the Library of Congress
 
Collecting and Coding Twitter Data in DiscoverText
Collecting and Coding Twitter Data in DiscoverTextCollecting and Coding Twitter Data in DiscoverText
Collecting and Coding Twitter Data in DiscoverText
 
Semantics And Multimedia
Semantics And MultimediaSemantics And Multimedia
Semantics And Multimedia
 
From Essence to Assets. Making sense of an audiovisual archive
From Essence to Assets. Making sense of an audiovisual archiveFrom Essence to Assets. Making sense of an audiovisual archive
From Essence to Assets. Making sense of an audiovisual archive
 
Digital Technology and Programming Combined Presentation
Digital Technology and Programming Combined PresentationDigital Technology and Programming Combined Presentation
Digital Technology and Programming Combined Presentation
 
Data Security and the Hard Outer Shell
Data Security and the Hard Outer ShellData Security and the Hard Outer Shell
Data Security and the Hard Outer Shell
 
Five Ways To Do Data Analytics "The Wrong Way"
Five Ways To Do Data Analytics "The Wrong Way"Five Ways To Do Data Analytics "The Wrong Way"
Five Ways To Do Data Analytics "The Wrong Way"
 
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdfAdvanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
Advanced View of Projects Raspberry Pi List - Raspberry PI Projects.pdf
 
Is Talent the Missing Link in the Next Evolution of Supply Chain?
Is Talent the Missing Link in the Next Evolution of Supply Chain?Is Talent the Missing Link in the Next Evolution of Supply Chain?
Is Talent the Missing Link in the Next Evolution of Supply Chain?
 
Memahamialirprosesproduksiprodukmultimedia3 (ind)
Memahamialirprosesproduksiprodukmultimedia3 (ind)Memahamialirprosesproduksiprodukmultimedia3 (ind)
Memahamialirprosesproduksiprodukmultimedia3 (ind)
 
Mag pi18 Citation "PhotoReportage"
Mag pi18 Citation "PhotoReportage"Mag pi18 Citation "PhotoReportage"
Mag pi18 Citation "PhotoReportage"
 
Crowdsourcing video annotation
Crowdsourcing video annotationCrowdsourcing video annotation
Crowdsourcing video annotation
 
xAPI in Action
xAPI in ActionxAPI in Action
xAPI in Action
 
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
Planning Beyond Digitization: Digital Preservation for Audiovisual Collections
 
A Hybrid Memory Data Cube Approach for High Dimension Relations
A Hybrid Memory Data Cube Approach for High Dimension RelationsA Hybrid Memory Data Cube Approach for High Dimension Relations
A Hybrid Memory Data Cube Approach for High Dimension Relations
 

More from Patrick Michl

Spec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRSSpec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRSPatrick Michl
 
Structure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksStructure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksPatrick Michl
 
Structure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksStructure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksPatrick Michl
 
Structure learning with Deep Autoencoders
Structure learning with Deep AutoencodersStructure learning with Deep Autoencoders
Structure learning with Deep AutoencodersPatrick Michl
 
Regulation Analysis using Restricted Boltzmann Machines
Regulation Analysis using Restricted Boltzmann MachinesRegulation Analysis using Restricted Boltzmann Machines
Regulation Analysis using Restricted Boltzmann MachinesPatrick Michl
 
Concept of Regulation Analysis using Restricted Boltzmann Machines
Concept of Regulation Analysis using Restricted Boltzmann MachinesConcept of Regulation Analysis using Restricted Boltzmann Machines
Concept of Regulation Analysis using Restricted Boltzmann MachinesPatrick Michl
 
Synchronization and Collective Dynamics
Synchronization and Collective DynamicsSynchronization and Collective Dynamics
Synchronization and Collective DynamicsPatrick Michl
 

More from Patrick Michl (8)

Spec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRSSpec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRS
 
Structure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksStructure learning with Deep Neural Networks
Structure learning with Deep Neural Networks
 
Structure learning with Deep Neural Networks
Structure learning with Deep Neural NetworksStructure learning with Deep Neural Networks
Structure learning with Deep Neural Networks
 
Structure learning with Deep Autoencoders
Structure learning with Deep AutoencodersStructure learning with Deep Autoencoders
Structure learning with Deep Autoencoders
 
Regulation Analysis using Restricted Boltzmann Machines
Regulation Analysis using Restricted Boltzmann MachinesRegulation Analysis using Restricted Boltzmann Machines
Regulation Analysis using Restricted Boltzmann Machines
 
Concept of Regulation Analysis using Restricted Boltzmann Machines
Concept of Regulation Analysis using Restricted Boltzmann MachinesConcept of Regulation Analysis using Restricted Boltzmann Machines
Concept of Regulation Analysis using Restricted Boltzmann Machines
 
Synchronization and Collective Dynamics
Synchronization and Collective DynamicsSynchronization and Collective Dynamics
Synchronization and Collective Dynamics
 
Epidemic Spreading
Epidemic SpreadingEpidemic Spreading
Epidemic Spreading
 

Recently uploaded

Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 bAsymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 bSérgio Sacani
 
Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.Silpa
 
300003-World Science Day For Peace And Development.pptx
300003-World Science Day For Peace And Development.pptx300003-World Science Day For Peace And Development.pptx
300003-World Science Day For Peace And Development.pptxryanrooker
 
Genome sequencing,shotgun sequencing.pptx
Genome sequencing,shotgun sequencing.pptxGenome sequencing,shotgun sequencing.pptx
Genome sequencing,shotgun sequencing.pptxSilpa
 
LUNULARIA -features, morphology, anatomy ,reproduction etc.
LUNULARIA -features, morphology, anatomy ,reproduction etc.LUNULARIA -features, morphology, anatomy ,reproduction etc.
LUNULARIA -features, morphology, anatomy ,reproduction etc.Silpa
 
Thyroid Physiology_Dr.E. Muralinath_ Associate Professor
Thyroid Physiology_Dr.E. Muralinath_ Associate ProfessorThyroid Physiology_Dr.E. Muralinath_ Associate Professor
Thyroid Physiology_Dr.E. Muralinath_ Associate Professormuralinath2
 
Molecular markers- RFLP, RAPD, AFLP, SNP etc.
Molecular markers- RFLP, RAPD, AFLP, SNP etc.Molecular markers- RFLP, RAPD, AFLP, SNP etc.
Molecular markers- RFLP, RAPD, AFLP, SNP etc.Silpa
 
Module for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learningModule for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learninglevieagacer
 
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRLGwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRLkantirani197
 
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIACURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIADr. TATHAGAT KHOBRAGADE
 
GBSN - Microbiology (Unit 3)Defense Mechanism of the body
GBSN - Microbiology (Unit 3)Defense Mechanism of the body GBSN - Microbiology (Unit 3)Defense Mechanism of the body
GBSN - Microbiology (Unit 3)Defense Mechanism of the body Areesha Ahmad
 
Human genetics..........................pptx
Human genetics..........................pptxHuman genetics..........................pptx
Human genetics..........................pptxSilpa
 
Atp synthase , Atp synthase complex 1 to 4.
Atp synthase , Atp synthase complex 1 to 4.Atp synthase , Atp synthase complex 1 to 4.
Atp synthase , Atp synthase complex 1 to 4.Silpa
 
The Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptxThe Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptxseri bangash
 
Use of mutants in understanding seedling development.pptx
Use of mutants in understanding seedling development.pptxUse of mutants in understanding seedling development.pptx
Use of mutants in understanding seedling development.pptxRenuJangid3
 
GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry
GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry
GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry Areesha Ahmad
 
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....muralinath2
 
CYTOGENETIC MAP................ ppt.pptx
CYTOGENETIC MAP................ ppt.pptxCYTOGENETIC MAP................ ppt.pptx
CYTOGENETIC MAP................ ppt.pptxSilpa
 
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune WaterworldsBiogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune WaterworldsSérgio Sacani
 

Recently uploaded (20)

Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 bAsymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
 
Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.Phenolics: types, biosynthesis and functions.
Phenolics: types, biosynthesis and functions.
 
PATNA CALL GIRLS 8617370543 LOW PRICE ESCORT SERVICE
PATNA CALL GIRLS 8617370543 LOW PRICE ESCORT SERVICEPATNA CALL GIRLS 8617370543 LOW PRICE ESCORT SERVICE
PATNA CALL GIRLS 8617370543 LOW PRICE ESCORT SERVICE
 
300003-World Science Day For Peace And Development.pptx
300003-World Science Day For Peace And Development.pptx300003-World Science Day For Peace And Development.pptx
300003-World Science Day For Peace And Development.pptx
 
Genome sequencing,shotgun sequencing.pptx
Genome sequencing,shotgun sequencing.pptxGenome sequencing,shotgun sequencing.pptx
Genome sequencing,shotgun sequencing.pptx
 
LUNULARIA -features, morphology, anatomy ,reproduction etc.
LUNULARIA -features, morphology, anatomy ,reproduction etc.LUNULARIA -features, morphology, anatomy ,reproduction etc.
LUNULARIA -features, morphology, anatomy ,reproduction etc.
 
Thyroid Physiology_Dr.E. Muralinath_ Associate Professor
Thyroid Physiology_Dr.E. Muralinath_ Associate ProfessorThyroid Physiology_Dr.E. Muralinath_ Associate Professor
Thyroid Physiology_Dr.E. Muralinath_ Associate Professor
 
Molecular markers- RFLP, RAPD, AFLP, SNP etc.
Molecular markers- RFLP, RAPD, AFLP, SNP etc.Molecular markers- RFLP, RAPD, AFLP, SNP etc.
Molecular markers- RFLP, RAPD, AFLP, SNP etc.
 
Module for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learningModule for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learning
 
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRLGwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
Gwalior ❤CALL GIRL 84099*07087 ❤CALL GIRLS IN Gwalior ESCORT SERVICE❤CALL GIRL
 
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIACURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
CURRENT SCENARIO OF POULTRY PRODUCTION IN INDIA
 
GBSN - Microbiology (Unit 3)Defense Mechanism of the body
GBSN - Microbiology (Unit 3)Defense Mechanism of the body GBSN - Microbiology (Unit 3)Defense Mechanism of the body
GBSN - Microbiology (Unit 3)Defense Mechanism of the body
 
Human genetics..........................pptx
Human genetics..........................pptxHuman genetics..........................pptx
Human genetics..........................pptx
 
Atp synthase , Atp synthase complex 1 to 4.
Atp synthase , Atp synthase complex 1 to 4.Atp synthase , Atp synthase complex 1 to 4.
Atp synthase , Atp synthase complex 1 to 4.
 
The Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptxThe Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptx
 
Use of mutants in understanding seedling development.pptx
Use of mutants in understanding seedling development.pptxUse of mutants in understanding seedling development.pptx
Use of mutants in understanding seedling development.pptx
 
GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry
GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry
GBSN - Biochemistry (Unit 2) Basic concept of organic chemistry
 
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
 
CYTOGENETIC MAP................ ppt.pptx
CYTOGENETIC MAP................ ppt.pptxCYTOGENETIC MAP................ ppt.pptx
CYTOGENETIC MAP................ ppt.pptx
 
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune WaterworldsBiogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
 

Attention please! Attention Mechanism in Neural Networks

  • 1. Attention Please! Attention Mechanisms in Neural Networks PyData Heidelberg #4 2019-11-21 Patrick Michl www.frootlab.org
  • 2. Patrick Michl www.frootlab.org Slide 2 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Human Perception since 1990s Milestones in Attention Research
  • 3. Patrick Michl www.frootlab.org Slide 3 PyData Heidelberg #4 2019-11-21 Attention Please! Image source: John Henderson and Taylor Hayes, UC Davis Visual attention in human perception is based on the dynamics between Recognition and Selection Visual Attention in Human Perception
  • 4. Patrick Michl www.frootlab.org Slide 4 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception since 1990s since 2014
  • 5. Patrick Michl www.frootlab.org Slide 5 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Decoder: RNN Architecture
  • 6. Patrick Michl www.frootlab.org Slide 6 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Convolutional features entangle spatial with semantic information Decoder: RNN Encoder
  • 7. Patrick Michl www.frootlab.org Slide 7 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Convolutional features entangle spatial with semantic information Decoder: RNN Visual Attention allows to recover spatial relations of current outputs Decoder
  • 8. Patrick Michl www.frootlab.org Slide 8 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Encoder: CNN Convolutional features entangle spatial with semantic information Decoder: RNN Visual Attention allows to recover spatial relations of current outputs
  • 9. Patrick Michl www.frootlab.org Slide 9 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception since 1990s since 2014 since 2015 Global Attention in Machine Translation
  • 10. Patrick Michl www.frootlab.org Slide 10 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Encoder: RNN Decoder: RNN Architecture
  • 11. Patrick Michl www.frootlab.org Slide 11 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Encoder: RNN Sequence of hidden states encodes the context building stack Decoder: RNN Encoder
  • 12. Patrick Michl www.frootlab.org Slide 12 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Decoder Encoder: RNN Sequence of hidden states encodes the context building stack Decoder: RNN Global Attention allows to recover the context of current outputs
  • 13. Patrick Michl www.frootlab.org Slide 13 PyData Heidelberg #4 2019-11-21 Attention Please! Global Attention in Machine Translation Context Encoder: RNN Sequence of hidden states encodes the context building stack Decoder: RNN Global Attention allows to recover the context of current outputs
  • 14. Patrick Michl www.frootlab.org Slide 14 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception since 1990s since 2014 since 2015 since 2016 Global Attention in Machine Translation Multi-Step Attention in Natural Language Processing
  • 15. Patrick Michl www.frootlab.org Slide 15 PyData Heidelberg #4 2019-11-21 Attention Please! Multi-Step Attention in Natural Language Processing Encoder: CNN Decoder: CNN Architecture
  • 16. Patrick Michl www.frootlab.org Slide 16 PyData Heidelberg #4 2019-11-21 Attention Please! Multi-Step Attention in Natural Language Processing Encoder: CNN Sequence of convolutional features encodes the context building stack Decoder: CNN Encoder
  • 17. Patrick Michl www.frootlab.org Slide 17 PyData Heidelberg #4 2019-11-21 Attention Please! Multi-Step Attention in Natural Language Processing Encoder: CNN Sequence of convolutional features encodes the context building stack Decoder: CNN Multi-Step Attention allows to recover the current context of current deconvolutional steps Decoder
  • 18. Patrick Michl www.frootlab.org Slide 18 PyData Heidelberg #4 2019-11-21 Attention Please! Visual Attention in Image Captioning Visual Attention in Human Perception Self-Attention in Natural Language Processing since 1990s since 2014 since 2015 since 2016 since 2017 Global Attention in Machine Translation Multi-Step Attention in Natural Language Processing
  • 19. Patrick Michl www.frootlab.org Slide 19 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Decoder: Stacked ANN Architecture
  • 20. Patrick Michl www.frootlab.org Slide 20 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Sequence of hidden features captures the encoder context building stack Decoder: Stacked ANN Encoder Stack
  • 21. Patrick Michl www.frootlab.org Slide 21 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Sequence of hidden features captures the encoder context building stack Decoder: Stacked ANN Multi-Step Attention unrolls the context stack to the decoder Decoder Stack
  • 22. Patrick Michl www.frootlab.org Slide 22 PyData Heidelberg #4 2019-11-21 Attention Please! Self-Attention in Natural Language Processing Encoder: Stacked ANN Sequence of hidden features captures the encoder context building stack Decoder: Stacked ANN Multi-Step Attention unrolls the context stack to the decoder ------ Self-attention aggregates context dependencies within the inputs Self-Attention
  • 23. Patrick Michl www.frootlab.org Slide 23 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #1 Human Perception is based on the dynamics between selection and recognition
  • 24. Patrick Michl www.frootlab.org Slide 24 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #2 Attention machanisms immitate this behaviour by using intermediate states, that entangle context with semantic information #1 Human Perception is based on the dynamics between selection and recognition
  • 25. Patrick Michl www.frootlab.org Slide 25 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #2 Attention machanisms immitate this behaviour by using intermediate states, that entangle context with semantic information #1 Human Perception is based on the dynamics between selection and recognition #3 The incorporation of context provides dynamic features that are context specific and therefore improve the model perfomance
  • 26. Patrick Michl www.frootlab.org Slide 26 PyData Heidelberg #4 2019-11-21 Attention Please! Summary #2 Attention machanisms immitate this behaviour by using intermediate states, that entangle context with semantic information #1 Human Perception is based on the dynamics between selection and recognition #3 The incorporation of context provides dynamic features that are context specific and therefore improve the model perfomance #4 After all - attention mechanisms can also help to understand the decisions of deep networks
  • 27. Patrick Michl www.frootlab.org Slide 27 PyData Heidelberg #4 2019-11-21 Attention Please! Thank you for your attention!