Downloaded 18 times
Uploaded byleopauly
Introduction to Visual transformers
The document discusses visual transformers and attention mechanisms in computer vision. It summarizes recent work on applying transformers, originally used for natural language processing, to vision tasks. This includes Vision Transformers which treat images as sequences and apply self-attention. The document reviews key papers on attention mechanisms, the Transformer architecture, and applying transformers to computer vision through Vision Transformers.
More Related Content
![ViT (Vision Transformer) Review [CDM]](https://cdn.slidesharecdn.com/ss_thumbnails/vitreviewcdm-201012184226-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Introduction to Visual transformers
Introduction to Visual transformers
- 1. Visual transformers Leo Pauly PhDstudent | Visual AI Advisors: Prof. David Hogg, Prof. Raul Fuentes University of Leeds, UK
- 2. Visual transformers Leo Pauly PhDstudent | Visual AI Advisors: Prof. David Hogg, Prof. Raul Fuentes University of Leeds, UK
- 3. Visual transformers Leo Pauly PhDstudent | Visual AI Advisors: Prof. David Hogg, Prof. Raul Fuentes University of Leeds, UK
- 4. Visual transformers Leo Pauly PhDstudent | Visual AI Advisors: Prof. David Hogg, Prof. Raul Fuentes University of Leeds, UK
- 5.
- 6. Vaswani et.al, NeurlPS2017 Dosovitskiy et.al, ICLR 2021
- 7. Vaswani et.al, NeurlPS2017 Dosovitskiy et.al, ICLR 2021 Bahdanau et.al, ICLR 2015
- 8. Dosovitskiy et.al, ICLR2021 Vaswani et.al, NeurlPS 2017 Bahdanau et.al, ICLR 2015 Sutskever et.al, NeurlPS 2014
- 9. Vaswani et.al, NeurlPS2017 Sutskever et.al, NeurlPS 2014 Dosovitskiy et.al, ICLR 2021 Bahdanau et.al, ICLR 2015
- 10. Attention Mechanism yi=RNN(yi-1,c,si-1) s1 s2 y3 yoy1 y2 c Bahdanau et.al, ICLR 2015
- 11. Attention Mechanism yi=RNN(yi-1,c,si-1) s1 s2 y3 yoy1 y2 c Bahdanau et.al, ICLR 2015 • Bottleneck at the context vector (c) • Information loss • Back propagation issues
- 12.
- 13. Attention Mechanism yi=RNN(yi-1,c,si-1) s1 s2 y3 yoy1 y2 c yi=RNN(yi-1,ci,si-1) ci=f(hj) j=1…Tx
- 14. Attention Mechanism s1 s2 y3 yoy1 y2 c yi=RNN(yi-1,ci,si-1) Figure from: https://medium.datadriveninvestor.com/attention-in-rnns-321fbcd64f05
- 15. Attention Mechanism s1 s2 y3 yoy1 y2 c yi=RNN(yi-1,ci,si-1) Figure from: https://medium.datadriveninvestor.com/attention-in-rnns-321fbcd64f05
- 16. Attention Mechanism s1 s2 y3 yoy1 y2 c yi=RNN(yi-1,ci,si-1) More reading: https://medium.datadriveninvestor.com/attention-in-rnns-321fbcd64f05
- 17. Attention Mechanism Figure from:https://trungtran.io/2019/03/29/neural-machine-translation-with-attention-mechanism/ x= y=
- 18. Attention is allyou Need Vaswani et.al, NeurlPS 2017
- 19. Attention is allyou Need
- 20. Attention is allyou Need • Scaled dot product attention • Multi-headed attention • Self attention
- 21. Attention is allyou Need
- 22. Attention is allyou Need Basics explained Y (Query) X (Value) x1 x2 x3 x1 x2 x3 X Attention Map X Output x1 x2 x3 y1 y2 y3 XT (KeyT) y1 y2 y3 Q KT V =(Q.KT). V
- 23. Attention is allyou Need Basics explained Y (Query) X (Value) x1 x2 x3 x1 x2 x3 XT (KeyT) y1 y2 y3
- 24. Attention is allyou Need Basics explained Y (Query) X (Value) x1 x2 x3 x1 x2 x3 X XT (KeyT) y1 y2 y3 Q KT
- 25. Attention is allyou Need Basics explained Y (Query) X (Value) x1 x2 x3 x1 x2 x3 X XT (KeyT) y1 y2 y3 Q KT Attention Map x1 x2 x3 y1 y2 y3
- 26. Attention is allyou Need Basics explained Y (Query) X (Value) X XT (KeyT) Q KT Attention Map ‘I’ ‘am’ ‘Leo’ ‘Je’ ‘suis’ ‘leo’ ‘I’ ‘am’ ‘Leo’ ‘Je’ ‘suis’ ‘leo’ ‘I’ ‘am’ ‘Leo’
- 27. Attention is allyou Need Basics explained Y (Query) X (Value) x1 x2 x3 x1 x2 x3 X XT (KeyT) y1 y2 y3 Q KT Attention Map x1 x2 x3 y1 y2 y3 X
- 28. Attention is allyou Need Basics explained Y (Query) X (Value) x1 x2 x3 x1 x2 x3 X Attention Map X Output x1 x2 x3 y1 y2 y3 XT (KeyT) y1 y2 y3 Q KT V =(Q.KT). V
- 29. Attention is allyou Need Basics explained Y (Query) X (Value) X Attention Map X Output XT (KeyT) Q KT V =(Q.KT). V ‘I’ ‘am’ ‘Leo’ ‘Je’ ‘suis’ ‘leo’ ‘I’ ‘am’ ‘Leo’ ‘Je’ ‘suis’ ‘leo’ ‘I’ ‘am’ ‘Leo’
- 30. Attention is allyou Need
- 31. Attention is allyou Need
- 32. Attention is allyou Need Self attention !!! X
- 33. Attention is allyou Need Transformer Architecture
- 34. Attention is allyou Need
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43. • Transformers vsCNNs : Is it worth the hype ? Vision Transformers Insights Ref: https://youtu.be/TvVc1e_4648 ? MaaS ?
- 44. • Transformers vsCNNs : Is it worth the hype ? Vision Transformers Insights ? ?
- 45. • Transformers vsCNNs : Is it worth the hype ? Vision Transformers Insights Higher resolutions ?
- 46. Vision Transformers • Canwe do (un)self-supervised pre-training ? Insights Goyal et.al, Arxiv 2021
- 47. • Architecture-level unificationacross domains Multi-modal AI systems Vision Transformers Insights
- 48.