【DL輪読会】Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
•
1 like•400 views
The document proposes modifications to self-attention in Transformers to improve faithful signal propagation without shortcuts like skip connections or layer normalization. Specifically, it introduces a normalization-free network that uses dynamic isometry to ensure unitary transformations, a ReZero technique to implement skip connections without adding shortcuts, and modifications to attention and normalization techniques to address issues like rank collapse in Transformers. The methods are evaluated on tasks like CIFAR-10 classification and language modeling, demonstrating improved performance over standard Transformer architectures.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to 【DL輪読会】Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Similar to 【DL輪読会】Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation (20)
More from Deep Learning JP
More from Deep Learning JP (20)
Recently uploaded
Recently uploaded (20)
【DL輪読会】Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
- 1. Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation Shohei Taniguchi, Matsuo Lab 1
- 2. Deep Transformers without Shortcuts ॻࢽใ ஶऀ • Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, Yee Whye Teh (DeepMind) ֓ཁ • TransformerΛlayer normalizationskip connectionͳ͠ͰֶशͰ͖ΔΑ͏ʹվྑ • ICLR 2023 accepted 2
- 3. ൃද֓ཁ • എܠ • ؔ࿈ڀݚ • ख๏ • ࣮݁ݧՌ • ·ͱΊ 3
- 4. എܠ Transformer • TransformerAttentionͱMLPͷ܁Γฦ͠ • ֤ϞδϡʔϧͰskip connectionͱlayer normalizationΛ ద༻͢Δͷ͕Ұൠత • ͜ΕΒ͕࣮ࡍʹͲ͏͍͏ׂΛՌ͍ͨͯ͠Δ͔ ݱঢ়ෆ໌ • ͏·ֶ͘श͢ΔͨΊͷςΫχοΫͱ͍͏Ґஔ͚ 4
- 5. ؔ࿈ڀݚ Normalization-free Network • MLPCNNͰɼskip connectionਖ਼نԽ͕ͳͯ͘ਂ͍ωοτϫʔΫΛֶश Ͱ͖Δํ๏͕ΒΕ͍ͯΔ • جຊతʹɼޯফࣦ/രൃ͕͜ىΒͳ͍Α͏ʹదʹॏΈͷॳظԽΛߦ͑ ਖ਼نԽͳͲΛΘͳͯ͘େৎ • Dynamic isometryͱ͍͏֓೦͕ಛʹॏཁ 5
- 6. Isometry ํੑ ͷMLPΛߟ͑Δͱɼೖྗ͔Βग़ྗͷϠίϏߦྻ ɼ֤ͷॏΈͷߦྻੵ ͨͩ͠ɼ Λຬͨ͢ର֯ߦྻ L J xl = ϕ (hl ), hl = Wl xl−1 + bl J = ∂xL ∂h0 = L ∏ l=1 Dl Wl Dl Dl ij = ϕ′  (hl i) δij 6
- 7. Isometry ํੑ • ͜ͷϠίϏߦྻ͕ফࣦ/രൃ͍ͯ͠ͳ͚Εɼ҆ఆֶͯ͠शͰ͖Δͣ ߦྻͷಛҟ͕1ۙʹͳ͍ͬͯΕྑ͍ • ͷಛҟͷฏ͕ۉ1ͷͱ͖ɼํੑΛຬͨ͢ • ʹ͍ͭͯɼ׆ੑԽ͕ؔ߃ͰۙݪؔͳΒํతʢtanhͳͲʣ J = ∂xL ∂h0 = L ∏ l=1 Dl Wl Wl Dl 7
- 8. Dynamic Isometry ಈతํੑ • ͞Βʹɼͯ͢ͷಛҟ͕1ͷͱ͖ɼಈతํੑΛຬͨ͢ • ͜ΕΛຬͨ͢ͷɼॏΈ͕ަߦྻͷͱ͖ ަॳظԽΛ͢Εɼޯফࣦ/രൃ͠ͳ͍ʂ J = ∂xL ∂h0 = L ∏ l=1 Dl Wl 8
- 9. ؔ࿈ڀݚ [1] • MLPͰCIFAR-10ͷྨ • ަॳظԽ + tanhଞΑΓ͘ऩଋ͢Δ 9
- 10. ؔ࿈ڀݚ [2] CNNͷ߹ • CNNಈతํੑΛຬͨ͢Α͏ʹॳظԽ͢Εɼਂ͍ϞσϧΛਖ਼نԽͳ͠Ͱ ֶशՄೳ • ΈࠐΈΧʔωϧͷதԝͷΈΛަॳظԽͯ͠ɼΓͯ͢0ͰॳظԽ • 1x1 convΛަॳظԽͯ͠ɼͦͷपΓΛ0ຒΊ͢Δܗ • ΈࠐΈॲཧશମΛߦྻԋࢉͱʹ͖ͱͨݟަߦྻʹͳΔ 10
- 11. ؔ࿈ڀݚ [2] CNNͷ߹ • MNISTΛ4,000ͷCNNͰֶश • ਖ਼نԽskip connectionೖΕͳ͍ • ਖ਼نͰॳظԽ͢ΔΑΓֶश͕͘ͳΔ 11
- 12. ؔ࿈ڀݚ [2] CNNͷ߹ • MNISTͱCIFAR-10Ͱ༷ʑͳਂ͞ͷϞσϧΛֶश • 10,000·Ͱ૿ֶͯ͠शͰ͖Δ • ͨͩ͠ɼCIFAR-10Ͱςετͷਫ਼͕ανΔ ਖ਼نԽskip connectionֶशͷ҆ఆԽΑΓ ൚Խʹد༩͍ͯ͠Δ͜ͱΛࣔࠦ 12
- 13. ؔ࿈ڀݚ [3] ReZero • Skip connectionΛೖΕΔ߹ͰɼಈతํੑΛຬͨ͢Α͏ʹ ॳظԽ͢Εɼ͞ΒʹੑೳΛ্͛ΒΕͦ͏ • ௨ৗ ʹ͢Δ͕ɼ ͰॳظԽͯ͠ ֶशύϥϝʔλʹ͢Δ • ॳظԽ࣌Ͱɼ ͳͷͰɼ໌Β͔ʹಈతํੑΛຬͨ͢ xi+1 = xi + αiF (xi) αi = 1 αi = 0 αi xi+1 = xi 13
- 14. ؔ࿈ڀݚ [3] ReZero • CIFAR-10Ͱ32ͷMLPΛֶश • ਖ਼نԽͳ͠Ͱ͔ͳΓֶश͕͘ͳΔ 14
- 15. ؔ࿈ڀݚ [3] ReZero • CIFAR-10ͰResNetΛֶश • ֶश͕͘ͳΓɼੑೳ্͕Δ 15
- 16. ؔ࿈ڀݚ [4] ReLUͷ߹ • ReLUͷ߹ɼަॏΈͷҰ෦Λసͤ͞ΕಈతํੑΛຬͨͤΔ • ײతʹɼReLUͰෛͷʹͳͬͨೖྗ৴߸͕ͯ͢0ʹःஅ͞ΕΔͷͰɼ ͦΕΛଧͪফ͢Α͏ʹූ߸Λసͤ͞Εྑ͍ͱ͍͏͜ͱ 16
- 17. ؔ࿈ڀݚ [5] Transformerͷrank collapse • MLP, skip connection, LayerNormͷͳ͍ attentionͷΈͷTransformerɼॳظԽͷ ࣌ͰϞσϧશମͷߦྻ͕ʹରͯ͠ ࢦతʹϥϯΫམͪ͢Δ͜ͱ͕ཧతʹ ࣔͤΔ • AttentionͷΈͰTransformerֶशͰ͖ ͳ͍͜ͱΛࣔࠦ 17
- 18. Deep Transformers without Shortcuts • TransformerͰਖ਼نԽskip connectionͳ͠ͰֶशͰ͖Δʁ ؤுΕͰ͖Δ • ७ਮʹਖ਼نԽͱskipΛൈ͘ͱޯ͕ രൃ͢Δ • ఏҊ๏͍ͩͿ͑ΒΕ͍ͯΔ
- 19. Deep Transformers without Shortcuts • ຊจͰɼGPTͰܥΘΕΔΑ͏ͳCausal masked attentionΛରʹ͢Δ • ະདྷͷྻܥΛࢀর͠ͳ͍Α͏ʹ ͰϚεΫ͢Δ Attn(X) = A(X)V(X) A(X) = softmax ( M ∘ 1 dk Q(X)K(X)⊤ − Γ(1 − M) ) Mi,j = 1i≥j ेେ͖͍ఆ Γ
- 20. Deep Transformers without Shortcuts • ·ͣɼMLPͷͳ͍attention-onlyͷϞσϧΛߟ͑Δͱɼ ͷಛྔ • ͱ͓͘ͱɼ ͕ަߦྻͷͱ͖ L XL = [ALAL−1…A1] X0W, W = L ∏ l=1 WV l WO l Σl = XlX⊤ l , Πl = AlAl−1…A1 W Σl = Πl ⋅ Σ0 ⋅ Π⊤ l
- 21. Deep Transformers without Shortcuts • ͱ͓͘ͱɼ ͕ަߦྻͷͱ͖ • ͕୯Ґߦྻʹ͚ۙΕɼޯ͕҆ఆ͢Δ ͦΕ͕͜ىΔΑ͏ʹ Λઃ͍ͨ͠ܭ • ͨͩ͠ɼ ཁૉ͕ඇෛͷԼࡾ֯ߦྻͱ͍͏੍͖ Σl = XlX⊤ l , Πl = AlAl−1…A1 W Σl = Πl ⋅ Σ0 ⋅ Π⊤ l Σl Al Al
- 22. Deep Transformers without Shortcuts • ͱ͓͘ͱɼ ͕ΓཱͭͱͰ • ͜ΕίϨεΩʔղʹ૬͢Δ ଥͳ Λઃͯ͠ܭɼͦͷίϨεΩʔղ ΛٻΊΕɼ݅Λຬͨ͢ Λ ࡞ΕΔ Al = LlL−1 l−1 L−1 0 Σ0L−1⊤ 0 = IT Σl = LlL⊤ l Σl Ll Al
- 23. Deep Transformers without Shortcuts U-SPA • ର͕֯1ͰͦΕҎ֎͕ ͷߦྻ • Λຬͨͤɼ݅Λຬͨ͢ • ϥϯΫམͪ͛Δ Σl (ρl) = (1 − ρl) IT + ρl11⊤ ρl 0 ≤ ρ0 ≤ ρ1 ≤ ⋯ ≤ ρL < 1
- 24. Deep Transformers without Shortcuts E-SPA • ର͕֯1ͰͦΕҎ֎ର֯ઢ͔ΒͷͰڑ͕ఆ·Δߦྻ • Λຬͨͤɼ݅Λຬͨ͢ • ϥϯΫམͪ͛Δ (Σl (γl))i,j = exp (−γl |i − j|) γ0 ≥ γ1 ≥ ⋯ ≥ γL > 0
- 25. Deep Transformers without Shortcuts Attentionͷ࠶ఆٛ • લड़ͷ ͔Βͨͬ࡞ͯ͠ࢉٯ Λɼ ͱղ • ਖ਼ͷର֯ߦྻɼ ֤ߦͷ͕1ͷԼࡾ֯ߦྻ • ͱ͓͍ͯɼҎԼͷΑ͏ʹattentionΛ࠶ఆٛ • ͷॏΈ Λ0ͰॳظԽ͢Δ͜ͱͰɼॳظʹ͓͍ͯ ͕ॴͷͳʹܗΔ Σ A A = DP D P B = log(P) Q(X) WQ Σ Attn(X) = DP(X)V(X), P(X) = softmax M ∘ [ 1 dk Q(X)K(X)⊤ + B ] − Γ(1 − M)
- 26. ࣮ݧ WikiText-103 • 36ͷTransformerΛֶश • ૉʹskipΛͳͨ͘͠ͷɼશֶ͘शͰ͖ͳ͍ • ఏҊ๏ɼͪΌΜͱֶशͰ͖ͯΔ • ͨͩ͠ɼskip + LNΛೖΕͨ௨ৗͷͷΑΓ ֶश͕͍ͩͿ͍
- 27. ࣮ݧ C4σʔληοτ • 32ͷTransformerΛֶश • ֶश࣌ؒΛ৳ͤɼskip + LN͋Γͷੑೳʹ౸ୡ͢Δ • 5ഒ͘Β͍͕͔͔࣌ؒΔ • Transformerʹ͓͍ͯɼskipLNֶशͷ ߴԽʹد༩͍ͯ͠Δʁ
- 28. ࣮ݧ C4σʔληοτͰͷ࣮ݧ • Skip connectionΛೖΕΔͱఏҊ๏͕ϕʔεϥΠϯͷskip + LNͷͷʹউͭ • ΓTransformerͰskip connection͕ ॏཁʁ
- 29. ·ͱΊ • MLPCNNͰɼಈతํੑΛຬͨ͢Α͏ʹॳظԽΛߦ͑ɼਖ਼نԽskip connectionͳ͠Ͱɼਂ͍ωοτϫʔΫΛֶशͰ͖Δ • TransformerͰɼಉ͡Α͏ʹॳظԽΛஸೡʹΕɼskipLNͳ͠ͰֶशͰ ͖Δ͜ͱ͕Θ͔ͬͨ • ͨͩ͠ɼֶश͕͔࣌ؒͳΓ͔͔Δ
- 30. ײ • एׯແཧΓײ൱Ίͳ͍ • ݁ظॳہԽ࣌ͷattention͕୯Ґߦྻʹۙ͘ͳΔΑ͏ʹ͢Εྑ͍ͱ͍͏͜ͱ ͷͣ • ͬͱγϯϓϧͳํ๏͋Γͦ͏ͳ͕͢ؾΔ • ֶश͕͘ͳΔݪҼ͕Ͳ͜ʹ͋Δͷ͔͕͋·ΓΘ͔͍ͬͯͳ͍
- 31. ࢀߟจݙ [1] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems 30 (2017). [2] Xiao, Lechao, et al. "Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks." International Conference on Machine Learning. PMLR, 2018. [3] Bachlechner, Thomas, et al. "Rezero is all you need: Fast convergence at large depth." Uncertainty in Artificial Intelligence. PMLR, 2021. APA 31
- 32. ࢀߟจݙ [4] Burkholz, Rebekka, and Alina Dubatovka. "Initialization of relus for dynamical isometry." Advances in Neural Information Processing Systems 32 (2019). [5] Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth." International Conference on Machine Learning. PMLR, 2021. [6] He, Bobby, et al. "Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation." The Eleventh International Conference on Learning Representations. 2023. 32