- The document proposes a novel method to compute relevancy for Transformer networks by propagating relevance through attention maps in a layer-wise manner rather than averaging across layers or relying on simplistic assumptions. - It introduces a non-parametric relevance propagation method based on Taylor decomposition to backpropagate relevance through attention, skip connections and other operations in each layer separately. - Experiments show that the proposed layer-wise relevance propagation method outperforms existing methods at identifying relevant regions in images and is more robust to input perturbations.

2. Abstract
• Existing methods either rely on the obtained attention maps, or employ
heuristic propagation along the attention graph.
• We propose a novel way to compute relevancy for Transformer networks.

3. Introduction
Problems of previous works
1. This is usually done for a single attention layer
2. Combine multiple layers : Simply averaging
-> Blurring of signals
-> Not consider the different roles of the layers (deeper layers are more semantic)
The rollout method is an alternative.
-> we show, by relying on simplistic assumptions, irrelevant tokens often become highlighted

4. Rollout
attention rollout attention flow
information
ratio
information
capacity

5. Layer-wise Relevance Propagation

6. Methods
𝐶 : number of classes
𝑥(𝑛) : input of layer 𝐿(𝑛)
(𝑥(1) ∶ network output, 𝑥(𝑁) : network input)
𝐿𝑖
𝑛
(𝑋, 𝑌) : layer operation on two tensors X and Y
Deep Taylor Decomposition formulation :

7. Methods
• GELU ...

8. Non parametric relevance propagation
Skip connection
: 𝐿 𝑛
𝑢, 𝑣 = 𝑢 + 𝑣
Matrix Multiplication
: 𝐿 𝑛 𝑢, 𝑣 = 𝑢𝑣

9. Non parametric relevance propagation
Attention
v
u
u + v
Attention
𝑅𝑣
𝑅𝑢
𝑅(𝑛−1)

10. Non parametric relevance propagation

11. Non parametric relevance propagation

12. Relevance and gradient diffusion
𝐴(𝑏): attention map of block 𝑏
∇𝐴(𝑏) : gradients
𝑅(𝑛) : relevance
𝐶 ∈ ℝ𝑠×𝑠
𝔼ℎ : is the mean across the
“heads” dimension.
⨀ : Hadamard product.

13. Experiments

14. Experiments

15. Experiments
First, a pre-trained network is used for extracting visualizations for the
validation set of ImageNet.
Second, we gradually mask out the pixels of the input image, and
measure the mean top-1 accuracy of the network.

16. Experiments

17. Conclusion
How to construct LRP + rollout