The Longformer model introduces an efficient attention mechanism that allows transformer models to process longer sequences than previous approaches. Longformer uses sliding window attention and dilated attention to attend to a subset of context positions while still enabling attention to positions far from the query. Experiments show Longformer outperforms RoBERTa on character-level language modeling tasks and achieves better results on downstream question answering tasks when finetuned from a Longformer pretraining compared to a RoBERTa pretraining.

1. Longformer
Allen AI

2. Outline
• Intro
• Longformer’s attention
• Intuition
• Structure
• Questions & Concerns
• Experiments
• Results & Ablation
• For Pretraining
• Discussion

3. Intro
• Limitation of document length (tf-based model)
• Bottleneck: Self-Attention layers are expensive
• Time, Memory
• Contributions:
• Cheaper for long document tasks(e.g. QA)
• Contextual representation of entire context
• Exisiting approach:
• Divided(Chunk) from long into pieces -
• Truncated: Information loss - e.g. BERT 512
• Two-step model - e.g. Pool >> candidates >> answer, e.g. Multihop
3

4. Longformer’s Attention
• Capture far context(even full sequece) efficiently
• Avoid full attention…(Sparse but still cost)
• Types: Windowed attention / Dilated attention
• Evaluate ability(v.s. RoBERTa)
• Countinually trained on RoBERTa’s checkpoint(?), and
apply on donwstream tasks

5. • Sliding Window:
• Fixed window size W(say 2, means attend last&next only)
• Stacked with multiple W for each layers(small W: local info)
• Dilated Window:
• Add dilated size D with sliding window
• Larger acceptive fields(longer input/can attend far away from D)
• Global + Sliding Window:
• Customized attention(pre-select, depends on tasks) e.g. MLM
• Two set of QKV projections(One for global, One for sliding window)

6. • Q1: Contextual informations loss: (sort of distortion)
• Increasing W from bottom to top.
Low for local, while high for entire
• Use dilated sliding window only at higher layer
• Q2: Must be Stacked layers to expand recpetive fields.
• Combine(share) the each layer’s attn
• Local first / stacked / Increasing W&D / More Elastic
Questions

7. Experiments
• Task: Character-level Autoregressive LM (for longer sequence) on
text8 & enwik8
• Boundary: Longest sequence with limited memory
• Training:
• Staged training: 5 phase
• Length from 2048 to 23040
• Evaluation:
• Max length to 32256…(no optim/grads?)

8. Result & Ablation
• BPC: Bit per character, smaller the better
• Same #param, better performance
• Window size should be increasing
• Add some dilation may improved

9. Pretraining
• Pretrained Longformer and then finetuning on 6 tasks
• Countinue from RoBERTA checkpoint(??)
* longformer attn can be load into any tfmodel
• Pretrained MLM…
• Same tokenizer: Wordpiece
• W=512/D=0 for all layer
• L(longformer)=4096…while L(roberta)=512
• How to utilize pos-emb??
• Finetuning on QA tasks
• Better than RoBERTa, but

10. Results & Ablation
• All better models use
• multistage, GNNs…
• Confirmed that performance
NO improved due to additional
pretraining
•

11. Discussion
• For long documnets tasks:
• QAs(multihop, OpenDomainQA)
• Long document generation e.g. summarization
• Other tasks…
• For Explainable
• Task specific
• Contextual Embeddings
• Multihead