This document describes MEGABYTE, a model for predicting million-byte sequences using multiscale transformers. MEGABYTE uses a patch embedder to chunk input sequences into fixed-length patches, a global transformer to contextualize patches, and a local transformer to autoregressively predict the next patch. The local transformer is needed to increase parameters for fixed compute and reuse established transformer components. MEGABYTE achieves state-of-the-art performance on language modeling tasks and can process sequences over 1 million tokens efficiently by splitting computation between global and local transformers.

1. MEGABYTE: Predicting Million-byte Sequences
with Multiscale Transformers
2023.6.7
유용상
NLP 티타임

2. Introduction

3. Introduction

4. Introduction
- Patch embedder
: It takes an input of a discrete sequence,
embeds each element, and chunks it into
patches of a fixed length.
- Global Module
: It is a large autoregressive transformer that
contextualizes patch representations
by performing self-attention over previous patches
- Local Module
: It is a small local transformer that inputs
a contextualized patch representation from
the global model, and autoregressively
predicts the next patch

5. Components
1. Patch Embedder
Byte sequence x0~xT
Patch size : P
Dimension size :
Patch length :
<- byte embedding
reshape
<- input to the global model

6. Components
2. Global model

7. Components
3. Local model

8. Motivation
- Why is local model needed?

9. Motivation
- Increasing Parameters for Fixed Compute
: Larger feedforward layers across patches rather than individual tokens
- Re-use of Established Components
: increases the likelihood that the architecture will inherit the desirable scaling properties of transformers.

10. Efficiency Analysis
Training Efficiency
Sequence length : T
Vanila transformer
Sparse transformer
Routing transformer
Sequence length : T
Patch size : P
Patch Length : T/P
Global model
Local model
MEGABYTE(Overall)

11. Efficiency Analysis
Feedforward Layers
in the GPT3 architecture, the quadratic self-attention computation accounts for only 1.4% of FLOPS
대부분은 feedforward에서 FLOPS를 잡아먹음..
# of non-embedding parameters : m
Sequence length : T
MEGABYTE
# of global model parameters
# of local model parameters
2mT FLOPS

12. Efficiency Analysis
Combined Analysis

13. Performance
Language Modeling

14. Performance
Generation Speed

15. Performance
Effective Use of Context

16. Conclusion
- SOTA in existing byte-level models across a range of tasks and modalities
- Allowing large models of sequences of over 1M tokens
- gives competitive language modeling results with subword models,
which may allow byte-level models to replace tokenization
- Scale was far below those of SOTA LLMs