This document summarizes a survey on model compression techniques for large language models. It describes several common compression methods like pruning, knowledge distillation, low-rank factorization, and quantization. Pruning techniques include unstructured and structured pruning to remove unnecessary model parameters. Knowledge distillation transfers knowledge from a large teacher model to a smaller student model. Quantization converts floating point numbers to integers to reduce model size and computations. The document analyzes these methods and their impact on metrics like model size, compression ratio, and inference speed.

1. A Survey on Model Compression
for Large Language Models
Paper presentation
Sanjana
Kothari
CMPE 297

2. Introduction
● Advancements like GPT-4 have
pushed the boundaries of AI with
human-like language processing,
but their large size limits their
deployment and access.
● Model compression techniques are
critical to shrink these LLMs,
making them viable for low-
resource devices and reducing their
environmental footprint.
LLMs can understand and generate human-like text,
enabling them to perform a wide range of language tasks.

3. The survey by Xunyu Zhu et al.
sheds light on strategies for
model compression of Large
Language Models
Reference:
https://arxiv.org/pdf/2308.07633.pdf

4. Compression techniques

5. Pruning
● Pruning is a powerful technique to reduce the size or complexity of a model by
removing unnecessary or redundant components.
● It also makes the model storage-friendly, memory efficient and compute efficient.
● Two types:
○ Unstructured pruning
○ Structured pruning

6. Unstructured pruning
● Simplifies an LLM by removing specific parameters without considering its internal
structure.
● Targets individual weights or neurons in the LLM, usually by applying a threshold to
zero out parameters below it.
● Drawbacks:
○ Disregards the overall LLM structure, resulting in an irregular sparse model
composition, which in turn demands specialized compression techniques for
efficient storage and computation of the pruned model.
○ Often involves substantial retraining of the LLM to regain accuracy, which is
especially expensive for LLMs.
Eg. SparseGPT

7. Structured pruning
● Simplifies an LLM by removing entire structural components, such as neurons,
channels, or layers.
● Targets whole sets of weights at once, offering the advantage of reducing model
complexity and memory usage while maintaining the overall LLM structure intact.
Eg. GUM and LLM Pruner

8. Knowledge Distillation
A technique that enhances the performance of a smaller, simpler ‘student’ model by
transferring knowledge from a larger, more complex ‘teacher’ model, streamlining the
comprehensive information into a more efficient form.

9. White-box and Black-box Knowledge Distillation
● White-box distillation goes beyond traditional KD by not only using the teacher
model’s outputs but also its internal parameters and representations. This gives the
student model insight into the teacher’s reasoning and decision-making processes.
● Eg. MINILLM, GKD, TF-LLMD
● In Black-box distillation, only the predictions made by the teacher LLM are
accessible.
● These LLMs exhibit Emergent Abilities, when tackling intricate tasks
● Different facets of emergent abilities include In-Context Learning (ICL), Chain-of-
Thought (CoT) and Instruction Following (IF)

10. Types of Black-box Knowledge Distillation
● In-Context Learning (ICL) distillation is a technique where LLMs teach smaller
language models to perform new tasks using structured prompts that include task
descriptions and examples.
● Chain of Thought (CoT) distillation includes intermediate reasoning steps in the
prompts, not just input-output examples.
● Instruction Fine-tuning (IF) distillation aims to upgrade the ability of language
models to perform tasks described by instructions without requiring explicit examples. It
fine-tunes models on a variety of tasks framed as instructions, thus enabling them to
understand and execute previously unseen directives.

11. Low-Rank Factorization
● Compresses LLMs by decomposing a
weight matrix W into two smaller
matrices U and V, with W≈UV. For
example, if U is an m×k matrix and V is
a k×n matrix, and k is substantially
smaller than m and n.
● It greatly reduces the number of
parameters and computational
overhead.
Eg. LORA and TensorGPT

12. Quantization
● Reduces the storage and computational demands of deep learning models by
transforming floating-point numbers into integers or other discrete forms.
● While traditional representation employs floating point numbers, quantization converts
them to integers or other discrete forms. This transformation significantly reduces
storage requirements and computational complexity.
● Effective quantization methods can significantly compress models with minimal impact
on accuracy.
● Two types of quantization:
○ Quantization-aware training (QAT)
○ Post-training quantization (PTQ)

13. Quantization-Aware Training (QAT)
● Models are adjusted to low-precision formats during their training to better handle the
precision loss from quantization while maintaining performance.
● LLM-QAT tackles the challenge of obtaining training data by using outputs from a pre-
trained model for data-free distillation, quantizing weights, activations, and key-value
(KV) caches to as low as 4 bits, which is crucial for achieving high efficiency in large
models such as LLaMA.
● PEQA and QLORA, both types of Parameter-Efficient Fine-Tuning (PEFT), aim to
compress models and speed up inference. These ideas are aimed at conserving
memory without compromising performance.

14. Post-Training Quantization (PTQ)
● PTQ simplifies the reduction of a LLM's storage and computational demands by
quantizing its parameters after training. This method is valued for its
straightforwardness and ability to compress models efficiently without altering the
architecture or requiring retraining.
● There are two approaches to PTQ:
○ Weight Quantization: Qauntize only the weights of LLMs to enhance efficiency and
reduce computational demands. To name some, LUT-GEMM, GPTQ and AWQ
work using this technique.
○ Weight and Activation Quantization: Quantize both weights and activations of
LLMs. ZeroQuant and SmoothQuant are some of the more popular models that

15. Measuring inference efficiency of LLMs
Number of parameters: It refers to the total number of learnable weights or variables that
the LLM needs to optimise during training.
Model size: This refers to the disk space required to store the entire LLM.
Compression ratio: This is the ratio between the original size of the uncompressed LLM
and the size of the compressed LLM.
Inference time: The time taken by the LLM to process and generate responses for input
data.
Floating point operations (FLOPs): Measure of the number of arithmetic operations
involving floating-point numbers that the LLM performs when processing input data.

16. Future Direction
Performance-Size Tradeoff: Enabling the design of more efficient compression techniques
within existing hardware limits.
Dynamic LLM Compression: Reducing or eliminating the reliance on trial-and-error and
experimentation to determine the compressed size and structure of LLMs. Hence,
developing techniques like NAS (Neural Architecture Search) that reduce the dependence
on human-designed architectures.
Explainability: Adopting these transparent approaches will improve our understanding,
ease the evaluation of compressed models, and ultimately lead to more reliable AI systems.

17. Thank you