言語モデルのデコーダーを用いて画像中の物体認識を効率的に行う試みについて紹介しています。

1. Paper Discussion #15
Object Recognition as Next Token Prediction (CVPR 2024)

2. © NABLAS Inc.
2
Idea
Use a pair of an image encoder and a language decoder as an (open-ended) image recognizer
which returns a list of all objects in a given image
In this case, we’ll get a sequence of tokens as output
[“so”, “fa”, “[SEP]”, “cat”, “[SEP]”, “blank”, “et”, “[SEP]”]
→ “sofa”, “cat”, “blanket” after post processing

3. © NABLAS Inc.
3
Problems that current open-ended image recognizer (e.g. CLIP) have
● Need to predefine a set of class descriptions
● As the set becomes larger, accuracy decreases
← Is it possible to eliminate this step?

4. © NABLAS Inc.
4
Straightforward way: using LLM
● With a few-shot learning, it requires good samples (& it doesn’t scale?)
● With a zero-shot learning, No explicit way to specify target classes → low accuracy

5. © NABLAS Inc.
5
CLIP image encoder + FC
※ Except the last 6 blocks, it is
frozen
First 6 blocks and the last block only
Pipeline in more details
Image Embeddings [IMG] “the objects in the image are”
Learnable

6. © NABLAS Inc.
6
Data preprocess

7. © NABLAS Inc.
7
Formulation: current image recognizer (e.g. ResNet, CLIP)
Average pooling (ResNet)
[cls] token or token pooling
Fully-connected layer (ResNet)
Set of embedding vectors of predefined class descriptions
Feature map (ResNet)
Set of token (image patch) vectors
Softmax

8. © NABLAS Inc.
8
Formulation: proposed image recognizer (in the case of each class is represented as single token)
Projection layer + LLM
Fully-connected layer (+ layer normalization)
Set of token (image patch) vectors
Softmax

9. © NABLAS Inc.
9
Formulation: proposed image recognizer (in the case of each class is represented as possibly multiple tokens)

10. © NABLAS Inc.
10
Final objective function (multiple labels with multiple tokens each)

11. © NABLAS Inc.
11
Customized non-causal attention mask
Causal attention mask
Proposed non-causal
attention mask
Query Key

12. © NABLAS Inc.
12
One-shot sampling (or parallel sampling)
This is the first token for the first label
This is also the first token for the second label
The key to its parallelism lies in the non-causal masking
mechanism, which also avoids the repetition issue (?)

13. © NABLAS Inc.
13
Experiment settings
Train dataset
(1) G3M - CC3M / COCO Captions / SBU
(2) G70M - 67M from LAION-Synthetic-115M / G3M
Eval dataset
Eval splits of CC3M / COCO Captions / OpenImages V7
Input image preprocessing
● Same to CLIP image encoder
● 224 x 224 resolution
Others
● No [cls] token in CLIP image encoder
● (32K-1) tokens (text) for output
● No [eos] token (instead of it [sep] token is used)
● We shuffle labels for each image in training (?)
● The global batch size is 512

14. © NABLAS Inc.
14
Metric
BERTScore is used
The number of objects in a given image
The number of predicted objects in a given image

15. © NABLAS Inc.
15
Recall@10 is higher while Precision@10 is lower
→ What does it mean? → It generates various classes that cover gt but some doesn’t match

16. © NABLAS Inc.
16
First 11 blocks are more important to image recognition

17. © NABLAS Inc.
17
Truncating larger LM is better than using smaller LM as it is

18. © NABLAS Inc.
18
Proposed sampling works comparable (& beam search works worse for some reason)

19. © NABLAS Inc.
19
Proposed attention masking slightly contributes to the results

20. © NABLAS Inc.
20
For larger train dataset, LLaMA 1 works better 🤔

21. © NABLAS Inc.
21
vs GPT-4V Preview (gray)

22. © NABLAS Inc.
22
They say training on CC13M (more noisy) underperforms training on CC3M

23. © NABLAS Inc.
23
Removing intermediate blocks of LLM doesn’t affect score much
It works even with single block 😯