Learning Graph Embeddings for Compositional Zero-shot Learning

1. Learning Graph Embeddings
for Compositional Zero-shot
Learning
Tien-Bach-Thanh Do
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: osfa19730@catholic.ac.kr
2024/03/18
Muhammad Ferjad et al.
CVPR 2021

2. 2
Introduction

3. 3
Previous Approaches
● Learn States and Objects separately and transform the output
○ RedWine [Misra et al.]
○ SymNet [Li et al.]
● Learn a joint Compatibility function for state-object Compositions
○ LabelEmbed+ [Misra et al.]
○ TMN [Purushwalkam et al.]
● However, these works treat each state-object composition independently, ignoring the rich dependency
structure of different states, objects and their compositions
● Example: learning the composition old dog is not only dependent on the state old and object dog, but also
can be supported by other compositions like cute dog, old car,...
Dependency structure provides a strong regularization which allows the network to better generalize
to novel compositions

4. 4
Defining a Compositional function
What is composition?
● Compositionality: ability to
decompose an observation into its
primitives
● Composition = Observed objects +
Their states
● Ex: Old dog = object dog + state old
Model to jointly by learning a compatibility function that
captures the compatibility score between an image, a
state and an object
image
feature
extracted
output the label
embedding of the
state-object pair

5. 5
Defining a Compositional Graph

6. 6
Defining a Compositional Graph
Node feature
● Remember, they are still feature
extracted
● They argued that there exists a
complex dependency structure
between states, objects and their
compositions and learning this
dependency structure is crucial
● Represent each node with a proper
feature embedding
● Use the word embeddings pretrained
on a large text corpus like Wikipedia,
because they capture rich semantic
similarities among words
● Ex: dog is closer to cat than to car in
the word embedding space

7. 7
Compositional Graph Embedding (CGE)

8. 8
How graph created?
• G(s, o) that learns the label embedding for each compositional label y = (s,o) (GCE)
• Compositional Graph K = |S| + |O| + |Y| nodes that represent states S, objects O and compositional labels
Y
• Two nodes are connected if they are related
• Each compositional label y = (s, o) defines a dependency relationship between the state s, object o and
their composition y
• Edges built by connecting (s, o), (s, y) and (o, y) for every y = (s, o) belong Y
O:DOG S:CUTE
Y: CUTE DOG

9. 9
Defining a Compositional Graph
Compositional Graph

10. 10
Defining a Compositional Graph
Objective
● Objective: predict the
classifier weights of
the compositional
labels, the node
embedding of the
output layer in the
GCN has the same
dimensionality
● Optimize cross-
entropy
● Inference: at test time,
given an input image
x, predict by searching
the compositional
label that yields the
highest compatibility

11. 11
Compositional Graph Embedding (CGE)

12. 12
Datasets
Compositional GQA (C-GQA)
• A new dataset curated from GQA
○ 870 objects, 453 states
○ Training - 26k images, 6963 compositions
○ Validation - 7k images, 1173 seen compositions, 1368 unseen compositions
○ Test pairs - 5k images, 1022 seen, 1047 unseen compositions

13. 13
Datasets
Results

14. 14
Datasets
Results

15. 15
Datasets
Qualitatives

16. 16
Conclusion
• Proposed a novel graph formulation for Compositional Zero-shot learning in the challenging generalized
zero-shot setting
• By propagating knowledge in the graph against training images of seen compositions, CGE learn
classifiers for all compositions end-to-end