“Few-shot Image Generation using Scene Graphs” by Azade Farshad - Research Scientist at Technical University of Munich (@TU_Muenchen) Abstract: In this talk, we present "Meta Image Generation using Scene Graphs". It focuses on the few-shot generation of scenes in the wild using meta-learning and improving the quality of image generation using scene graphs. This presentation took place during a joint WiMLDS meetup between Paris & Dakar.

1. Highway
City
street Daytime Dawn/dusk Night
Few-shot Image Generation using Scene Graphs
WiMLDS Meetup, 08.02.2022
A. Farshad*, S. Musatian*, H. Dhamo, N. Navab

2. Overview
• Images generated from scene graphs normally have low quality
• Limited amount of paired scene graph and image data
2
left of
car car
truck
left of
below
car
J. Johnson, A. Gupta and L. Fei-Fei, "Image Generation from Scene Graphs" CVPR 2018.

3. Solution:
• Meta Image Generation from Scene Graphs
• Higher image generation quality using meta-learning
• Episodic training
• Episodes based on shared characteristics
Overview
3
left of
car car
truck
left of
below
car
J. Johnson, A. Gupta and L. Fei-Fei, "Image Generation from Scene Graphs" CVPR 2018.

4. Semantic scene graphs
Graph that describes a scene
● Nodes: objects
● Edges: relationships between objects
○ Action (holding, eating, riding, ...)
○ Proximity (near, left of, front of, above, ...)
○ Support (on, hanging on, ...)
○ Comparison (same as, smaller than, ...)
● Attributes: object properties
○ color, shape, material, …
man
sky
bike
T-shirt
red
old
above
wearing
riding
road
on
on
4

5. Semantic scene graphs
5
From noise to scene graph
girl horse
tree
grass
riding
behind
under
on
graph
prediction
source image
image
generation
modified image
beside
interactive graph modification
From image to scene graph to image
From image to scene graph From scene graph to image
[Xu CVPR’17]
[Johnson CVPR’18] Purely semantic nodes (object class)
[Ashual ICCV’19] Also use visual features for objects
S. Garg, H. Dhamo, A. Farshad, S. Musatian, N. Navab, and F. Tombari. Unconditional scene graph generation. ICCV 2021.
H. Dhamo, A. Farshad, I. Laina, N. Navab, G. D. Hager, F. Tombari, and C. Rupprecht. Semantic image manipulation using scene graphs. CVPR 2020.
O. Ashual and L. Wolf. Specifying object attributes and relations in interactive scene generation. ICCV 2019.
J. Johnson, A. Gupta, and Li Fei-Fei. Image generation from scene graphs. CVPR 2018.
D. Xu, Y. Zhu, C. Choi, Li Fei-Fei. Scene Graph Generation by Iterative Message Passing. CVPR 2017
[Dhamo CVPR’20]
[Garg ICCV’21]

6. Image Generation from Scene Graphs
Scene Graph GCN Bounding boxes, Masks Layout Decoder Image
6
Source: J. Johnson, A. Gupta and L. Fei-Fei, "Image Generation from Scene Graphs" CVPR 2018.

7. Meta-learning
7
Source: Ye, Han-Jia & Sheng, Xiang-Rong & Zhan, De-Chuan. (2020). Few-shot learning with
adaptively initialized task optimizer: a practical meta-learning approach. Machine Learning. 109.
10.1007/s10994-019-05838-7. Source: https://openai.com/blog/reptile/

8. Reptile
8
Source: Nichol, A., & Schulman, J. (2018). Reptile: a scalable meta-learning algorithm. arXiv preprint arXiv:1803.02999.
Φ
W1 *
W2 *

9. Method
9
Fine-tuning
Fine-tuning the meta-trained
model on test tasks
+
Few-shot image generation
Task Construction
Dividing the dataset based on
the scene attributes
● Berkeley Deep Drive
○ Attributes: Time of
day, Place
○ e.g: Day, Highway
● Action Genome
○ Video frames
● Visual Genome
○ Clustering using
SCAN [1]
[1] Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., & Van Gool, L. (2020, August). Scan: Learning to classify images without labels. In ECCV.
Meta-training
Meta-optimization on
randomly sampled tasks
using Reptile [2]
[2] Nichol, A., & Schulman, J. (2018). Reptile: a scalable meta-learning algorithm. arXiv preprint arXiv:1803.02999.

10. SG2Im
10
Input: Scene graph
Scene layout
on the side of
person
sofa
blanket
sitting on
covered by
GCN
Box
Mask
x1, y1, x2, y2
x1, y1, x2, y2
x1, y1, x2, y2
Layout
prediction
Noise
Output: Image
downsampled
layout
Conv
Conv
Upsample
Cascaded Refinement Network
Source: J. Johnson, A. Gupta and L. Fei-Fei, "Image Generation from Scene Graphs" CVPR 2018.
Embeddings

11. Method
Task 1 Task N
...
Task k
...
Training tasks
Train
Update meta
model
parameters
θ ⇽ θ + є(θk
- θ)
Pick random task k
GCN Scene layout
G
Model parameters θk
Train on task k for t steps
on the side of
person
sofa
blanket
sitting on
covered by
11
Train for each test task
G
Model parameters θ
Test task A Test task Y
...
∇
θ
Y ∇
θ
A
θA
θY
Test
SG2Im objective GAN objective Meta objective
Meta Model Parameter Update

12. Datasets
12
Visual Genome
● Diverse dataset with objects and
their semantic relationships
Krishna, R., Zhu, Y.,, ... & Fei-Fei, L. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV 2017.
https://bdd-data.berkeley.edu/
Ji, J., Krishna, R., Fei-Fei, L., & Niebles, J. C. Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs. CVPR 2020.
Berkeley Deep Drive (BDD) Action Genome (AG)
● Split into classes based on image
attributes - 23 classes
● Pre-processing to get rid of the
background and close-shot on the
objects
● Annotates the object that
person interact with (if no
interaction then they are not
annotated)
● Split based on the videos
Source: https://www.bdd100k.com/

13. Evaluation Metrics
• Frechet Inception Distance (FID)
• Estimates the distance between the Inception feature vectors for real and generated images
• Looks at the embedding layer as a continuous multivariate Gaussian and calculates
Wasserstein-2 distance between two Gaussians
• Distance is used to quantify the quality of generated samples
• Lower scores -> higher-quality images
• Kernel Inception Distance (KID)
• Does not assume any specific form of the estimated distributions
• Uses squared maximum mean discrepancy (MMD) between Inception representations as a
measure of dissimilarity between two probability distributions
• Due to unbiasedness, serves as a better estimation over FID, especially in situations where the
evaluation data is scarce
13

14. Few-shot Image Generation from Scene Graphs on BDD
Highway
City
street
Daytime Dawn/dusk Night
160-shot 10-shot 5-shot
Method Decoder FID ↓ KID*10^3 ↓ FID ↓ KID*10^3 ↓ FID ↓ KID*10^3 ↓
SG2Im CRN 194 210 176 186.5 196.8 224.2
MIGS (Ours) CRN 158.5 156.4 157 158.4 183.5 187.6
SG2Im SPADE 66.1 42.2 70.6 48.3 95.2 73.1
MIGS (Ours) SPADE 49.5 26.7 46.1 24 53.5 30.7
14

15. left of
car car
truck
left of
below
car
Ground
Truth
15
10-shot
160-shot
5-shot
SG2Im + CRN SG2Im + SPADE
MIGS (Ours) + CRN MIGS (Ours) + SPADE
Few-shot Image Generation from Scene Graphs on BDD

16. SG2Im
+
CRN
SG2Im
+
SPADE
MIGS
(Ours)
+
CRN
MIGS
(Ours)
+
SPADE
Ground
Truth
16
Method Decoder FID↓ KID *
10^3↓
SG2Im CRN 198 163.4
MIGS
(Ours)
CRN 174.5 137.8
SG2Im SPADE 141.3 76.3
MIGS
(Ours)
SPADE 98.1 47.4
● Action Genome
○ Video frames
● Qualitative &
Quantitative Results
Few-shot Image Generation from Scene Graphs on AG

17. 17
● Visual Genome
○ Clustering
● Qualitative &
Quantitative Results
160-shot 10-shot 5-shot
Method Decoder FID↓ KID*10^3↓ FID↓ KID*10^3↓ FID↓ KID*10^3↓
SG2Im SPADE 55.20 35.54 81.42 59.39 91.79 68.52
MIGS (Ours) SPADE 54.24 29.00 75.96 50.69 83.54 55.28
Few-shot Image Generation from Scene Graphs on VG

18. Conclusion
• Meta-learning enables training models with as few as 5 samples
• It can provide a better initialization for non few-shot training
• The first work for a few-shot image generation of scenes in the wild
• Achieved state-of-the-art results for scene graph to image
generation task
• Image generation from scene graphs
• Still a long way to go for high quality image generation
18

19. Questions?
19

20. Project Page
Visit Our project page for the source
code, pre-trained models, etc.
migs2021.github.io
20