This presentation was used in Ridge-i Yomekai event in decemver 2018 for a NIPS2018 paper named Video-to-Video Synthesis delivered by researchers from Nvidia and MIT.

1. Paper discussion:
Video-to-Video Synthesis
(NIPS 2018)
Dec,21 2018Motaz Sabri

2. Abstract
▰Through GAN coupled with Spatiotemporal adversarial
objective, its possible to create a temporally coherent 30 seconds
videos with 2K resolution from segmentation masks, poses and
sketches.
▰Variety of datasets and applications were used for evaluation.
2

3. Outline
▰The problem.
Absence of temporal incoherence.
▰Previous efforts
State of the art methods and limitations
▰Proposed model.
Experiment
▰Conclusions
3

4. The problem
▰Video synthesis models aim to generate realistic videos without
specifying scene geometry, material dynamics or lightening.
▰Most of current models focus on textural information
▰Latest proposals generate videos that are short in duration and
contain many artifacts.
4

5. Previous efforts
Video to Video Synthesis
5
Pix2PixHD COVST
PredNet

6. Proposed model
▰Sequential generator:
▰The generation depends on three elements:
▰The current frame
▰Previous source frames
▰Last generated frames (Depth =2)
6

7. Spatially and temporally progressing
7

8. Introducing foreground/background to generator
▰Such division for hallucination network allows
generators to have specialties:
▰Background regions can be generated accurately.
▰Background hallucination network needs to construct occluded part.
▰Foreground (Comes with movement) is offered strong optical flow.
8

9. Multimodal synthesis support
▰Encode ground truth into 3-dimensional feature maps.
▰Apply average pooling in order to group pixels of the same object under the same
feature vector.
▰Feed all the average pooled features and the semantic masks to the generator.
▰Given different vectors, generator F can create objects with a variety of visual
appearances.
9

10. Experiments - Technical details:
Starting with few frames with low resolution to 30 seconds length
2k videos.
10
Epochs Optimizer Learning rate Batch Machine
40 Adam 0.0002 1 Video Nvidia DGX1

11. Experiments: Samples –Synthesizing
11

12. Experiments: Samples –Synthesizing
12

13. Experiments: Samples –Edge Texturing
13

14. Experiments: Samples –Pose copy
14

15. Experiments: Samples –Prediction
15

16. Experiments: Results -CityScapes Dataset
16
Fréchet Inception
dist
13D ResNeXt Human preference
score
Short seq Long Seq
Pix2PixHD 5.57 0.18 Vid2Vid/Pix2PixHD 0.87/0.13 0.83/0.17
COVST 5.55 0.18 Vid2Vid/COVST 0.84/0.16 0.80/0.20
Vid2Vid 4.66 0.15

17. Experiments: Results –Predictions and
evaluation
17
Human preference score
Vid2Vid /No background-foreground prior 0.80/0.20
Vid2Vid / No conditional video discriminator 0.84/0.16
Vid2Vid No flow Wrapping 0.67/0.33
Fréchet Inception dist 13D ResNeXt Human preference score Video Prediction
PredNet 11.18 0.59 Vid2Vid/PredNet 0.92/0.08
MCNet 10.00 0.43 Vid2Vid/MCNet 0.98/0.02
Vid2Vid 3.44 0.18

18. Conclusion
18

19. Thanks for your attention
Any questions?
19