This document discusses controllable image-to-video translation for generating facial expression videos from a single neutral photo. It describes generating videos that smoothly transition from the neutral input photo to the peak expression state while preserving identity. The method trains models to jointly generate different expression types using adversarial and temporal continuity losses, as well as landmark prediction to maintain natural facial dynamics. Experiments analyze temporal smoothness and visualize generated expression videos.

1. Controllable image-to-video
translation: A case study on facial
expression generation

2. Introduction
■ Task Description
– how to generate video clips of rich facial expressions from a single profile photo of
the neutral expression
■ Difficulties
– The image-to-video translation might seem like an ill-posed problem because the
output has much more unknowns to fill in than the input values
– humans are familiar with and sensitive about the facial expressions
– the face identity is supposed to be preserved in the generated video clips

3. Introduction
■ Different people express emotions in similar manners
■ the expressions are often “unimodal” for a fixed type of emotion
■ the human face of a profile photo draws a majority of users’ attention, leaving the
quality of the generated background less important

4. Method
■ Problem formulation
■ Given an input image I ∈ RH×W×3 where H andW are respectively the height and
width of the image, our goal is to generate a sequence of video frames {V (a) := f(I,a);a
∈ [0,1]}, where f(I,a) denotes the model to be learned.
■ Properties:
– a=0, f(I,a)=I
– Smooth, f(I,a) and f(I,a+ Δa) should be visually similar when ∆a is small
– V (1) be the peak state of the expression

5. Method

6. Method
Training loss
adversarial loss
Temporal continuity
Facial landmark prediction Lk:

7. method
■ Jointly learning the models of different types of facial expressions

8. Experiments
Visualization

9. Experiments
■ Analysis on temporal continuity

10. Experiments