발표자: 홍성은 (SK T-Brain) 발표일: 2018.4. Real-world face recognition using a single sample per person (SSPP) is a challenging task. The problem is exacerbated if the conditions under which the gallery image and the probe set are captured are completely different. To address these issues from the perspective of domain adaptation, we introduce an SSPP domain adaptation network (SSPP-DAN). In the proposed approach, domain adaptation, feature extraction, and classification are performed jointly using a deep architecture with domain-adversarial training. However, the SSPP characteristic of one training sample per class is insufficient to train the deep architecture. To overcome this shortage, we generate synthetic images with varying poses using a 3D face model. Experimental evaluations using a realistic SSPP dataset show that deep domain adaptation and image synthesis complement each other and dramatically improve accuracy. Experiments on a benchmark dataset using the proposed approach show state-of-the-art performance.

1. SSPP-DAN: Deep Domain Adaptation Network
for Face Recognition with Single Sample Per Person
T-Brain, AI Research Center
Sungeun Hong

2. Contents
Motivation & Problem Definition
 Single sample per person (SSPP)
 Challenges in real-world face recognition
Proposed method
 Domain adaptation
 Face synthesis
Experiments
 New heterogeneous dataset
 LFW for SSPP
An overview of Domain Adaptation
2

3. Motivation & Problem Definition
3

4. SSPP face recognition
Face recognition using Single Sample Per Person (SSPP)
 Identify or verify identities using only one single gallery image
 Related to the recently attracted one-shot learning
4
Sample images of the AR database
ExpressionIllumination
Disguise Multiple
Train (on single gallery)
Test (on probe images)

5. SSPP face recognition
Limitations of most existing SSPP datasets
 Lab controlled environment
 Shooting environments (e.g., shooting location, camera) are same
5
Lu, Jiwen, Yap-Peng Tan, and Gang Wang. "Discriminative multimanifold analysis for face recognition from a single
training sample per person." IEEE transactions on pattern analysis and machine intelligence 35.1 (2013): 39-51.
Gallery
(train)
Probe
(test)

6. Real-world SSPP Face recognition
6
Registration
(Gallery)
Identification
(Probe)
Gallery
A stable image like
clear frontal mugshot
e.g., ID card or e-passport
Probe
Unstable images including
non-trivial variations
e.g., surveillance camera, web images
Variations:
camera sensor, blur,
noise, pose, illumination

7. Real-world SSPP Face recognition
Challenges
1. Heterogeneity of the shooting environments
 Gallery: stable environment
 Probe: highly unstable environment
2. Shortage of training samples
 Only one training sample per person is available
7
Registration
(Gallery)
Identification
(Probe)

8. Proposed method
8

9. Real-world SSPP Face recognition
Challenges
1. Heterogeneity of the shooting environments
 Gallery: stable environment
 Probe: highly unstable environment
9
Gap
Registration
(Gallery)
Identification
(Probe)

10. Real-world SSPP Face recognition
10
Domain
Adaptation
Registration
(Gallery)
Identification
(Probe)
Challenges
1. Heterogeneity of the shooting environments
 Gallery: stable environment
 Probe: highly unstable environment

11. Adjust a model from the source domain knowledge to a
different but related target domain
Domain Adaptation (DA)
11
With label
Source domain Target domain
?
Training Test

12. Adjust a model from the source domain knowledge to a
different but related target domain
Unsupervised DA: no label in target domain
Domain Adaptation (DA)
12
Without labelWith label
Source domain Target domain Target domain
?
Training Test

13. Domain Adaptation (DA)
Source
domain
w/ label
13

14. Domain Adaptation (DA)
Model
Source
domain
w/ label
14

15. Domain Adaptation (DA)
Model
Source
domain
w/ label
Fine
classification
15

16. Target
Domain
Domain Adaptation (DA)
Model
fine
classification
16
Worse
result
?

17. Domain Adaptation (DA)
Model
Source
domain
w/ label
Fine
classification
17

18. Adjust a model to a different target domain starting from
the source domain knowledge
Domain Adaptation (DA)
Model
Source
domain
w/ label
Target
domain
w/o label
18
Model

19. Adjust a model to a different target domain starting from
the source domain knowledge
Domain Adaptation (DA)
Model
Source
domain
w/ label
Target
domain
w/o label
19
Domain
Adaptation
Model

20. Adjust a model to a different target domain starting from
the source domain knowledge
Domain Adaptation (DA)
Model
Source
domain
w/ label
Target
domain
w/o label
Better
result
20
Fine
classification
Domain
Adaptation
Model

21. Domain Adaptation (DA)
Model
Source
domain
w/ label
Target
domain
w/o label
Better
result
21
Fine
classification
Domain
Adaptation
Model
Purpose Train Test
Domain Source Target Target
Image condition Stable Unstable Unstable
Label O X -

22. Domain Adaptation (DA)
22
 Basic assumptions of DA
 sample distribution of each domain is
related but different (o)
 samples are abundant in each domain (x)
Registration
(Gallery)
Identification
(Probe)
Domain
Adaptation

23. Face synthesis
Generate virtual samples for lack of samples.
23
Registration
(Gallery)
Identification
(Probe)

24. SSPP-DAN
24
Without
Synthesis
Without
Synthesis
t: Target domain
s: Source domain

25. SSPP-DAN
25
Without
Synthesis
Without
Synthesis
t: Target domain
s: Source domain
illumination,
noise,
blur

26. Without
Synthesis
With
Synthesis
SSPP-DAN
26
t: Target domain
s: Source domain
t: Target domain
s: Source domain

27. SSPP-DAN
27
Without
Synthesis
With
Synthesis
DA
fails
DA
succeeds

28. Real-world SSPP Face recognition
28
Registration
(Gallery)
Identification
(Probe)
Challenges
1. Heterogeneity of the shooting environments
 Gallery: stable environment
 Probe: highly unstable environment
2. Shortage of training samples
 Only one training sample per person is available

29. SSPP-DAN:
1. Domain adaptation network
 From stable face domain (source)
to unstable face domain (target)
2. Face synthesis
 Generate virtual samples
29
Domain Adaptation Network for Single Sample Per Person

30. SSPP-DAN
1. Domain adaptation network
with domain-adversarial training
 Feature learning
 Domain adaptation
 Classifier learning
30Ganin et al,"Unsupervised domain adaptation by backpropagation." ICML. 2015.
Jointly
Source
or
target?
Class
label?

31. SSPP-DAN
1. Domain adaptation network
with domain-adversarial training
 Feature learning
 Domain adaptation
 Classifier learning
31Ganin et al,"Unsupervised domain adaptation by backpropagation." ICML. 2015.
Jointly
Source
or
target?
Class
label?

32. SSPP-DAN
1. Domain adaptation network
with domain-adversarial training
 Feature learning
 Domain adaptation
 Classifier learning
32Ganin et al,"Unsupervised domain adaptation by backpropagation." ICML. 2015.
Jointly
Source
or
target?
Class
label?

33. SSPP-DAN
33
1) Landmark detection
• Supervised descent method
2) 2D → 3D mapping
3) Pose estimation
4) Image synthesis
• (yaw: -80°~+80°, pitch: -10°~40°)
1. Domain adaptation network
 From stable face domain (source)
to unstable face domain (target)
2. Face synthesis
 Generate virtual samples

34. Domain Adaptation
Data Feed
 Source (with label): frontal images + synthesized images
 Target (without label): surveillance camera images
34

35. Experiments
35

36. Heterogeneous dataset
36

37. Heterogeneous dataset
37
Source Target
2/3: train
1/3: test

38. Heterogeneous dataset
38
Lower bound
Upper bound

39. Heterogeneous dataset
39
only using
face synthesis
only using
domain adaptation

40. Heterogeneous dataset
40
Semi: 3 samples per
person from target
domain are revealed

41. Labeled Faces in the Wild (LFW)
41
Source Target
Dataset summary
 Rearranged LFW-A for SSPP face recognition
 Gallery: 50 images for 50 people
 Generic set: 108 subjects

42. Labeled Faces in the Wild (LFW)
42
Source Target
LFW for SSPP protocol
 Gallery: 50 images for 50 people
 Generic set: 108 subjects
Deep
learning

43. Summary
43
Challenges
1. Heterogeneity of the shooting environments
2. Shortage of training samples
Solution
 Deep neural network with domain adversarial training
 Facial image synthesis
Complement
each other

44. Questions?
44

45. Domain Adaptation
Baseline and variants
 Domain-invariant feature learning
 Pixel DA using GAN
Application
 Classification
 Segmentation
 Detection
 Etc.
45

46. Baseline and Variants
46

47. Domain-invariant feature learning
47

48. Residual Transfer Network
48
Long, Mingsheng, et al. "Unsupervised domain adaptation with residual transfer networks." NIPS. 2016.

49. Domain Separation Net
49
Bousmalis, Konstantinos, et al. "Domain separation networks." NIPS. 2016.

50. Adversarial Discriminative Domain Adaptation
50
Shrivastava, Ashish, et al. "Learning from simulated and unsupervised images through
adversarial training.“ CVPR 2017.

51. Pixel-level DA
51

52. Application
52

53. Classification
53
Linemod
Office dataset

54. Semantic Segmentation
54

55. Object Detection
55

56. Etc.
56

57. Application in Naver Service?
57

58. 58

59. Appendix
59

60. Domain Adaptation (DA)
Domain Adversarial Network
(Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." ICML 2015.)
60
Feature extractor Label classifier
Domain classifier

61. Domain Adaptation (DA)
Domain Adversarial Network
(Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." ICML 2015.)
61
Feature extractor Label classifier
Domain classifier