The document discusses recent advances in generative adversarial networks (GANs) for image generation. It summarizes two influential GAN models: ProgressiveGAN (Karras et al., 2018) and BigGAN (Brock et al., 2019). ProgressiveGAN introduced progressive growing of GANs to produce high resolution images. BigGAN scaled up GAN training through techniques like large batch sizes and regularization methods to generate high fidelity natural images. The document also discusses using GANs to generate full-body, high-resolution anime characters and adding motion through structure-conditional GANs.

1. #denatechcon
#denatechcon

2. #denatechcon

3. #denatechcon

4. #denatechcon
00./ 0

5. #denatechcon
00./ 0

6. #denatechcon

7. #denatechcon

8. #denatechcon
// . /

9. #denatechcon
// . /

10. #denatechcon

11. #denatechcon

12. #denatechcon

13. #denatechcon

14. #denatechcon

15. #denatechcon
AGENDA

16. #denatechcon
(@hamadakoichi)
Mobage
2010 6
Mobage2011 4
DeNA
DeNA2014 10
: ( )
TokyoWebmining
- 1500
- 2010 60
DeNA 8 AI AI
AI

17. #denatechcon
50 /

18. #denatechcon

19. #denatechcon , , , , , .
“ ”. 232 (7/20/2017).
0
3

20. #denatechcon
5 4 /0 /
5 4 /0 : 6 / .

21. #denatechcon
Full-body High-resolution Anime Generation with
Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida.
In ECCV Workshop 2018.
(ECCV: European Conference on Computer Vision)

22. #denatechcon
AGENDA

23. #denatechcon
AGENDA

24. #denatechcon

25. #denatechcon
1 3 5 7
2 4 6 8

26. #denatechcon
1 3 5 7
2 4 6 8

27. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
1 3 5 7
2 4 6 8

28. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

29. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

30. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

31. #denatechcon

32. #denatechcon
Generative Adversarial Nets.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-
Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio.
arXiv:1406.2661. In NIPS 2014.

33. #denatechcon

34. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

35. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

36. #denatechcon
Progressive Growing of GANs for Improved Quality, Stability, and Variation.
Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen. In ICLR 2018.
(1024X1024)
(256x256)

37. #denatechcon
.441 7 545 7 4 /
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen. In ICLR 2018.

38. #denatechcon
/ 5. 44 5
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen. In ICLR 2018.

39. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

40. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

41. #denatechcon
+ Spectral Normalization on Generator
+ Self Attention
+ Two Time Scale Update Rule
(512x512)
+ Spectral Normalization on Discriminator
+ Projection Discriminator
SNGAN with Projection (Miyato+, ICLR’18)
SAGAN (Zhang+, 18)
BigGAN (Brock+, ICLR’19)
+ Large Batch Size (256→2048)
+ Large Channel (64→96)
+ Shared Embedding
+ Hierarchical Latent Space
+ Truncation Trick
+ Orthogonal Regularization
+ First Singular Value Clamp
+ Zero-centered Gradient Penalty
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2018.

42. #denatechcon
(512x512)
Generator
Typical Architecture
Res Block
Architecture for ImageNet at 512x512
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.

43. #denatechcon
(512x512)
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.

44. #denatechcon
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
(512x512)

45. #denatechcon
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
(512x512)

46. #denatechcon
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
(512x512)

47. #denatechcon
(512x512)
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.

48. #denatechcon
(512x512)
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.

49. #denatechcon
Large Scale GAN Training for High Fidelity Natural Image Synthesis.
Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
(512x512)

50. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

51. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

52. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

53. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

54. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

55. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

56. #denatechcon

57. #denatechcon

58. #denatechcon

59. #denatechcon

60. #denatechcon

61. #denatechcon

62. #denatechcon

63. #denatechcon

64. #denatechcon

65. #denatechcon

66. #denatechcon
AGENDA

67. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

68. #denatechcon

69. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

70. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

71. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
0
0
0
0
0
0 0
0
0
0
0

72. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

73. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

74. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
0
0
0
0
0
0 0
0
0
0
0

75. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

76. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

77. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

78. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

79. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

80. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0
Full-body anime generation at 1024x1024 with Progressive Structure-conditional GANs

81. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
00./ 0

82. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
// . 0/0
Adding action to full-body anime characters with Progressive Structure-conditional GANs

83. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

84. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
(ICLR’18)

85. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
(ICLR’18)

86. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
(ICLR’18)
(NIPS’17) (NIPS’17)

87. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
(ICLR’18)
(NIPS’17) (NIPS’17)

88. #denatechcon
Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.

89. #denatechcon

90. #denatechcon

91. #denatechcon
AGENDA

92. #denatechcon
n e
n 4 3 33 36 4
n 0231 33 36 4
n c
b a
)3 (
0 06I J@ N LT A F
( D T

93. #denatechcon
n
n
n

94. #denatechcon
n
n

95. #denatechcon
n C
n 0 1 .
n 0:: /1:0 6 ..:. 21 )()

96. #denatechcon
n CEAD A
n ) 17:
n 7 2 -( ) 17: 0/ 1 7 0 7 . :0. 0 7 1 7 :

97. #denatechcon
AGENDA

98. #denatechcon

99. #denatechcon
: ( . ) / .

100. #denatechcon
•
•
•
•
: ( . ) / .

101. #denatechcon

102. #denatechcon
O .10 2:: ( / /:7 10
* .) A L A

103. #denatechcon
O .10 2:: ( / /:7 10
* .) A L A

104. #denatechcon
•
O .10 2:: ( / /:7 10
* .) A L A

105. #denatechcon
•
•
O .10 2:: ( / /:7 10
* .) A L A

106. #denatechcon
O .10 2:: ( / /:7 10
* .) A L A
•
•

107. #denatechcon

108. #denatechcon
) ) 3

109. #denatechcon
) ) 3

110. #denatechcon

111. #denatechcon

112. #denatechcon

113. #denatechcon

114. #denatechcon
8 1 1
1
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz. In CVPR 2018.

115. #denatechcon
/30 480 6 2/81 4C
+ 60 2
• 8 ,
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz. In CVPR 2018.
https://youtu.be/MjViy6kyiqs
Research at NVIDIA: Transforming Standard Video Into Slow Motion with AI

116. #denatechcon
7
1
Video Frame Synthesis using Deep Voxel Flow.
Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. In ICCV 2017.

117. #denatechcon
N I 7 B7 =: B = P
77 = 7: :=D
• /0 (+ /0 ,
Video Frame Synthesis using Deep Voxel Flow.
Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. In ICCV 2017.
BB F=CBC 67 ?. / 3: 1 B
Video Frame Synthesis using Deep Voxel Flow

118. #denatechcon
D
F 6 + 23C
• 1 76 , P SV P J IOM S R
• ( ,24 c SV P J ,24 cP
/ ++ C
• 1 76 , P J SV P
• 4 8 4 0 L a
Super SloMo(Adobe)
Super SloMo
Deep Voxel Flow
Video Frame Synthesis using Deep Voxel Flow. Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. In ICCV 2017.
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. Huaizu Jiang, Deqing Sun, Varun
Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz. In CVPR 2018.
F

119. #denatechcon
2

120. #denatechcon

121. #denatechcon
F
8 0 O T PL
n S 4 )
n F 4 (
W4000301 32 ( 6 A 7

122. #denatechcon
F F F 0 -1 12 W
n
n S
L ***7* 7 A A
OT 0 -1 12

123. #denatechcon

124. #denatechcon

125. #denatechcon

126. #denatechcon
n
n

127. #denatechcon
n
n
n

128. #denatechcon
n
n
n
n F

129. #denatechcon
n
n
n
n F

130. #denatechcon
G
I0 I4
Optical Flow
I1, I2, I3
(Generated)

131. #denatechcon
G
I0 I4
D
Optical Flow
I1, I2, I3
(Generated)
Ground Truth

132. #denatechcon
G
I0 I4
D
Optical Flow
I1, I2, I3
(Generated)
Ground Truth
MSE

133. #denatechcon
G
I0 I4
D
Optical Flow
Ground Truth
I1, I2, I3
(Generated)
Ground Truth
Generated
MSE
MSE

134. #denatechcon
G
I0 I4
D
Optical Flow
Ground Truth
I1, I2, I3
(Generated)
Ground Truth
Generated
MSE
MSE

135. #denatechcon
2

136. #denatechcon
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
Local Discriminator
“Real” or “Fake”
Local Patch
(16×16pix)

137. #denatechcon
Conv-BN-ReLU
Conv-BN-ReLU
Generated Image
Sequense
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
Conv-BN-ReLU
FC
Local Discriminator
Temporal Discriminator
“Real” or “Fake”
Local Patch
(16×16pix)
Image
Sequense
“Real” or “Fake”

138. #denatechcon

139. #denatechcon
Video

140. #denatechcon
image0 image1 image2 image3 image4Video

141. #denatechcon
⁃ step size = 4 7FPS -> 30FPS 001.png, 005.png, 009.png, 013.png, 017.png
⁃ step size = 1 30FPS -> 120FPS 001.png, 002.png, 003.png, 004.png, 005.png

142. #denatechcon
⁃ step size = 4 7FPS -> 30FPS 001.png, 005.png, 009.png, 013.png, 017.png
⁃ step size = 1 30FPS -> 120FPS 001.png, 002.png, 003.png, 004.png, 005.png

143. #denatechcon

144. #denatechcon
Frame
Frame
Deep Voxel FlowInput
// . /
Experimental Results: “Anime Frame Generation with Structure-consistent Prediction GANs”

145. #denatechcon
step size = 1 step size = 4 step size = 7 step size = 10
Input
SPGAN
(Ours)
Deep Voxel Flow
4
// . /
Experimental Results: “Anime Frame Generation with Structure-consistent Prediction GANs”

146. #denatechcon
Deep Voxel Flow Ours
1.average PSNR/SSIM on test dataset step size=4
PSNR SSIM
Deep Voxel Flow 23.32 0.9294
SPGAN(Ours) 24.27 0.9407

147. #denatechcon
AGENDA

148. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

149. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

150. #denatechcon
ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)

151. #denatechcon
00./ 0

152. #denatechcon
00./ 0

153. #denatechcon
// . /

154. #denatechcon
// . /

155. #denatechcon

156. #denatechcon * 10 2 1

157. #denatechcon

158. #denatechcon
L0*** * 2 D 1 A 7
1 D 1 7

159. #denatechcon
TZ ... ./ 0 KD SL KA N O
N KD SL K N W
(3 Frame) AI (x16 )
Input Frames Generated Frames
78 : : 102*0DeNA AI :

160. #denatechcon