Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AIによるアニメ生成の挑戦

26,549 views

Published on

「AIによるアニメ生成の挑戦」。DeNAの「構造的生成学習」技術開発により、従来AI生成で難しかった “複雑な構造での高品質生成”、"構造変化が大きい画像間の補完生成・中割” 等の課題を解消し、多様なキャラクター全体でのアニメ生成・中割生成、を実現しています。DeNA TechCon 2019での DeNA "アニメ生成プロジェクト" の活動紹介、発表資料です。

[AIによる生成アニメ例]
https://www.youtube.com/watch?v=tOZW_KWb8b0

---
"AIによるアニメ生成の挑戦".
濱田晃一 , 李天琦.
DeNA TechCon 2019.
---
"Challenges toward Anime Generation with Deep Generative Models".
Koichi Hamada and Tianqi Li.
In DeNA Technology Conference 2019.

Published in: Engineering

AIによるアニメ生成の挑戦

  1. 1. #denatechcon #denatechcon
  2. 2. #denatechcon
  3. 3. #denatechcon
  4. 4. #denatechcon 00./ 0
  5. 5. #denatechcon 00./ 0
  6. 6. #denatechcon
  7. 7. #denatechcon
  8. 8. #denatechcon // . /
  9. 9. #denatechcon // . /
  10. 10. #denatechcon
  11. 11. #denatechcon
  12. 12. #denatechcon
  13. 13. #denatechcon
  14. 14. #denatechcon
  15. 15. #denatechcon AGENDA
  16. 16. #denatechcon (@hamadakoichi) Mobage 2010 6 Mobage2011 4 DeNA DeNA2014 10 : ( ) TokyoWebmining - 1500 - 2010 60 DeNA 8 AI AI AI
  17. 17. #denatechcon 50 /
  18. 18. #denatechcon
  19. 19. #denatechcon , , , , , . “ ”. 232 (7/20/2017). 0 3
  20. 20. #denatechcon 5 4 /0 / 5 4 /0 : 6 / .
  21. 21. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCV Workshop 2018. (ECCV: European Conference on Computer Vision)
  22. 22. #denatechcon AGENDA
  23. 23. #denatechcon AGENDA
  24. 24. #denatechcon
  25. 25. #denatechcon 1 3 5 7 2 4 6 8
  26. 26. #denatechcon 1 3 5 7 2 4 6 8
  27. 27. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019) 1 3 5 7 2 4 6 8
  28. 28. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  29. 29. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  30. 30. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  31. 31. #denatechcon
  32. 32. #denatechcon Generative Adversarial Nets. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio. arXiv:1406.2661. In NIPS 2014.
  33. 33. #denatechcon
  34. 34. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  35. 35. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  36. 36. #denatechcon Progressive Growing of GANs for Improved Quality, Stability, and Variation. Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen. In ICLR 2018. (1024X1024) (256x256)
  37. 37. #denatechcon .441 7 545 7 4 / Progressive Growing of GANs for Improved Quality, Stability, and Variation Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen. In ICLR 2018.
  38. 38. #denatechcon / 5. 44 5 Progressive Growing of GANs for Improved Quality, Stability, and Variation Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen. In ICLR 2018.
  39. 39. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  40. 40. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  41. 41. #denatechcon + Spectral Normalization on Generator + Self Attention + Two Time Scale Update Rule (512x512) + Spectral Normalization on Discriminator + Projection Discriminator SNGAN with Projection (Miyato+, ICLR’18) SAGAN (Zhang+, 18) BigGAN (Brock+, ICLR’19) + Large Batch Size (256→2048) + Large Channel (64→96) + Shared Embedding + Hierarchical Latent Space + Truncation Trick + Orthogonal Regularization + First Singular Value Clamp + Zero-centered Gradient Penalty Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2018.
  42. 42. #denatechcon (512x512) Generator Typical Architecture Res Block Architecture for ImageNet at 512x512 Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
  43. 43. #denatechcon (512x512) Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
  44. 44. #denatechcon Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019. (512x512)
  45. 45. #denatechcon Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019. (512x512)
  46. 46. #denatechcon Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019. (512x512)
  47. 47. #denatechcon (512x512) Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
  48. 48. #denatechcon (512x512) Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019.
  49. 49. #denatechcon Large Scale GAN Training for High Fidelity Natural Image Synthesis. Andrew Brock, Jeff Donahue, Karen Simonyan. arXiv:1809.11096. In ICLR 2019. (512x512)
  50. 50. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  51. 51. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  52. 52. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  53. 53. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  54. 54. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  55. 55. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  56. 56. #denatechcon
  57. 57. #denatechcon
  58. 58. #denatechcon
  59. 59. #denatechcon
  60. 60. #denatechcon
  61. 61. #denatechcon
  62. 62. #denatechcon
  63. 63. #denatechcon
  64. 64. #denatechcon
  65. 65. #denatechcon
  66. 66. #denatechcon AGENDA
  67. 67. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  68. 68. #denatechcon
  69. 69. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  70. 70. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  71. 71. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 0 0 0 0 0 0 0 0 0 0 0
  72. 72. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  73. 73. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  74. 74. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 0 0 0 0 0 0 0 0 0 0 0
  75. 75. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  76. 76. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  77. 77. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  78. 78. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  79. 79. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  80. 80. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0 Full-body anime generation at 1024x1024 with Progressive Structure-conditional GANs
  81. 81. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. 00./ 0
  82. 82. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. // . 0/0 Adding action to full-body anime characters with Progressive Structure-conditional GANs
  83. 83. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  84. 84. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. (ICLR’18)
  85. 85. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. (ICLR’18)
  86. 86. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. (ICLR’18) (NIPS’17) (NIPS’17)
  87. 87. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018. (ICLR’18) (NIPS’17) (NIPS’17)
  88. 88. #denatechcon Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. In ECCVW 2018.
  89. 89. #denatechcon
  90. 90. #denatechcon
  91. 91. #denatechcon AGENDA
  92. 92. #denatechcon n e n 4 3 33 36 4 n 0231 33 36 4 n c b a )3 ( 0 06I J@ N LT A F ( D T
  93. 93. #denatechcon n n n
  94. 94. #denatechcon n n
  95. 95. #denatechcon n C n 0 1 . n 0:: /1:0 6 ..:. 21 )()
  96. 96. #denatechcon n CEAD A n ) 17: n 7 2 -( ) 17: 0/ 1 7 0 7 . :0. 0 7 1 7 :
  97. 97. #denatechcon AGENDA
  98. 98. #denatechcon
  99. 99. #denatechcon : ( . ) / .
  100. 100. #denatechcon • • • • : ( . ) / .
  101. 101. #denatechcon
  102. 102. #denatechcon O .10 2:: ( / /:7 10 * .) A L A
  103. 103. #denatechcon O .10 2:: ( / /:7 10 * .) A L A
  104. 104. #denatechcon • O .10 2:: ( / /:7 10 * .) A L A
  105. 105. #denatechcon • • O .10 2:: ( / /:7 10 * .) A L A
  106. 106. #denatechcon O .10 2:: ( / /:7 10 * .) A L A • •
  107. 107. #denatechcon
  108. 108. #denatechcon ) ) 3
  109. 109. #denatechcon ) ) 3
  110. 110. #denatechcon
  111. 111. #denatechcon
  112. 112. #denatechcon
  113. 113. #denatechcon
  114. 114. #denatechcon 8 1 1 1 Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz. In CVPR 2018.
  115. 115. #denatechcon /30 480 6 2/81 4C + 60 2 • 8 , Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz. In CVPR 2018. https://youtu.be/MjViy6kyiqs Research at NVIDIA: Transforming Standard Video Into Slow Motion with AI
  116. 116. #denatechcon 7 1 Video Frame Synthesis using Deep Voxel Flow. Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. In ICCV 2017.
  117. 117. #denatechcon N I 7 B7 =: B = P 77 = 7: :=D • /0 (+ /0 , Video Frame Synthesis using Deep Voxel Flow. Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. In ICCV 2017. BB F=CBC 67 ?. / 3: 1 B Video Frame Synthesis using Deep Voxel Flow
  118. 118. #denatechcon D F 6 + 23C • 1 76 , P SV P J IOM S R • ( ,24 c SV P J ,24 cP / ++ C • 1 76 , P J SV P • 4 8 4 0 L a Super SloMo(Adobe) Super SloMo Deep Voxel Flow Video Frame Synthesis using Deep Voxel Flow. Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala. In ICCV 2017. Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, Jan Kautz. In CVPR 2018. F
  119. 119. #denatechcon 2
  120. 120. #denatechcon
  121. 121. #denatechcon F 8 0 O T PL n S 4 ) n F 4 ( W4000301 32 ( 6 A 7
  122. 122. #denatechcon F F F 0 -1 12 W n n S L ***7* 7 A A OT 0 -1 12
  123. 123. #denatechcon
  124. 124. #denatechcon
  125. 125. #denatechcon
  126. 126. #denatechcon n n
  127. 127. #denatechcon n n n
  128. 128. #denatechcon n n n n F
  129. 129. #denatechcon n n n n F
  130. 130. #denatechcon G I0 I4 Optical Flow I1, I2, I3 (Generated)
  131. 131. #denatechcon G I0 I4 D Optical Flow I1, I2, I3 (Generated) Ground Truth
  132. 132. #denatechcon G I0 I4 D Optical Flow I1, I2, I3 (Generated) Ground Truth MSE
  133. 133. #denatechcon G I0 I4 D Optical Flow Ground Truth I1, I2, I3 (Generated) Ground Truth Generated MSE MSE
  134. 134. #denatechcon G I0 I4 D Optical Flow Ground Truth I1, I2, I3 (Generated) Ground Truth Generated MSE MSE
  135. 135. #denatechcon 2
  136. 136. #denatechcon Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU Local Discriminator “Real” or “Fake” Local Patch (16×16pix)
  137. 137. #denatechcon Conv-BN-ReLU Conv-BN-ReLU Generated Image Sequense Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU Conv-BN-ReLU FC Local Discriminator Temporal Discriminator “Real” or “Fake” Local Patch (16×16pix) Image Sequense “Real” or “Fake”
  138. 138. #denatechcon
  139. 139. #denatechcon Video
  140. 140. #denatechcon image0 image1 image2 image3 image4Video
  141. 141. #denatechcon ⁃ step size = 4 7FPS -> 30FPS 001.png, 005.png, 009.png, 013.png, 017.png ⁃ step size = 1 30FPS -> 120FPS 001.png, 002.png, 003.png, 004.png, 005.png
  142. 142. #denatechcon ⁃ step size = 4 7FPS -> 30FPS 001.png, 005.png, 009.png, 013.png, 017.png ⁃ step size = 1 30FPS -> 120FPS 001.png, 002.png, 003.png, 004.png, 005.png
  143. 143. #denatechcon
  144. 144. #denatechcon Frame Frame Deep Voxel FlowInput // . / Experimental Results: “Anime Frame Generation with Structure-consistent Prediction GANs”
  145. 145. #denatechcon step size = 1 step size = 4 step size = 7 step size = 10 Input SPGAN (Ours) Deep Voxel Flow 4 // . / Experimental Results: “Anime Frame Generation with Structure-consistent Prediction GANs”
  146. 146. #denatechcon Deep Voxel Flow Ours 1.average PSNR/SSIM on test dataset step size=4 PSNR SSIM Deep Voxel Flow 23.32 0.9294 SPGAN(Ours) 24.27 0.9407
  147. 147. #denatechcon AGENDA
  148. 148. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  149. 149. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  150. 150. #denatechcon ProgressiveGAN (Karras et al., ICLR 2018) BigGAN (Brock et al., ICLR 2019)
  151. 151. #denatechcon 00./ 0
  152. 152. #denatechcon 00./ 0
  153. 153. #denatechcon // . /
  154. 154. #denatechcon // . /
  155. 155. #denatechcon
  156. 156. #denatechcon * 10 2 1
  157. 157. #denatechcon
  158. 158. #denatechcon L0*** * 2 D 1 A 7 1 D 1 7
  159. 159. #denatechcon TZ ... ./ 0 KD SL KA N O N KD SL K N W (3 Frame) AI (x16 ) Input Frames Generated Frames 78 : : 102*0DeNA AI :
  160. 160. #denatechcon

×