Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[NDC2017] 딥러닝으로 게임 콘텐츠 제작하기 - VAE를 이용한 콘텐츠 생성 기법 연구 사례

11,420 views

Published on

2017년 4월 26일, NDC2017 발표자료입니다.

콘텐츠 제작은 게임 개발에서 많은 노력과 시간 투자를 필요로하는 작업입니다. 최근 폭발적인 관심을 받고 있는 딥러닝을 통해 여기에 드는 시간을 크게 줄일 수 있습니다. 이 발표에서는 VAE(Variational AutoEncoder)를 이용한 모방을 통한 콘텐츠 생성 기법에 대해서 다룹니다.

Published in: Engineering
  • Sex in your area is here: www.bit.ly/sexinarea
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: www.bit.ly/2AJerkH
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[NDC2017] 딥러닝으로 게임 콘텐츠 제작하기 - VAE를 이용한 콘텐츠 생성 기법 연구 사례

  1. 1. 넥슨 신규개발본부 Blast팀 김환희 VAE를 이용한 콘텐츠 생성 기법 연구 사례 딥러닝으로 게임 콘텐츠 제작하기
  2. 2. 발표자 소개 6년차 게임기획자 NDC2014 <기획자가 유니티를 만났을 때 : 자체 툴 제작을 해보자> NDC2016 <신경망은 컨텐츠 자동생성의 꿈을 꾸는가 : 딥러닝을 이용한 SRPG 맵 평가 사례 연구>
  3. 3. 안내 이 발표는 개인 연구 프로젝트입니다.
  4. 4. 목차 1. 계기 2. Autoencoder 3. Variational Autoencoder 4. 생성 사례
  5. 5. 1. 계기 - 왜 이 주제를 선택했는가
  6. 6. 작년 발표 자료 https://www.slideshare.net/HwanheeKim2/ndc2016-61418391
  7. 7. 기술의 빠른 발전 http://fvae.ail.tokyo/ • <FACIAL VAE: Conditional VAE-GAN with Attribute Inference for Faces>
  8. 8. FaceApp https://itunes.apple.com/us/app/faceapp-free-neural-face-transformations/id1180884341?mt=8 https://play.google.com/store/apps/details?id=io.faceapp&hl=ko
  9. 9. 기술의 빠른 발전 • <Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks> https://www.semanticscholar.org/paper/Unsupervised-Representation-Learning-with-Deep-Radford- Metz/35756f711a97166df11202ebe46820a36704ae77
  10. 10. 기술의 빠른 발전 • <BEGAN: Boundary Equilibrium Generative Adversarial Networks> https://github.com/Heumi/BEGAN-tensorflow
  11. 11. 기술의 빠른 발전 • <BEGAN: Boundary Equilibrium Generative Adversarial Networks> https://github.com/Heumi/BEGAN-tensorflow
  12. 12. 개인적인 필요 • 컨텐츠 제작 시간, 능력 부족
  13. 13. 딥러닝을 이용한 생성 모델 • VAE (Variational Auto-Encoder) • GAN (Generative Adversarial Networks) • PixelRNN https://blog.openai.com/generative-models/
  14. 14. GAN (Generative Adversarial Networks)• 적대적 생성 모델 • 이미지를 생성하는 생성자(Generator, G)와 이미지의 진짜/가짜를 판단하는 판별자(Discriminator, D)가 상호 경쟁하며 발전 https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f
  15. 15. Pixel RNN(Recurrent Neural Networks)• 픽셀 단위로 반복 학습 • 제한된 정보를 이용해서 다음 상태를 예측 https://arxiv.org/abs/1601.06759
  16. 16. 딥러닝을 이용한 생성 모델 • VAE (Variational Auto-Encoder) • GAN (Generative Adversarial Networks) • PixelRNN
  17. 17. 딥러닝을 이용한 생성 모델 • VAE (Variational Auto-Encoder) • GAN (Generative Adversarial Networks) • PixelRNN
  18. 18. 2. Autoencoder - 자기 자신을 재생산
  19. 19. 일반적인 딥러닝 모델 - 분류 cat dog horse table flower 93% 46% 12% 20% 18%
  20. 20. 일반적인 딥러닝 모델 - 분류 cat dog horse table flower 93% 46% 12% 20% 18% X Yf
  21. 21. AutoEncoder http://kvfrans.com/variational-autoencoders-explained/
  22. 22. AutoEncoder http://kvfrans.com/variational-autoencoders-explained/ X Xf
  23. 23. 수학적으로는 f(x) = x 와 비슷 https://ocw.mit.edu/ans7870/18/18.013a/textbook/HTML/chapter01/section04.html
  24. 24. 본질은 압축 http://codingplayground.blogspot.kr/2015/10/what-are-autoencoders-and-stacked.html
  25. 25. 본질은 압축 https://foundationsofvision.stanford.edu/chapter-8-multiresolution-image-representations/
  26. 26. 비복원 압축 http://goelhardik.github.io/2016/06/04/mnist-autoencoder/
  27. 27. 본질은 압축 http://codingplayground.blogspot.kr/2015/10/what-are-autoencoders-and-stacked.html
  28. 28. 이미지의 차이를 줄이는 쪽으로 학습 • 이미지의 차이(error) = MSE = Mean Squared Error https://blog.insightdatascience.com/isee-removing-eyeglasses-from-faces-using-deep-learning-d4e7d935376f
  29. 29. 3. Variational Autoencoder - 자기 자신을 재생산 + 약간의 변화
  30. 30. Autoencoder http://kvfrans.com/variational-autoencoders-explained/
  31. 31. Autoencoder + α (variational) http://kvfrans.com/variational-autoencoders-explained/
  32. 32. 압축된 정보 = latent variable https://www.slideshare.net/TJTorres1/deep-style-using-variational-autoencoders-for-image-generation
  33. 33. Unit Gaussian → latent variable • 평균μ, 분산σ2, 표준편차σ를 가지는 분포의 형태 https://www.slideshare.net/TJTorres1/deep-style-using-variational-autoencoders-for-image-generation
  34. 34. Unit Gaussian • 분산=1(Unit) 인 정규분포 http://cshlife.tistory.com/217
  35. 35. Error = KL-div + MSE https://www.slideshare.net/TJTorres1/deep-style-using-variational-autoencoders-for-image-generation
  36. 36. KL-div • Kullback–Leibler divergence • Cross-entropy
  37. 37. KL-div • Kullback–Leibler divergence • Cross-entropy 사람 이름 차이 Solomon Kullback Richard Arthur Leibler
  38. 38. KL-div • Kullback–Leibler divergence • Cross-entropy : 정보량의 차이 http://blog.evjang.com/2016/08/variational-bayes.html
  39. 39. http://blog.fastforwardlabs.com/2016/08/12/introducing-variational-autoencoders-in-prose-and.html MNIST dataset (handwriting) 28 x 28 Reconstructed 28 x 28
  40. 40. Latent Variable Space http://blog.fastforwardlabs.com/2016/08/12/introducing-variational-autoencoders-in-prose-and.html
  41. 41. Latent Space Walking http://blog.fastforwardlabs.com/2016/08/22/under-the-hood-of-the-variational-autoencoder-in.html Reconstructed
  42. 42. 4. 생성 사례 - Dungeon Shape, 캐릭터 Portrait, Sprite 등
  43. 43. 개발 환경 • Tensorflow v1.0.1 (windows 10) • Python 3.5 • GTX1070 • Javascript (Web Demo)
  44. 44. Network Overview • VAE Network + recurrent generation • Pooling Layer 대신 Convolution Layer 에서 stride=2 (2칸씩 건너뛰기) • Kevin frans 의 코드 참고 http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html
  45. 45. Network - Encoder Input 64,64,c Conv1 32,32,64 Conv2 16,16,128 Conv3 8,8,256 Reshape 16384 Dropout 16384 Dense(μ) 800 Dense(σ) 800
  46. 46. Network - Encoder Input 64,64,c Encoder Dense(μ) 800 Dense(σ) 800
  47. 47. • 이미지를 800개의 가우시안 분포로 변환(압축) Network - Encoder Input 64,64,c Encoder Dense(μ) 800 Dense(σ) 800
  48. 48. Network – Decoder Conv_T 32,32,64 Conv_T 16,16,128 Dense 16384 Reshape 8,8,256 Dense(μ) 800 Dense(σ) 800 Conv_T 64,64,c Sample 800 Random 800
  49. 49. Network - Decoder • Random = 표준정규분포 난수 • Sample = μ + σ * Random(mean=0,std=1) = 정규분포 난수 >>> import numpy as np >>> print(np.random.normal(0,1)) -0.3835458243630663 https://www.mathsisfun.com/data/images/standardizing.svg
  50. 50. Network – Decoder Conv_T 32,32,64 Conv_T 16,16,128 Dense 16384 Reshape 8,8,256 Dense(μ) 800 Dense(σ) 800 Conv_T 64,64,c Sample 800 Random 800
  51. 51. Network – Decoder Output 64,64,c Conv_T 32,32,64 Conv_T 16,16,128 Dense 16384 Reshape 8,8,256 Dense(μ) 800 Dense(σ) 800 Conv_T 64,64,c Sample 800 Random 800
  52. 52. Network – Decoder Output 64,64,c Conv_T 32,32,64 Conv_T 16,16,128 Dense 16384 Reshape 8,8,256 Dense(μ) 800 Dense(σ) 800 Conv_T 64,64,c Sample 800 Random 800 Recurrent generation
  53. 53. Recurrent Generation import math import numpy as np import matplotlib.pyplot as plt %matplotlib inline def sigmoid(a): return 1/(1+math.exp(-a)) N = 200 random_y = np.random.randn(N) / 5000 _x = np.random.randn(N) plt.figure() plt.xlim(-6, 6) plt.scatter(_x, random_y) _x2 = _x * 6 plt.figure() plt.xlim(-6, 6) plt.scatter(_x2, random_y) sigmoid1 = [sigmoid(m) for m in _x] plt.figure() plt.xlim(-6, 6) plt.ylim(0, 1) plt.scatter(_x, sigmoid1, c='red') sigmoid2 = [sigmoid(m) for m in _x2] plt.figure() plt.xlim(-6, 6) plt.ylim(0, 1) plt.scatter(_x2, sigmoid2, c='red') x x * 6 sigmoid(x) sigmoid(x * 6)
  54. 54. Network - Decoder • Recurrent Generation Original Generated (step=6)
  55. 55. Network – Decoder Output 64,64,c Dense(μ) 800 Dense(σ) 800 Sample 800 Random 800 Decoder
  56. 56. • 샘플링된 데이터에서 이미지 데이터를 만들어냄 (복원) Network – Decoder Output 64,64,c Dense(μ) 800 Dense(σ) 800 Sample 800 Random 800 Decoder
  57. 57. Play with Data • 목표 - Latent Variable 탐색 - Shape morphing (e.g. Town → Maze) - Online Demo
  58. 58. Data1 : Dungeon Shape http://imgur.com/tYeJX8W
  59. 59. Dungeon Shape • 64 x 64, Binary Image roomsAndCorridor openCave diamondMine town division maze
  60. 60. Dungeon Shape • Procedural Generation • N = 50,000 openCave diamondMine town division mazeroomsAndCorridor
  61. 61. Dungeon Shape • t-SNE • 고차원→저차원 • 800D→2D
  62. 62. • t-SNE • 일부분 확대 Dungeon Shape
  63. 63. Latent Variable (z) roomsAndCorridor z1 Encoder Network (conv) Decoder Network (conv) roomsAndCorridor
  64. 64. Latent Variable (z) z2 Encoder Network (conv) Decoder Network (conv) diamondMine diamondMine
  65. 65. Shape Morphing z1 . . . . zn=z1+(z2-z1)*n/step . . . . z2 Decoder Network (conv) step ??? ??? ???
  66. 66. Latent Variable (z) roomsAndCorridor z1 Encoder Network (conv) Decoder Network (conv) roomsAndCorridor mu1 std1
  67. 67. Shape Morphing mu1 . mun=mu1+(mu2- mu1)*n/step . mu2 Decoder Network (conv) ??? ??? ??? mu1 std1 mu2 std2 std1 . stdn=std1+(std2- std1)*n/step . std2 z1 . . . . zn . . . . z2
  68. 68. Shape Morphing
  69. 69. Shape Morphing
  70. 70. Shape Morphing
  71. 71. Interpolation • Linear Interpolation (Lerp) • Spherical linear Interpolation (Slerp) http://www.geeks3d.com/20140205/glsl-simple-morph-target-animation-opengl-glslhacker-demo/
  72. 72. Shape Morphing • Slerp • Lerp
  73. 73. • random
  74. 74. • random
  75. 75. • random (threshold = 0.5)
  76. 76. Online Demo • mattya, carpedm20 의 코드 참고 https://github.com/mattya/chainer-DCGAN https://github.com/carpedm20/DCGAN-tensorflow
  77. 77. Data2 : Character Portrait • Game Character Hub 의 Face Set • N = 50,000 • 128 px → 108 px (crop) → 64 px (resize)
  78. 78. • mean
  79. 79. • random
  80. 80. Character Portrait • t-SNE • Y값 없으므로 단색
  81. 81. Character Portrait • t-SNE • 일부분 확대
  82. 82. • t-SNE
  83. 83. • t-SNE(x2)
  84. 84. • t-SNE(x2)
  85. 85. • t-SNE(x2)
  86. 86. Analogy • 유추 • A : B = C : ? • ? = (B + C) – A <Sampling Generative Networks>, Tom White J-Diagram http://www.nibcode.com/en/psychometric-training/abstract-reasoning-test/
  87. 87. Analogy A B C ?
  88. 88. Analogy A B C ? = (B + C) – A
  89. 89. Analogy A B C ? = (B + C) – A
  90. 90. Analogy A B C ? = (B + C) – A
  91. 91. Online Demo
  92. 92. Data3 : Character Sprite • Game Character Hub: Portfolio Edition 의 Character Set • N = 450,000 = 50,000 x 9 (frames) • 64 x 64
  93. 93. Mean Calculation https://github.com/Newmu/dcgan_code/tree/gh-pages
  94. 94. Mean Calculation ? 3rd frame 3rd frame mean 5th frame mean
  95. 95. Mean Calculation 3rd frame 3rd frame mean 5th frame mean guessed
  96. 96. Mean Calculation 3rd frame 3rd frame mean 5th frame mean guessed real
  97. 97. Mean Calculation
  98. 98. Mean Calculation
  99. 99. Experiment 3rd frame +3rd frame mean +3rd frame mean +3rd frame mean +3rd frame mean +3rd frame mean +3rd frame mean -3rd frame mean -3rd frame mean -3rd frame mean -3rd frame mean -3rd frame mean -3rd frame mean
  100. 100. Online Demo • https://greentec.github.io
  101. 101. 결론 & 논의 • 레벨 디자인 같은 데이터는 실사용 가능 • 이미지 데이터는 실사용 무리 → 멀지 않은 미래에 가능할 듯 (BEGAN 등) • Original 데이터는 많을수록 좋음 • 이 방법으로는 최소 N > 2,000 • Latent variable 탐색 방법은 발전의 여지가 많음
  102. 102. 더 알아보기 • VAE-GAN • BEGAN • PixelVAE
  103. 103. Q&A
  104. 104. 감사합니다.
  105. 105. Unit Gaussian https://experimentationground.wordpress.com/2016/10/01/intuition-and-math-behind-variational-autoencoder/
  106. 106. Unit Gaussian https://experimentationground.wordpress.com/2016/10/01/intuition-and-math-behind-variational-autoencoder/
  107. 107. Data3 : Korean Font • 142개 무료 폰트 활용 • 표준 글자 2,442자 사용 (특수문자 포함)

×