Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Generative Adversarial Networks and Their Applications in Medical Imaging

612 views

Published on

의료 인공지능 교육 워크샵 (17.09.16) 발표자료
- Generative Adversarial Network에 대한 소개
- Medical imaging 분야에서의 연구 사례
- Our research

Published in: Technology
  • Be the first to comment

Generative Adversarial Networks and Their Applications in Medical Imaging

  1. 1. Copyright © KakaoBrain Corp. All rights reserved. Generative Adversarial Networks and Their Applications in Medical Imaging September 16, 2017 Sanghoon Hong
  2. 2. 1. Generative Adversarial Network (GAN) 2. GAN in Medical Imaging + Our Research Project Contents
  3. 3. • Discriminative vs. Generative Generative Model CNN classifier GAN “cat” an inverse function?
  4. 4. • Discriminative vs. Generative Generative Model https://duphan.wordpress.com/tag/generative-model/ data distribution p(x,z) or p(x|z) Difficult, but important
  5. 5. • Data distribution Generative Model A slide from “Crash Course on Machine Learning Part II” 주요 특징만 필요 세밀한 특성, 분포를 완전히 알아야함
  6. 6. • Applications? • Structured prediction (e.g., output text) • Much more robust prediction • Anomaly detection • … Generative Model
  7. 7. Generative Model Slide credit Goodfellow 2016
  8. 8. “the coolest idea in ML in the last twenty years” - Yann LeCun Generative Adversarial Network (GAN)
  9. 9. Generative Adversarial Network (GAN) https://github.com/hindupuravinash/the-gan-zoo/
  10. 10. • Variational Auto-Encoder • Data의 generation은 잘 됨 • 그러나 blurry face Why GANs are different? “Variational Autoencoder and Extensions” by Aaron CouCourville Sample interpolation
  11. 11. • Why are generated samples blurry? • Regress to the mean => blurry images Why GANs are different? Ledig, C. et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. MSE or Euclidean distance
  12. 12. Why GANs are different? from “Tutorial on Theory and Application of Generative Adversarial Networks” (CVPR17)
  13. 13. • Difficult to hand-craft a good perceptual loss function • => 그럴듯한지 여부를 판단하는 neural network가 있다면? Generative Adversarial Network
  14. 14. Generative Adversarial Network Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
  15. 15. Generative Adversarial Network Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016) real / fake 판단하는 네트워크
  16. 16. Generative Adversarial Network https://ibmathsresources.com/2014/08/27/zenos-paradox-achilles-and-the-tortoise/ DG Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. DG
  17. 17. Training GANs • Discriminator training Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
  18. 18. Training GANs • Generator training Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
  19. 19. • Adversarial loss (vanilla GAN) Training GANs D Real sample은 D(x) -> 1 Generated sample은 D(x) -> 0 G Generated sample도 D(x) -> 1
  20. 20. • Iterate these two steps until convergence • Eventually (we hope) that the generator gets so good that it is impossible for the discriminator to tell the difference between real and generated images. Discriminator accuracy = 0.5 Training GANs Discriminator Data Model Distribution Random guess
  21. 21. Training GANs Tutorial on Theory and Application of Generative Adversarial Networks (CVPR17)
  22. 22. • In practice? • Training stability (D vs. G balance, …) • Mode collapse Training GANs Generative Models II (CIFAR-CRM DLSS 2017) by Aaron Courville An image from “Generative adversarial networks” by Namju Kim
  23. 23. Training GANs https://github.com/hindupuravinash/the-gan-zoo
  24. 24. • Realistic image! GAN Results & Applications Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. realistic & diverse samples Nguyen, A. et al. (2016). Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
  25. 25. • Representation interpolation GAN Results & Applications Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. Memorizing X
  26. 26. • Super-resolution GAN Results & Applications Ledig, C. et al. (2016) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.
  27. 27. GAN Results & Applications Low-res image Generated high-res image Generated or not?
  28. 28. • Image-to-image translation GAN Results & Applications Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2016). Image-to- Image Translation with Conditional Adversarial Networks.
  29. 29. GAN Results & Applications
  30. 30. GAN Results & Applications
  31. 31. Copyright © KakaoBrain Corp. All rights reserved. GAN in Medical Imaging
  32. 32. • Low-dose CT (noisy) => Routine-dose CT Literature Review (1) De-noising
  33. 33. Literature Review (1) De-noising loss = (voxel-similarity loss) + adversarial loss
  34. 34. Literature Review (1) De-noising Low-dose Routine-doseGAN-denoised “Training with an adversarial network allows the generator to better learn the noise distribution in routine-dose CT and produce more realistic images for more accurate coronary calcium quantification.” MSE-denoised
  35. 35. • CT => PET image translation Literature Review (2) - Image-to-image translation
  36. 36. Literature Review (2) - Image-to-image translation
  37. 37. Literature Review (2) - Image-to-image translation SL + GAN A mask with high predicted SUV values (>2.5) (???)
  38. 38. Literature Review (2) - Image-to-image translation input CT ground-truth PET generated PET
  39. 39. • MRI => CT image translation Literature Review (2) - Image-to-image translation
  40. 40. Literature Review (2) - Image-to-image translation Supervised training w/ real data Blurry More realistic
  41. 41. Literature Review (2) - Image-to-image translation Pelvic datasetBrain dataset Best mean absolute error & peak signal-to-noise ratio
  42. 42. • GAN research가 있지만, 아직 결과물의 수준이나 양 이 부족 • 그럼 우리는 무엇을 해볼 수 있을까? Our Research
  43. 43. • Medical image generation with GAN Our Research Noise
  44. 44. • Applications? • Unsupervised or semi-supervised training • Progression forecast & Visualization Our Research
  45. 45. • Medical image generation from scratch? • Structure + Dynamics + Variations + … How to Tackle
  46. 46. • Medical image generation from scratch? • Structure + Dynamics + Variations + … • Might be too difficult How to Tackle (GAN은 global structure / counting 같은 것에서 특히 약함)
  47. 47. • (First step) image generation w/ structural hints • Structure + Dynamics + Variations + … • Image-to-image translation과 유사 How to Tackle
  48. 48. • Gaze estimation task • Model-based synthetic data => GAN => Realistic data How to Tackle
  49. 49. How to Tackle Self-regularization Visual Turing test => 51.7% acc. State of the art w/o label real data
  50. 50. How to Tackle Self-regularization (minimizing feature dist.) Local adversarial loss (=PatchGAN) History of refined images History 안쓰면 artifacts 발생
  51. 51. • Possible hints? • Model-based synthetic data • Normal image + conditioning • Normal image + synthetic or expert-guided label How to Tackle Synthetic or guided label Generator
  52. 52. First Trial Generator Discriminator Real normal Real patient Image + label (for controllability) Generated patient Our target - synthetic or supervised label - Matching image-segmentation pair? - Realistic image?
  53. 53. First Trial Generator Discriminator Image + label (for controllability) PatchGAN DCGAN or MAD-GAN Real normal Real patient Image matching Generated patient
  54. 54. • Typical GAN issues (training, …) • Sample quality? (feat. M.D. researchers) • How to evaluate generated samples • Practical effectiveness? • Unsupervised (or semi-supervised) segmentation/classification in medical domain Have far to go
  55. 55. • One more thing…
  56. 56. Visualization of disease progression Identity Age
  57. 57. • GAN: an interesting & effective way to generate data • GAN in Medical Imaging? => 아직 초기 단계 • “Medical Image Generation with GAN” Conclusion
  58. 58. THANK YOU

×