Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Interpretability and informatics of deep learning in medical images3

대한의료정보학회 2018 김남국 발표자료.

  • Login to see the comments

Interpretability and informatics of deep learning in medical images3

  1. 1. Interpretability and informatics of deep learning in medical images Namkug Kim, PhD Medical Imaging & Intelligent Reality Lab. Convergence Medicine/Radiology, University of Ulsan College of Medicine Asan Medical Center South Korea
  2. 2. Researches with LG Electronics Coreline Soft Inc. Osstem Implant CGBio VUNO Kakaobrain Conflict of Interests Stockholder Coreline Soft, Inc. AnyMedi Co-founder Somansa Inc. Cybermed Inc. Clinical Imaging Solution, Inc AnyMedi, Inc. Selected Grants as PI NRF, South Korea 7T용 4D 자기공명유속영상을이용한 심뇌혈관 질환의 in-vivo유동 정량화 SW개발, 2016 4D flow MRI을 이용한 심혈관 질환의 in-vivo유동 연구, 2015-7 자기공명분광영상 및 MRI의 통합 분석 소프트웨어 개발 KEIT, South Korea DigitalDentistry, 2018-2022 의료영상 인공지능 PACS 과제, 2016-20 3DP 척추 맞춤형 임플란트, 2016-20 3D 프린터 기반 무치악 및 두개악안면결손환자용 수복 보철물 제작, 재건 시스템 개발, 2015-9 근골격계 복구 수술 로봇 개발, 2012-7 영상중재시술 로봇시스템 개발, 2012-7 Spine및 Neurosurgery 수술보조용 항법 시스템 개발, 2001 의료용 3차원 모델 제작 S/W 기술 개발, 정통부, 2000 의료영상재구성에 의한 가상시술 소프트웨어 개발, 중소기업기술혁신개발, 중기청, 2001 KHIDI, South Korea 연구중심병원 육성과제, 2019-2028 인공지능 학습센터 과제, 2018-2023 영상 뇌졸중 예후 예측 및 치료방침 결정 시스템 개발, 2012-8 관동맥 관류 CT 의 자동 진단 프로그램을 활용한 허혈성 질환의 진단과 치료, 2013-6 RP를 이용한 척추나사못 삽입술 계획 프로그램 개발, 2000 Companies Fundings SiemensGermany, Hyundai Heavy Industry, OsstemImplant, S&G Biotech, Corelinesoft, MidasIT, AnyMedi, Hitachi Medical,Japan, Kakaobrain
  3. 3. Big data : IoT Thermometer IoT Thermometer Kinsa : Startup @ USA Real-time body temperature bigdata@USA Patients-derived health data Regional basis Influenza stats Kinsa : Realtime vs CDC : 3 week delay B2B model: Demands and production 독감 예방 접종 혹은 항생제, 살균제 등의 약 칫솔, 오렌지 쥬스, 수프 등 7 Influenza trends: comparison with CDC 2.5 y
  4. 4. Big data : Google Trends 8Nature 2008
  5. 5. Big data : Facebook 9 Correlation between Facebook usage vs drug addiction Accuracy : Tobacco(86%), Alcohol(81%), Drug(84%)
  6. 6. Healthcare Bigdata 7.5 Exa Byte/day (30% of every data) 생물학적 특성, 건강 이력, 웰빙 상태, 가는 곳, 지출 내역, 수면 상태, 식사와 배 설 / 실험 기록, 의학 영상, 유전정보, 액체생체검사, 심전도 /보험금 청구서, 임 상시험, 처방전, etc 10IBM Healthcare
  7. 7. Precision Medicine 11 / 46 • Bigdata • -Omics • Genomics, Transcriptomics, Proteonomics, Metabolomics, … • Personalized/Precision Backgrounds Big Data NGS Lab Imaging EMR
  8. 8. Opportunity 12 8 trillion exam: Healthcare Industry 2 trillion : wastes in healthcare industry Better experience Imaging : Unnecessary tests Lower cost Oncology: Variability of Care Better outcomes Life sciences: Failed clinical trials Government: Fraud, Waste and Abuse Value Based Care: Cost of chronic disease 360 billion : total IT and healthcare market opportunity *IBM Watson
  9. 9. Beyond Human-level Performance • Now, Machines Beat Human in Tasks Once Considered Impossible 5:0 vs Fan Hui (Oct. 2015) 4:1 vs Sedol Lee (Mar. 2016) Modified by KH Jung, PhD
  10. 10. Beyond Human-level Performance • Now, Machines Beat Human in Tasks Once Considered Impossible TPU Server used against Lee Sedol TPU Board used against Ke Jie
  11. 11. Beyond Human-level Performance • Now, Machines Beat Human in Tasks Once Considered Impossible Libratus(Jan 30, 2017) DeepStack(Science, Mar 02, 2017)
  12. 12. Beyond Human-level Performance • Now, Machines Beat Human in Tasks Once Considered Impossible
  13. 13. AI vs ML vs DL 17
  14. 14. Machine Learning scikit-learn algorithm cheat-sheet.
  15. 15. Feature Extraction 19
  16. 16. Comparison btw Brain and NN 20
  17. 17. Bio Plausible Neural Network Mimic human visual recognition system Neocognitron, proposed by Hubel & Wiesel in 1959 Visual primary cortex by cascading from S-Cell to C-Cell Each unit connected to a small subset of other units Based on what it sees, it decides what it wants to say Units must learn to cooperate to accomplish the task 21From Gallant and van Esses, Simon Thorpe
  18. 18. CNN : Major Breakthroughs in Feedforward NN K. Fukushima Yann Lecun G. Hinton, S. Ruslan Neocognitron (1979) • By Kunihiko Fukushima • First proposed CNN Convolutional Neural Networks (1989) • Yann Lecun • Back propagation for CNN • Theoretically learn any function Neocognitron LeNet-5 architecture Alex krizhevsky , Hinton LeNet-5 (1998) • Convolutional networks Improved by Yann Lecun • Classify handwritten digits D. Rumelhart, G. Hinton, R. Wiliams 1960 1970 1980 1990 2000 2010 2012 Perceptron XOR Problem Golden Age 1957 1969 1986 F. RosenblattM. Minsky, S. Papert • Adjustable weights • Weights are not learned • XOR problem is not linearly separble • Solution to nonlinearity separable problems • Big computation, local optima and overfiting CNN Breakthrough (2012) • By Alex Krizhevsky et al. • Winner of ILSVRC2012 by large marginDark Age (AI winter) Back propagation (1981) • Train multiple layers Multi-layer Perceptron (1986) 1950 Neocognitron (1959) • Hubel & Wiesel • by cascading from S- Cell to C-Cell
  19. 19. Feature Engineering vs Feature Learning Modified From Yann LeCun Knowledge-driven Feature Engineering Conventional Radiomics Data-driven Feature Learning Deep Radiomics •Feature Learning instead of Conventional Feature Engineering Removes Barriers for Multi-modal Studies and Data-driven Approaches in Medical Data Analysis
  20. 20. Machine Learning vs Deep Learning — Scale Matters — Millions to Billions of parameters — Data Matters — Regularize using more data — Productivity Matters — It’s simple, so we can make tools Data & Compute Accuracy Deep Learning Many previous methods Deep learning is most useful for large problems Modified by Nvidia DLI
  21. 21. Computational map 25 Dense Few Millions #ofVariables Completeness of Data (Sampling)Sparse More • Compute • Data • Storage • Bandwidth Computationally Intractable SpaceDeep Learning Neural Nets Statistical Analysis Algorithms, Closed Form Solutions Expert Systems Insufficient data for analysis [Un]supervised Learning Models Intuition
  22. 22. Perceptron 26
  23. 23. Multi-Layer Perceptron 27
  24. 24. Convolutional Neural Networks (CNN) A type of feed-forward neural network Inspired by biological process Weight sharing (convolution) + Subsampling (pooling) Reducing the number of parameters (Reduce over-fitting) Translation invariance Input 28 × 28 Feature maps 4@24 × 24 Feature maps 4@8 × 8 Feature maps 8@4 × 4 Feature maps 8@2 × 2 Feature maps 8 ⋅ 2 ⋅ 2 × 1 Output 10 × 1 Convolution layer Max-pooling layer Convolution layer Max-pooling layer Reshape Linear layer [LeCun, 1998]
  25. 25. Convolution and pooling 29
  26. 26. Feature Extraction by CNN 30
  27. 27. Convolution Neural Net 31
  28. 28. Spike Neural Net 32
  29. 29. AI Medical Device 33 • Verily@Google: Normal vs Abn, Anti-aging, Life Prolongation • IBM: 트루벤 인수등 40B USD • Apple: GSK, EMR +iPhone Healthcare Platform • Facebook: Incurable dx, Human cell atlas, 5000M USD • Zebra Medical Vision • AI medical imaging Dx : 1st place of investment • Medical imaging reading cloud service/1 USD
  30. 30. Clinical Unmet Needs on Deep Learning 효율적인 데이터 비식별화, Curation, 및 인공지능 기술을 이용한 스마트 레이블링 기술 다양한 장비와 병원마다의 차이를 극복하는 도메인 적응(Domain adaptation) 기술 블랙박스 성격을 완화하기 위한 인공지능 판단 해석(Interpretability) 및 시각화(Visualization) 연 구 의료 데이터가 가지고 있는 불확실성(Uncertainty), 인공지능의 판단이 가지고 있는 불확실성을 평가하는 기술 질환별 편향(imbalance) 문제를 해결하기 위한 전처리 및 인공지능 학습 기술 한번도 보지 못한 새로운 클래스를 미지판단(Novelty)을 통해서 검출해서 추후 의사가 따로 평가 희귀 데이터나 소수의 데이터를 학습하기 위한 One/Multi-shot Learning 기술 개발 반복측정된 영상데이터를 이용하여 딥러닝이 가지고 있는 재현가능성(Reproducibility) 연구 Adversarial Attack에 강인한 인공지능 연구 여러 물리 및 의학 법칙 등을 이용하여 기계학습(Physics Induced Machine Learning)의 효율을 증가하는 연구 34
  31. 31. AlphaGo Zero 35
  32. 32. 해석력 vs 정확도 36
  33. 33. DARPA XAI(eXplanable AI) 37
  34. 34. 해석력 강화 모델 38 Patrick Hall et al. (O’REILLY, 2017) Macro Tulio Ribeiro et al. (O’REILLY, 2016) Surrogate models Local-Interpretable-Model-agnostic Explanations (Perturbation)
  35. 35. 해석력 강화 모델 39 Yin Lou et al, ACM 2012 GAM, Generalized Additive Models Everything should be made as simple as possible, but not simpler. — Albert Einstein. • 변수간 상호작용 효과를 배제 • 설명력 제고 • 개별 변수별로 복잡한 구조의 알고리즘을 적용한 후 이를 더하기 형태로 종합
  36. 36. Analysis on Deep Learning Methods for Predicting Patient Survival Basic CNN Modified CNN Multi-layer CNN Residual NetworkFeed-forward NN
  37. 37. Interpretability : Machine Operable, Human Readable Visual attention Category – feature mapping Sparsity and diversity 41
  38. 38. Machine Operable, Human Readable Visualization of salient region in bone x-ray 42
  39. 39. Machine Operable, Human Readable Evidence hotpot for lesion visualization “SpineNet: Automatically Pinpointing Classification Evidence in Spinal MRIs” 43
  40. 40. Bias representation in CNN 44 Zhang et al., 2018b
  41. 41. Knowledge hierarchy hidden in conv-layers 45 Zhang et al., 2018a
  42. 42. Knowledge hierarchy hidden in conv-layers 46 Zhang et al., 2018a
  43. 43. Knowledge hierarchy hidden in conv-layers 47 Zhang et al., 2018a
  44. 44. 48
  45. 45. Decision tree Disentangled representation 49
  46. 46. Interpretable convolution layer 50
  47. 47. Visualization of interpretable filters in top conv-layer 51
  48. 48. Parse tree 52 Latent parse tree and configuration as quantitatively extractive rationale in detection
  49. 49. Filter’s location instability 53
  50. 50. 54
  51. 51. Uncertainty Uncertainty of training data In clinical situation, it is common Deep Bayesian Modeling Uncertainty of classification/prediction of Machine Learning 55
  52. 52. Novelty (Untrained catergory) In clinical situation Novelty is everywhere, especially supervised learning Rare diseases, but well known to medical doctors Hard to training How to determine novel (untrained) category Unsupervised learning Semi-unsupervised learning Normal vs abnormal Abnormality Detection 56
  53. 53. Reproducibility Test-retest is one of most important issues of biomarker Multiple scanning within similar date Evaluate reliability of AI/Deep learning Chest PA (Nodule) Nodule Size on Chest PA with YOLO – 56% : Variability of Size Marker – Chest PA 50 pairs DILD CT 57
  54. 54. Case Orchestration 58 Reading ordering by AI for efficient reading Agfa, IBM, Philips, etc
  55. 55. Big data PACS platform Bigdata PACS Quantifying every data Transforming PACS into big data platform Advanced processing service (APS) Lung nodules – Detection : location – Segmentation : boundary drawing » Quantifying size, long/short axis, volume, etc 59 Zebra Medical Vision + Google, EnvoyAI+TeraRecon, Arterys, Fujifilm Nuance + Partners HC, Philips, Siemens, etc
  56. 56. Normality/Abnormality Filtering BlockChain powered? Abnormality Filtering DeepRadiology, Inc. by Le Cun 60
  57. 57. AI apps Platform AI apps platform Philips, Nuance, Siemens, etc Radiologists’ tasks : 23,000 Follow/up development of imaging modality 61
  58. 58. Comparison btw Brain and NN 112 1. 10 billion neurons 2. 60 trillion synapses 3. Distributed processing 4. Nonlinear processing 5. Parallel processing 6. Efficiency (20~25W, 하루섭취량의 20~25%) 1. Faster than neuron (10-9 sec) cf. neuron: 10-3 sec 3. Central processing 4. Arithmetic operation (linearity) 5. Relatively Sequential processing 6. Efficiency (Titan X : 250W) cf. 1[kcal] = 1.16[Wh], 1W=1J=1Nm/s, 1cal=4.2J=1.163mWh Brain Computer
  60. 60. Strength of South Korea 주민번호 세계적 의료수준 예) 장기이식 생존률 대형병원 대부분의 질환의 Survival 데이 터를 쉽게 모을수 있음 ICT 산업 인프라 국내 산업 인프라 의료 인공지능은 수준이 떨어지 지 않음 114
  61. 61. [Parallel Worlds, Michio Kaku] Temperature Precipitation Wind … Weather E1 E2 E3 Ex Longitude C1 Landscape C2 Altitude C3 … C4 !! Disease Predictability ?? Prediction (Complexity)
  62. 62. Job Opening @ MI2RL_AMC, Seoul, SouthKorea Post-doc research fellow, PhD Students, Researchers 116 AMC, UoU Seoul South Korea Contact (
  63. 63. Collaborators Radiology Joon Beom Seo, SangMin LeeA,B, Dong Hyun, Yang, Hyung Jin Won, Ho Sung Kim, Seung Chai Jung, Ji Eun Park, So Jung Lee,Jeong Hyun Lee, Gilsun Hong Neurology Dong-Wha Kang, Chongsik Lee, Jaehong Lee, Sangbeom Jun, Misun Kwon, Beomjun Kim Cardiology Jaekwan Song, Jongmin Song, Young-Hak Kim Emergency Medicine Dong-Woo Seo Pathology Hyunjeong Go, Gyuheon Choi, Gyungyub Gong, Dong Eun Song Surgery Beom Seok Ko, JongHun Jeong, Songchuk Kim, Tae-Yon Sung Internal Medicine Jeongsik Byeon, Kang Mo Kim Anesthesiology Sung-Hoon Kim, Eun Ho Lee