인공지능 논문작성과 심사에관한요령
•
9 likes•2,473 views
지난주말에 있었던 제 4회 대한신경집중치료학회 편집위원회 워크샵에서 발표했던 내용중에 발췌한 것입니다. 원래 제목은 "인공지능 관련 연구: 논문 작성과 심사에 관한 요령" 입니다. 최근에 deep learning in medical imaging으로 2편의 리뷰와 논문 1편, CADD 논문, 앙상블 논문 1편이 되면서 요청이 온것 같습니다.부족한 제가 하기 어려운 주제를 맡았는데, 혹시 도움이 되실 분이 있으면 도움을 되시라고 올려드립니다. 결론은 인공지능 연구라고 특별히 다르지는 않지만, 공학 연구와 의학연구가 다르고, 인공지능 특성을 잘 이해해야 한다 정도 될것 같습니다. (상당부분 저희병원 박성호 교수님의 radiology 논문 Methodology for Evaluation of Clinical Performance and Impact of Artificial Intelligence Technology for Medical Diagnosis and Prediction을 참고했습니다.)
More Related Content
Similar to 인공지능 논문작성과 심사에관한요령
Similar to 인공지능 논문작성과 심사에관한요령 (20)
More from Namkug Kim
More from Namkug Kim (20)
Recently uploaded
Recently uploaded (20)
인공지능 논문작성과 심사에관한요령
- 1. 인공지능 논문작성과 심사에 관 한 요령 1 Namkug Kim, PhD namkugkim@gmail.com Medical Imaging & Intelligent Reality Lab Convergence Medicine/Radiology, University of Ulsan College of Medicine/Asan Medical Center South Korea
- 3. Big data : Google Trends/Facebook 4 Nature 2008 약물중독의 상관관계를 분석하여 담배, 알코올, 의약품의 중독에 대해 각각 86%, 81%, 84%의 정확도로 선별
- 4. Big data : IoT Thermometer • IoT 스마트폰 체온계 – 미국의 헬스케어 스타트업 Kinsa : – 미국 전역에서 실시간으로 측정 되는 '체온 빅데이터’ – Patients-derived health data – 아이들의 학교 기반으로도 데이 터 • CDC 독감 통계 몇주 vs Kinsa 실시간 • B2B 모델: – 수요나 생산량 • 독감 예방 접종 혹은 항생제, 살균제 등의 약 • 칫솔, 오렌지 쥬스, 수프 등 5 독감의 트렌드: 2.5년 동안 CDC의 결과 비교
- 5. Opportunity 6 8 trillion exam: Healthcare Industry 2 trillion : wastes in healthcare industry Better experience Imaging : Unnecessary tests Lower cost Oncology: Variability of Care Better outcomes Life sciences: Failed clinical trials Government: Fraud, Waste and Abuse Value Based Care: Cost of chronic disease 360 billion : total IT and healthcare market opportunity *IBM Watson
- 6. Artificial Intelligence(인공지능) 7 Machines (SW, robots) that think and act like humans Make machines do things at which humans are better Solve tasks that, if done by humans, require intelligence 1950: Turing’s paper, 1956: “Artificial Intelligence (AI)”
- 7. 8
- 8. New Definition • Any device that perceives its environment (E) and takes actions (A) that maximize its chance of success at some goal (G). • a machine mimics "cognitive" functions that humans associate with other human minds, such as "learning" and "problem solving” 9
- 9. Artificial Intelligence(인공지능) • Weak artificial intelligence(약인공지 능) – Narrow AI, applied AI – 정해진 목적을 위해 사용하는 인공지능 • 바둑, 체스, 스팸 필터링, 쇼핑 추천, 자율운전 • Strong artificial intelligence(강인공 지능) – 인간급의 인공 지능 – 사고, 계획, 문제해결, 추상화, 복잡한 개념 학습
- 10. 지도학습 vs 비지도학습 • Supervised Learning (지도학습) – Naïve Bayesian Classifier – Support Vector Machine – Artificial Neural Network • Deep Learning • Unsupervised Learning (비지도학습) – k-means
- 11. Paradigm shift 45 Analog Digital vs Program Deep Learning vs Data GS Results Data GS Program<< Ref Little cost for duplication Little cost for developing program
- 12. 병원, 약, 인공지능 46 ?
- 13. Deep learning & Medicine • Keyword Search “Deep learning” in PubMed Updated on September 14th, 2017
- 14. Better Decision in Medicine: Clinical Decision Support System / Risk Prediction • Precision medicine – Massive search of medical information – Mining medical records – Advanced analytics – Designing individualized treatment plans • Individualized/group risk prediction
- 15. IBM Watson for Oncology • IBM Watson for healthcare
- 16. Better Patient Management • Health assistance and medication management • Getting the most out of in-person and online consultations • Open AI helping people make healthier choices and decisions
- 17. Medication Monitoring Solution▪ AiCure ▪ A provider of a facial recognition and motion sensing technology to medical ingestion -Substantial funding from pharmaceutical industry, academic collaborators, and the National Institutes of Health ▪ Combine machine learning with smartphone technology to remind people to take their medicine ▪ The data it provides to its systems transmits in real time back to a clinician through a HIPPA – compliant network -Clinicians conforming through the system that the patients are taking their medicine as instructed Sends the patient a reminder, and then requests that they use the camera built into their phone to video themselves taking the medicine Visually confirms that the person in the video is the patient, and then to identify the pill in the mouth of the patient to prove that they have taken their medicine 1) Since 2009, New York-based, $12M Funding
- 18. Efficiency • Speech Recognition • Medical imaging – Image processing, detection, diagnosis, classification • AI assistant / Chatbot – Scheduling, consulation • Surgical Robot
- 19. Drug Discovery http://fortune.com/2016/04/22/berg-pacreatic-cancer-artificial-intelligence/ http://tech.co/berg-medicine-artificial-intelligence-2016-07 http://www.wired.co.uk/article/niven-r-narain-ai-drugs-wired2015 • Data analytics software + in- the-lab drug development to find new treatments • Analysis of massive amounts of biological data to uncover unexpected connections between healthy and sick patients -The resulting insights allow for a more informed hypothesis, which in turn enables more efficient drug development -Provides real time analytic solutions that predict the impact of treatment plans at the individual level to optimize population health strategies ✓ Starts by drawing sequencing data from human tissue samples, as well as information about protein formation, metabolites, and other elements of functional data. ✓ The process produces trillions of data points from a single sample. The data is then combined with patient clinical information and analyzed by our proprietary artificial intelligence machine learning analytics program. The BERG Interrogative Biology® Platform
- 20. AI Application in Medical Imaging • Almost all aspects – Image transformation – Lesion segmentation – Lesion classification – Lesion detection – Finding similar cases – Assistance of interpretation
- 22. • Skin lesion using images – 129,450 images – 757 classes / 2032 diseases • Validation – Benign vs. Malignant vs. non-neoplastic – Nine classes with similar treatment plan • Test – Malignancy vs. Benign * Would be applicable to smartphone: universal access to vital diagnostic care
- 23. 62 • 5만장의 사진을 학습 • region-based convolutional neural network (R-CNN) 42명의 피부과 전문의와 진단능력 비교 • 한승석, 장성은 (AMC)
- 24. Computer Vision and Pattern Recognition Mar. 2017 • Task: Detection of LN metastasis from breast ca. • Data: Camelyon 16 dataset • Network: Inception V3 • Results: – 92.4% sensitivity (with 8 FP per image) – Cf. 82.7% human pathologist
- 26. Surgical Robot with AI • Will robot steal surgeons’ job? – NO. • Will robot CHANGE surgeons’ job? – It may... • Will robot and SUPER COMPUTER steal surgeons’ job? – …
- 27. Burger Patty Flipping Robot David Zito, CEO of Miso Robotics
- 28. 인공지능 의료적용 분야 인공지능 분야 시각지능 언어지능 판단지능 자동분류 요약/창작 공간지능 임상시험 케이스선정 신약개발프로세스 진료보조 비서서비스 음성인식 의무기록 데이터기반 정밀의료 유전체분석 약혼합사용 및 합병증 예측 진단검사추천 판독보조 정상유무판정 유사증례검색 판독문 생성 병리분야 판독 보조 물류, 수술실, 병실 운영 로봇수술 의료 인공지능 인공지능 의료적용 분야 68
- 29. Artificial Intelligence (+Big Data) Will Redesign Healthcare 1. Precision medicine / Mining medical records / Designing treatment plans 2. Getting the most out of in-person and online consultations / Health assistance and medication management / Open AI helping people make healthier choices and decisions 3. Assisting repetitive jobs 4. Drug discovery / Clinical trial Case Matching 5. Analyzing, redesigning a healthcare system http://medicalfuturist.com/ Aug, 2016
- 30. Radiologist 71
- 31. Pathologist 72
- 32. 1. 영상 인식 및 분할 : CNN Image tagging, retrieval Object recognition Scene segmentation
- 33. 흉부X선 소견의 분류 @AMC (-) (+) Nodule Interstitial Opacity Consolidation Pleural Effusion
- 35. Kidney 3D Semantic Segmentation Volumetric CT with contrast enhancement └ 3, 8, 11, 13, 14, 18, 21, 24, 26, 30, 32, 35, 36, 39, 44, 46, 50, 51, 52, 53 (20 cases : 6 left, 14 right) 6 classes : artery, cancer, cyst, parenchyma, ureter, vein 4 classes : artery, (cancer, cyst, parenchyma), ureter, vein AMC 비뇨기과 김청수, 건강의학과 경윤수 교수
- 36. Human Segmentation Human Segmentation AI_ 1st Test AI_ 1st Test AI_ 2st Test AI_ 2st Test AI_ 3rd Test AI_ 3rd Test rebuilding ground truth increasing data 0.88 Active Learning DICE 0.91 0.95
- 37. 2. 음성/시그널 인식 및 번역 Speech Recognition Machine Translation Speech Recognition + Machine Translation
- 40. 3. 비디오 인식 Video understanding (Google, 2014) Scene parsing (NYU/Facebook , 2014) NVIDIA DRIVE PX, 2015 Google Lens 4 General Sensor
- 41. 중이염 동영상 인식 및 모니터링 AMC ENT 정종우 교수님 협력연구 수술장 동영상 기술 개발 중이염, 인공와우 수술 동영상자료의 분류 수술영상을 각 단계로 분류 : 1000 례 이상 분류된 영상의 구조물 명시와 시술 단계 구분 및 예측 93 수술장 영상 기술 개발 영상 내 물체 인식의 경우 동영상 같은 경우에는 CNN과 RNN을 결합 한 CNN-RNN 모델 사용 동영상 데이터 학습을 통한 수술 단 계 분류 알고리즘 - 개발정리된 영상을 이용한 인공지능 학습 알고리즘 개발 - 좌우측과 다양한 형태의 유양동에 따 른 적용 - 알고리즘의 임상 검증 및 평가(수술단 계 및 정량화 정보) AMC ENT 정종우 교수
- 42. 4. 비디오 + Device(차)
- 43. Deep Learning for Endoscopy: Detection and Classification of Colonic Polyp ▪ Overall architecture Polyp detection in endoscopy (video) Close-up shot on polyp of interests Pathological Diagnosis (multi-modal) Benign Adenoma (0.98) • https://arxiv.org/abs/1512.03385 “Deep Residual Learning for Image Recognition” • https://arxiv.org/abs/1512.04150 “Learning Deep Features for Discriminative Localization”
- 44. DDx: NICE Classification of Colon Polyp Data NICE I: 25 NICE II: 89 NICE III: 19 Byun JS, Park B, Kim N, AMC
- 45. 5. GAN Deep Convolutional Generative Adversarial Networks (DCGAN) Rotations are linear in latent space Bedroom generation Arithmetic on faces
- 46. Domain Adaptation : Overcoming Differences (thickness, kernel, vendor, etc) 101AMC 영상의학과 서준범, 이상민 교수
- 47. 6. 영상 해석/자막 생성 Image Caption Generation Video Caption Generation
- 48. Generated annotation - Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, Ronald M Summers, "Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation," CVPR 2016.
- 49. 104 7. CBIR for Medical Images AMC 영상의학과 서준범, 이상민 교수
- 50. 105 The Previous Research for Quantification Definition of Similar Lung Images Extraction of Distribution Features Extraction of Distribution Features * https://en.wikipedia.org/wiki/Large_margin_nearest_neighbor * Y.J.Chang, et al,. “A support vector machine classifier reduces interscanner variation in the HRCT classification of regional disease pattern in diffuse lung disease: Comparison to a Baysian classifier”, Medical Physics 40 (5), 051912 (2013) AMC 영상의학과 서준범, 이상민 교수
- 53. RSNA 2017 Theme : “Explore, Invent, Transform” About 70 scientific sessions related with ML/DL – Not radiology conference, but AI conference Machine Learning Pavilion (Showcases) 45 Companies Machine learning theater Educations Machine Learning Nvidia DLI handson 108
- 54. Interpretability : Machine Operable, Human Readable Visual attention Category – feature mapping Sparsity and diversity 109
- 55. Uncertainty Uncertainty of training data In clinical situation, it is common Deep Bayesian Modeling Uncertainty of classification/prediction of Machine Learning 115
- 56. Novelty (Untrained catergory) In clinical situation Novelty is everywhere, especially supervised learning Rare diseases, but well known to medical doctors Hard to training How to determine novel (untrained) category Unsupervised learning Semi-unsupervised learning Normal vs abnormal Abnormality Detection 116
- 57. Case Orchestration Emergency 등 을 미리 감별해서 먼저 판독할수 있 게 순서 조정 Agfa, IBM, Philips, etc 118
- 58. Big data PACS platform Bigdata PACS Arterys, Zebra Medical Vision Quantifying every data Transforming PACS into big data platform Advanced processing service (APS) Lung nodules – Detection : location – Segmentation : boundary drawing 119
- 59. Abnormality Filtering DeepRadiology, Inc. by Le Cun 120
- 60. AI apps Platform 의학의 발달은 terminology의 발달 Contemporarily contradictory, and evolutionary AI apps platform Philips, Nuance, Siemens, etc 영상의학과 의사의 tasks : 약 5 만개 새로운 질환 발견 정의 의료영상 장비의 발달 121
- 62. CT Image Texture Features of NSCLC 153 Unsupervised Hierarchical Clustering
- 64. Radiomics 기반기술 영상분할 157 영상 정합 영상 분할 정량 특징 획득 분석 (자동분류자) 영상처리를 통한 Radiomics* * Modified from Nature Comm. 5, #4006 영상정합 다차원분석, 검색기능영상영상표준화 영상시스템 • 다기관/이종장 비 표준화영상 획득 • 영상저장 표준 화 (DICOM) • 다기관지원 • 클라우드 PACS • 인포매틱스시 스템 통합분석 • 실감형 정보가 시화 (모바일) • 다양한 영상정 보 통합 • 이종임상영상 간 정합 (MRI, PET, CT, 병리 등) • 병소 구분 • 크기, 형태분석 • 질환별, 영상별 최적 분할 기술 • 고속처리, 시각 화기술 • 관류, 확산능 등 다양한 기능 정량분석 • 질환별, 영상별 최적 모델링기 술 • 다중모달리티 정보통합 및 다차원분석 인공지능 • 다차원영상 시각화 • 영상간 유사도 검색 • 이미지 온톨로지 검색 Radiomics
- 65. Technical Issues 질환별 빅데이타 시스템 구축시 시간 소요 및 질 (외적 타당도 검증) • 기구축 고품질 코호트 이용 • 전담 전문인력 배치를 통해 질 보증 • 지속적 다기관 영상 데이타 축적 의료기관마다 상이한 영상 및 의료 데이타 • 영상프로토콜 표준화, 팬텀 개발 • 정량 분석의 자동화를 통해 재현성 • 의료 공통데이타모델 (CDM) 사용 158 질환이 심한 영상 자동 분할 실패 / 2D 영상기반 영상처리기술 • Multi-atlas 기반의 영상분할 기법 이용 • 기개발기술 환자특징요소 보완/고도화 • 3차원 질감 및 형상 영상요소를 추출 임상환자의 질환 등의 변이 / 100만건 이상의 대용량 영상 처리 / 모바일환경 대응 • Self feature 생성 기법 • Deep Learning 등 최신 기법 도입 • GPU CUDA, OpenCL, OpenMP등 병렬화 • 모바일 환경 인터페이스 구축 빅데이타 공개에 따른 개인정보 침해 • 데이타 익명화 경험활용 • Virtual DB 형태의 개별데이타 이용
- 66. Source of error • Quantization • Quantization error 161
- 67. Phantom 162
- 68. Imaging in Drug Development beyond diagnostic imaging • Biomarker = Biological marker Objective indicator of a biological, pathological or pathogenic process • Quantitative • Accuracy & Precision (Producibility) • Multi-center Robust acquisition protocol • Multiple scanner platforms • Acquisition protocol by analyst/reader + physicist • Same-scanner imaging Scanner qualification & quality control • Phantoms Site personnel engagement • Training & education • Radiologist as co-PI? waterarc Outer air1 Inner air Bed 1 Bed 2 Outer air2 CT Phantom Manufact urer Scanner kVp Tube cur rent (mA) Average e ffective tu be current (mA) Slice thick ness Pitch Gantry r otation ti me Reconstru ction Filter Siemens Sensation 16 140 200 100 0.7 1.000 0.5 B30 Siemens Sensation 64 140 270 99.9 0.7 1.000 0.37 B30 GE LightSpeed 16 140 190 101.2 0.625 0.938 0.5 Standard GE LightSpeed VCT 64 140 250 101.6 0.625 0.984 0.4 Standard Philips Brilliance 16 140 142 100.2 0.8 1.063 0.75 B Philips Brilliance 64 140 135 99.9 0.625 1.014 0.75 B Variation in Emphysema indexes (%) from four different CT scanners. Before DC After DC 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5 Time Point(s) EmphysemaIndex(%) Density Correction (outair) ; Standard ; -950 HU Thresholding Siemens 16 Philips 16 (2) Philips 40 Toshiba 64 Outsider air volume density correction based on water and air in four different CT scanners (Water was assumed as 0 HU and air as -1000 HU). FEV1 FEV1/FVC Emphysema index (Base) -0.318 0.002 -0.510 <0.001 Emphysema index (Inner air correction) -0.597 <0.001 -0.612 <0.001 Emphysema index (Outer air correction) -0.394 <0.001 -0.497 <0.001 Mean lung density (Base) 0.259 0.011 0.460 <0.001 Mean lung density (Inner air correction) 0.487 <0.001 0.528 <0.001 Mean lung density (Outer air correction) 0.383 <0.001 0.499 <0.001 . Partial correlation analysis adjusted by age and sex between CT and PFT parameters in Philips and Toshiba (n=98). Accepted at RSNA 2011
- 69. Issues • High-dimensional or over- parameterized diagnostic/predictive models using artificial deep neural network • statistical methods – assessing the discrimination and calibration performances • effects of disease manifestation spectrum and disease prevalence on the performance results. 164
- 70. Data Set • The performance using internal and external datasets • importance of using an adequate external dataset obtained from a well- defined clinical cohort – to avoid overestimating the clinical performance due to overfitting in high- dimensional or over-parameterized classification model and spectrum bias, and the essentials for achieving a more robust clinical evaluation. 165
- 71. External Validation • data obtained in newly recruited patients (referred to as temporal validation) • collected by independent investigators at a different site (referred to as geographic validation) • randomly split from the entire dataset and kept untouched for use as a test – (as in the training-validation-test steps) dataset, while the remaining main portion of the data is used for the training and the validation steps 166
- 72. Effect of Spectrum On Diagnostic/Predictive Performance • Prospective clinical trials, – which typically recruit subjects uniformly and consecutively according to eligibility criteria explicitly defined for a particular clinical setting, • Data for a deep learning – for collected from multiple heterogeneous sources – unnatural ratio between disease vs normal • e.g., severity, stage, or duration – disease; – presence and severity of comorbidities; – demographic characteristics 167
- 73. Effect of Prevalence On Prediction of Probability • The probability of disease – when a particular test result is given, • as the post-test probability, – determined by the LR of the test and pretest probability • (i.e., disease prevalence) according to Bayes’ theorem • pretest odds × LR = post-test odds, – where odds = probability / (1 − probability) • post-test probability = pretest probability × LR ÷ (1 − pretest probability + pretest probability × LR) = prevalence × LR ÷ (1 − prevalence + prevalence × LR). 168
- 74. What’s Consideration • Clinical trials and observational outcome studies – for ultimate clinical verification of diagnostic/predictive artificial intelligence tools – through patient outcomes, beyond performance metrics, and how to design such studies. 169
- 75. A Critical Trade-Off ; Effect Size
- 79. Threshold Non-diseased cases Diseased cases TPF,sensitivity FPF, 1-specificity ROC curve ROC curve AUC
- 80. ROC Analysis
- 81. FROC 176
- 83. ROC Analysis ROC curve of a hypothetical machine learning algorithm that aims to distinguish lung cancer and benign lung nodules obtained from the results in Table. The AUC value is 0.923.
- 84. Calibration Example calibration plot. The x-axis shows average predicted probability values for each decile, and the y-axis show the corresponding observed probability in each decile. The error bars represent 95% CIs of the mean predicted probabilities.
- 85. Potential sources of pts exclusion
- 86. Clinical Trial Design for AI (I) A. Traditional randomized controlled trial
- 87. Clinical Trial Design for AI (II) Clinical trial designs to assess the impact of an artificial intelligence tool on patient outcome. B. Cluster randomization of time periods. (E.g, random sequence of four time periods)
- 91. Comparison btw Brain and NN 187 1. 10 billion neurons 2. 60 trillion synapses 3. Distributed processing 4. Nonlinear processing 5. Parallel processing 6. Efficiency (20~25W, 하루섭취량의 20~25%) 1. Faster than neuron (10-9 sec) cf. neuron: 10-3 sec 3. Central processing 4. Arithmetic operation (linearity) 5. Relatively Sequential processing 6. Efficiency (Titan X : 250W) cf. 1[kcal] = 1.16[Wh], 1W=1J=1Nm/s, 1cal=4.2J=1.163mWh Brain Computer
- 92. Clinical Collaborators@Asan Medical Center Radiology: Joon Beom Seo, SangMin LeeA,B, Dong Hyun Yang, Hyung Jin Won, Ho Sung Kim, Seung Chai Jung Neurology: Dong-Wha Kang, Chongsik Lee, Jaehong Lee, Sangbeom Jun, Misun Kwon, Beomjun Kim Cardiology: Jaekwan Song, Jongmin Song, Younghak Kim Internal Medicine: Jeongsik Byeon Pathology: Hyunjeong Go Surgery: Bumsuk Go, JongHun Jeong, Songchuk Kim MI2RL(Medical Imaging and Intelligent Reality Lab.)