The Importance of Medical Multimedia

1. The Importance of Medical Multimedia Michael Riegler2, Pål Halvorsen2, Bernd Münzer1, Klaus Schoeffmann1 1 Institute of Information Technology Klagenfurt University, Austria 2 Simula Research Laboratory Norway

2. • Introduction & Overview • Multimedia Data in Medicine • Characteristics of Endoscopic Video • Different Fields and Communities • Application 1: Post-Procedural Usage of Surgery Videos • Domain-Specific Storage for long-term Archiving • Medical Video Content Analysis • Medical Video Interaction • Application 2: Diagnostic Decision Support • Knowledge Transfer • Analysis • Feedback • Explainability and Trust • Conclusions & Outlook Agenda ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 2

3. Introduction ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 3

4. Inspections and intervention produce many kinds of data • Medical text • OR reports, Patient records… • Sensor signals • ECG, EEG, vital signs • Medical images (radiology) • Ultrasound, x-ray • CT, MRI, PET, … • Medical video • Screenings • Surgery Multimedia Data in Medicine ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 4 à Signal Processing à Medical Imaging à Robotics à Multimedia à Data Mining „Human EEG without alpha-rhythm“ by Andrii Cherninskyi / CC BY-SA „Pankreatitis“ by Hellerhoff/ CC BY-SA„Ultrasound“, Public Domain

5. • Traditional open surgery ? • Minimally invasive interventions • Reduced trauma for patient • Inherently available video signal • Microscopic surgery Video Data Sources in Medicine ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 5 „Lobektomia“ by Wojciech Filipiak/ CC BY-SA „Cataract Surgery“, Public Domain „Laparoscopy“, Public Domain

6. • Strongly increasing usage of videos and images in daily routine • Endoscopic imaging as gold standard • Availability of cheap storage capacity • Manifold use cases • Real-time support during surgery • Sophisticated documentation of surgeries • Diagnosis support à Strong demand for effective storage, content processing and visualization! Motivation ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 6

7. Medical Endoscopy ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 7

8. • Minimally-invasive surgery (“keyhole surgery”) or screening • Enabled by modern video technology • Endoscope as the ’’Eye of the Surgeon’’ • Videos are captured for • Documentation • Retrospective analysis • Teaching / Education Medical Endoscopy ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 8 „Laparoscopy“, Public Domain

9. Diagnostic Endoscopy ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 9 • Diagnosis / Inspections • Gastroenterology (colonoscopy, gastroscopy) • Bronchoscopy • Hysteroscopy • … • Flexible endoscope • Natural orifices • WCE (Wireless capsule endoscopy) „Kussmaul Gastroscopy“, Public Domain „Colonoscopy“, Public Domain „Kolon transversum“ by J.Guntau / CC BY-SA

10. Therapeutic Endoscopy ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 10 • Therapy / Surgery • Laparoscopy • Cholecystectomy • Gynecological surgery • Urological surgery • … • Arthroscopy • … • Rigid endoscope • Small incisions „Laparoscopy“ by BruceBlaus / CC BY „Arthroscopy“, Public Domain

11. Endoscopic Video Examples ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 11

12. Domain-specific Characteristics & Challenges ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 12 • Full HD or 4K (even stereo 3D) • Single shot recordings • Up to multiple hours • Homogenous color distribution • Visually very similar content • Circular content area • Restricted motion • Geometric distortion • Specular reflections • Occlusions • Smoke • Noise, motion blur, blood, flying particles

13. Literature Overview ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 13

14. Overview ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 14 Münzer, Bernd, Klaus Schoeffmann, and Laszlo Böszörmenyi. "Content-based processing and analysis of endoscopic images and videos: A survey." Multimedia Tools and Applications (2017): 1-40.

15. Pre-Processing • Image Enhancement • Contrast enhancement, color misalignment correction… • Camera calibration and distortion correction • Specular reflection removal • Comb structure removal & super resolution • … • Information Filtering • Frame filtering • Image segmentation ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 15 T. Stehle. Removal of specular reflections in endoscopic images. Acta Polytechnica: Journal of Advanced Engineering, 46(4):32–36, 2006. J. Barreto, J. Roquette, P. Sturm, and F. Fonseca. Automatic Camera Calibration Applied to Medical Endoscopy. In 20th British Machine Vision Conference (BMVC ’09), 2009. B. Münzer, K. Schoeffmann, and L. Böszörmenyi. Relevance Segmentation of Laparoscopic Videos. In 2013 IEEE International Symposium on Multimedia (ISM), pages 84–91, Dec. 2013. A. Chhatkuli, A. Bartoli, A. Malti, and T. Collins. Live image parsing in uterine laparoscopy. In IEEE International Symposium on Biomedical Imaging (ISBI), 2014.

16. Real-time Support at Intervention Time Applications § Diagnosis support § Robot-assisted surgery § Context awareness § Augmented reality ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 16 “Robotic surgical system”, Public Domain T. Collins, D. Pizarro, A. Bartoli, M. Canis, and N. Bourdel. Computer-Assisted Laparoscopic myomectomy by augmenting the uterus with pre-operative MRI data. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 243–248, Sept. 2014. „Da Vinci Surgical System“ by Cmglee / CC BY-SA Slightly modified from: M. P. Tjoa, S. M. Krishnan, et al. Feature extraction for the analysis of colon status from the endoscopic images. BioMedical Engineering OnLine, 2(9):1–17, 2003.

17. • 3D reconstruction • Deforming tissue tracking • Image registration • Instrument detection and tracking • Surgical workflow understanding Enabling Techniques ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 17 L. Maier-Hein, P. Mountney, A. Bartoli, H. Elhawary, D. Elson, A. Groch, A. Kolb, M. Rodrigues, J. Sorger, S. Speidel, and D. Stoyanov. Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Medical Image Analysis, 17(8):974–996, Dec. 2013. S. Giannarou, M. Visentini-Scarzanella, and G. Z. Yang. Affine-invariant anisotropic detector for soft tissue tracking in minimally invasive surgery. In Biomedical Imaging: From Nano to Macro, 2009. ISBI’09. IEEE International Symposium on, pages 1059–1062, 2009.

18. Post-Procedural Applications Management and Retrieval • Compression and storage • Content-based retrieval • Temporal video segmentation • Video summarization • Visualization & Interaction Quality Assessment § Skills assessment § Education & Training § Error Rating § Assessment of intervention quality ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 18 M. Lux, O. Marques, K. Schöffmann, L. Böszörmenyi, and G. Lajtai. A novel tool for summarization of arthroscopic videos. Multimedia Tools and Applications, 46(2-3):521–544, Sept. 2009. D. Liu, Y. Cao, W. Tavanapong, J. Wong, J. H. Oh, and P. C. de Groen. Quadrant coverage histogram: a new method for measuring quality of colonoscopic procedures. In Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, pages 3470–3473, 2007. J. Muthukudage, J. Oh, W. Tavanapong, J. Wong, and P. C. d. Groen. Color Based Stool Region Detection in Colonoscopy Videos for Quality Measurements. In Y.-S. Ho, editor, Advances in Image and Video Technology, number 7087 in Lecture Notes in Computer Science, pages 61–72. Springer Berlin Heidelberg, Jan. 2012.

19. Post-Procedural Use of Surgery Videos ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 19

20. • Video documentation of endoscopic procedures is on the rise • “a picture paints a thousand words“, a moving picture paints millions! • In some countries even mandatory already • Current documentation practice poses many problems: • Hard task to retrieve relevant information • Huge amounts of storage space • High ratio of irrelevant data (“rubbish”) • Very inefficient encoding (especially for HD content) Motivation for Video Documentation ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 20

21. • Later inspection of specific moments • Discussion of critical moments (e.g., with OP team) • Information to patients • Preparation of future interventions • Forensics & investigations (e.g., comparisons) • Training & teaching • Surgical quality assessment (technical errors) Use Cases ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 21

22. • Vision • Archive and combine all relevant text, image, and video data • Make it easily accessible • Support surgeons at diagnosis, surgery planning, teaching, … • Improve quality of interventions • Challenges • Isolated systems / separation of data • Very Big Data • Only small fraction is actually relevant • Very specific domain characteristics Towards a Medical Multimedia Information System ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 22

23. Full Storage of Endoscopic Videos • Exemplary hospital • 5 departments (Lap, Gyn, Arthro, GI, ENT) • 2 operation rooms, each 4 ops/day, each op ca. 1-2h • à i.e. 40 interventions per day, each ~ 90 mins. • 60 hours video per day! • Assumption: HD 1920x1080, H.264/AVC • 270 GB / day (1h=4.5 GB) • 1.9 TB / week • 100 TB / year (200 TB MPEG-2) 4K: even more ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 23 Great challenge for a hospital’s IT department!

24. How to Reduce Storage Requirements? Exploit domain-specific characteristics: 1. Spatial compression optimization 2. Temporal compression optimization 3. Perceptual quality based optimization 4. Long-term archiving strategy Transcoding ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 24 up to 30% up to 40% up to 93%

25. Study on Video Quality • Subjective quality assessment • Catharina Hospital Eindhoven, NL • 37 participants • 19 experienced surgeons and 18 trainees • 7 women, 30 men, average age: 40 years • Subjective tests regarding maximum compression 1) Perceivable quality loss • Double-Stimulus (ITU-R BT.500-11) • Switch between reference and test video 2) Perceivable semantic information loss • Single Stimulus (ITU-R P.910) • Assessing random videos (incl. reference) Münzer, B., Schoeffmann, K., Böszörmenyi, L., Smulders, J. F., & Jakimowicz, J. J. (2014, May). Investigation of the impact of compression on the perceptional quality of laparoscopic videos. In 2014 IEEE 27th International Symposium on Computer-Based Medical Systems (pp. 153-158). IEEE. Session 1 Session 2 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 25

26. Assessment of Video Quality (Session 1) -5 0 5 10 15 20 25 30 35 0 3000 6000 9000 12000 15000 18000 21000 24000 20 22 24 26 28 18 20 22 24 26 18 18 DifferenceMeanOpinionScore(DMOS) Bitrate(Kb/s) Test Conditions Average bitrate Rating difference 1920x1080 1280x720 960x540 640x360 subjectively better than reference Reference video (MPEG-2, HD, 20 (35) Mbit/s) “lossless” ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 26 crf (constant rate factor)

27. Assessment of Video Quality (Session 2) 1. Visually lossless with 8 Mbit/s Q1 (in comparison to 20 Mbit/s) Reduction: 60% data vs. 0% MOS 2. Good quality with 2,5 Mbit/s and Q2 reduced resolution (1280x720) Reduction: 88% data vs. 7% MOS 3. Acceptable quality with 1,4 Mbit/s Q3 and lower resolution (640x360) Reduction: 93% data vs. 31% MOS 1 2 3 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 27

28. Example Videos 1280x720 Weak compression 16 MB (crf 18) 640x360 Strong compression 0,8 MB (crf 26) 20x ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 28

29. Long-term Performance ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 29 0 300 600 900 1200 1500 1800 2100 2400 0 1 2 3 4 5 6 7 8 9 10 11 12 Storagespace(GB) Month 0 2500 5000 7500 10000 12500 15000 17500 20000 22500 0 12 24 36 48 60 72 84 96 108 120 Storagespace(GB) Month

30. Medical Video Content Analysis ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 30

31. 1000 frames (sampled from 17min with 1fps) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 3 1

32. Content Relevance Filtering / Instrument Recognition ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 32 Münzer, B., Schoeffmann, K., & Böszörmenyi, L. (2013, December). Relevance segmentation of laparoscopic videos. In Multimedia (ISM), 2013 IEEE International Symposium on (pp. 84-91). IEEE. Primus, M. J., Schoeffmann, K., & Böszörmenyi, L. (2015, June). Instrument classification in laparoscopic videos. In Content-Based Multimedia Indexing (CBMI), 2015 13th International Workshop on (pp. 1-6). IEEE. Instrument detection for content understanding (e.g., op phase segmentation, following instruments in robot-assisted surgery) Out-of-patient Scenes Blurry Scenes Border Area

33. Phase Segmentation (Cholecystectomy) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 33 Manfred J. Primus, Klaus Schoeffmann and Laszlo Böszörmenyi. “Temporal Segmentation of Laparoscopic Videos into Surgical Phases“, in Proceedings of the 14th International Workshop on Content-Based Multimedia Indexing (CBMI 2016), Bucharest, Romania, 2016 à Phase segmentation through instrument recognition (color analysis, image moments, rules/heuristics)

34. Instrument Recognition/Tracking ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 34

35. Classification of OP Scene (Cataract Surgeries) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 35 Manfred J. Primus, Doris Putzgruber-Adamitsch, Mario Taschwer, Bernd Münzer, Yosuf El-Shabrawi, Laszlo Böszörmenyi, and Klaus Schoeffmann. 2018. Frame-Based Classification of Operation Phases in Cataract Surgery Videos. In Proceedings of the 24th International Conference on Multimedia Modeling 2018 (MMM2018). Lecture Notes in Computer Science, vol 10704, Springer, Cham, 241-253.

36. Surgical Action Classification ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 36

37. Gynecologic Laparoscopy: Relevant Surgical Actions ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 37 Dissection Coagulation Cutting cold Cutting Hysterectomy Injection SuturingSuction & Irrigation Petscharnig, S., & Schöffmann, K. (2017). Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications, 1-19.

38. Gynecologic Laparoscopy: Relevant Surgical Actions ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 38 Dissection– 58 Segs / 35.517 Pics Coagulation– 212 Segs / 84.786 Pics Cutting cold – 271 Segs / 26.388 Pics Cutting– 106 Segs / 92.653 Pics Hysterectomy– 25 Segs / 68.466 Pics Injection– 52 Segs / 52.355 Pics Suturing– 92 Segs / 321.851 PicsSuction & Irrigation – 173 Segs / 73.977 Pics 1.105 segments (823.000 frames) 9h annotated video of 111 interventions 10-fold cross-validation Petscharnig, S., & Schöffmann, K. (2017). Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications, 1-19.

39. Deep Learning Surgical Actions ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 39 R...Recall P...Precision Petscharnig, S., & Schöffmann, K. (2017). Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications, 1-19.

40. • Early fusion • Integrate motion information from consecutive frames • Fded into CNN as additional input channel(s) • Compare two approaches • Block-Based Motion Estimation (BBME): using block matching • Residual Motion (ResM): local motion Fusing Temporal Information with CNNs ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 40

41. Early Fusion ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 41 HD video CNN input Stefan Petscharnig, Klaus Schöffmann, Jenny Benois-Pineau, Souad Chaabouni and Jörg Keckstein. 2018. Early and Late Fusion of Temporal Information for Classification of Surgical Actions in Laparoscopic Gynecology. In Proceedings of the 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. IEEE, Los Alamitos, CA, USA, 6 pages, 369-374.

42. Early Fusion - BBME ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 42 Create blocks of SxS pixels (e.g., 16x16) Extract motion vectors (dx, dy) from blocks and smooth them (d’x, d’y) HD video CNN input Mean shift and normalization to [0..255] ! = max min !( 2* ∗ 128 + 128, 255 , 0 (R,G,B) Stefan Petscharnig, Klaus Schöffmann, Jenny Benois-Pineau, Souad Chaabouni and Jörg Keckstein. 2018. Early and Late Fusion of Temporal Information for Classification of Surgical Actions in Laparoscopic Gynecology. In Proceedings of the 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. IEEE, Los Alamitos, CA, USA, 6 pages, 369-374.

43. Early Fusion – Residual Motion ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 43 compute length (E) and normalize it with maximum Estimate global motion for each pixel. Find perspective transform (global motion) from frame i-1 to i (using RANSAC) and subtract it à local/residual motion remains (E) HD video CNN input (R,G,B)

44. Early Fusion Results ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 44 RGB BBME as YCbCr Cblue=vertical Cred=horizontal ResM Stefan Petscharnig, Klaus Schöffmann, Jenny Benois-Pineau, Souad Chaabouni and Jörg Keckstein. 2018. Early and Late Fusion of Temporal Information for Classification of Surgical Actions in Laparoscopic Gynecology. In Proceedings of the 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. IEEE, Los Alamitos, CA, USA, 6 pages, 369-374.

45. • Preliminary study showed that residual motion works much better when using full-resolution HD videos as input Early Fusion – Residual Motion ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 45

46. • Early fusion • Integrate motion information from consecutive frames • Feed into CNN as additional input channel(s) • Compare two approaches • Block-Based Motion Estimation (BBME): using block matching • Residual Motion (ResM): local motion • Late fusion • Assume we already know scene boundaries and classify all frames of segments • Temporal aggregation of single-frame classifications • Majority vote (maximum occurrence of class in frames of scene) • Average confidence Fusing Temporal Information with CNNs ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 46

47. Evaluation Results – Early Fusion ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 47 worse better slightly better better baseline

48. Evaluation Results – Late Fusion ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 48 clearly better clearly better baseline

49. Evaluation Results – Early + Late Fusion ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 49 best result baseline

50. • Fusion of temporal information with common CNNs can clearly improve performance of surgical action classification (in Gynecologic Laparoscopy) • Observations • Additional temporal information clearly improves classification performance • Early fusion • BBME does not realy help • ResM improves performance (5% and 9% boost of Rec and Prec – GoogLeNet) • Late fusion • Works well with both tested nets (13% and 25% boost of Rec and Prec – GoogLeNet; AlexNet even more) • Averaging scheme outperforms majority vote • Combination of early & late fusion achieves best result • 17% and 33% boost of Rec and Prec for GoogLeNet • Further work • Evaluate with additional/deeper CNN architectures (deeper InceptionNet, ResNet) • Evaluate patch-based approach Surgical Action Classification – Summary ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 50

51. Deep Learning Surgical Actions ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 51 Confidence Thresholdslow high Stefan Petscharnig and Klaus Schoeffmann. 2018. ActionVis: An Explorative Tool to Visualize Surgical Actions in Gynecologic Laparoscopy. In Proceedings of the 24th International Conference on Multimedia Modeling 2018 (MMM2018). Lecture Notes in Computer Science, vol 10705, Springer, Cham, 348-351.

52. Medical Videos Dataset ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 57

53. LapGyn4: 4-part Laparoscopic Gynecology Dataset ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 58 Surgical Actions (~31K images) Anatomical Structures (~3K images) Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362. Instrument Count (~22K images) Suturing on Anatomy (~1K images) • Over 57,000 images • 500+ surgeries • Baseline Evaluations: GoogleNet • 5-fold cross validation over 100 epochs

54. LapGyn4: Surgical Actions ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 59 Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362. • Why surgical actions? • Key points in surgery • Relevant for post-surgical analyses • Classification (WAvg.) • Very high performance • 97% accuracy • 92% recall • Best in recognizing suturing 31,000+ images

55. LapGyn4: Anatomical Structures ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 60 3,000+ images • Why anatomical structures? • Main subjects to treatments • Relevant for post surgical analyses/AR tracking • Classification (WAvg.) • Very high performance • 95% accuracy • 91% recall • Best results across metrics • min. value: 0.8 Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362.

56. LapGyn4: Instrument Count ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 61 Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362. 22,000+ images • Why counting instruments? • Indicates action/inspection • Facilitates identifying surgical phases • Classification (WAvg.) • Good performance • 92% accuracy • 84% recall • Best in recognizing zero instruments

57. LapGyn4: Suturing on Anatomy ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 62 1,000+ images • Why actions on anatomy? • Key points in surgery • Likely relevant in post-surgical analyses • Classification (WAvg.) • Comparatively poor performance • 80% accuracy • 62% recall • Visual context to similar? • Not enough samples Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362.

58. Medical Video Interaction Tools ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 63

59. Past/Current Status ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 64 Patient names File Explorers & Segments to Download 2014 2009

60. Desired Status ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 65 Bernd Münzer, Klaus Schoeffmann and Laszlo Boeszoermenyi. “EndoXplore: A Web-based Video Explorer for Endoscopic Videos“. Proceedings of the IEEE International Symposium on Multimedia 2017 (ISM 2017), Taipei, Taiwan, 2017, pp. 1-2

61. Special Content Visualization ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 66

62. • Clinicians check full video recordings for occurrence of technical errors: • Errors are rated according to standardized schemes (e.g., OSATS, GERT) and surgeons are made aware of them • Studies have shown that this significantly improves surgical quality Surgical Quality Assessment (SQA) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 67

63. Surgical Quality Assessment (SQA) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 68

64. Surgical Quality Assessment (SQA) Software ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 69 • Integrating rating features • More efficient video navigation/browsing Marco A. Hudelist, Heinrich Husslein, Bernd Muenzer, Sabrina Kletz and Klaus Schoeffmann. “A Tool to Support Surgical Quality Assessment“, in Proceedings of the Third IEEE International Conference on Multimedia Big Data (BigMM), Laguna Hills, CA, USA, 2017, pp. 238-239.

65. Diagnostic Decision Support ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 71

66. Challenges and Requirements ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 72

67. There is a Need for Complete Systems! Medical knowledge transfer Automated analysis / detection / classification Feedback / visualization & administrative ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 73

68. • Medical knowledge transfers – need DATA w/Ground Truth • High detection accuracy • Fast and efficient: real-time feedback and large scale • Fit the normal examination procedures • Assist administrative and report writing work • Adhere to ethical, legal, privacy challenges & regulations Key Challenges & Requirements Multimedia Medicine ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 74

69. Gastrointestinal (GI) Case Study (challenges, system support, datasets, diagnostic decision support, ...) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 75

70. • Many types of diseases can potentially affect the human gastrointestinal (GI) tract – the digestive system • about 2.8 millions of new luminal GI cancers (esophagus, stomach, colorectal) are detected yearly • the mortality is about 65% • Screening of the GI tract using different types of endoscopy… • is costly (colonoscopy according to NY Times: $1100/patient, $10 billion dollars) • consumes valuable medical personnel time (1-2 hours) • does not scale to large populations • is intrusive to the patient • … • Current technology may potentially enable automatic algorithmic screening and assisted examinations à a true interdisciplinary activity with high chances of societal impact GI Tract Challenges and Potential ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 76

71. Colorectal Cancer Women Men Colorectal cancer is the third most common cause of cancer mortality for both women and men, and it is a condition where early detection is important for survival, i.e., a 5-year survival probability of going from a low 10-30% if detected in later stages to a high 90% survival probability in early stages. Colonoscopy is not the ideal screening test. Related to the cancer example, on average 20% of polyps (possible predecessors of cancer) are missed or incompletely removed. The risk of getting cancer largely depend on the endoscopists ability to detect and remove polyps. A 1% increase in detection can decrease the risk of cancer with 3%. The EU recommends screening of everyone above 50 years old. For example, Norway is now implementing a national screening program ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 77

72. • A polyp is an abnormal growth of tissue attached to the underlying mucosa • Detection accuracy depends on experience and skills • average miss rates of approx. 20% • large inter- and intra-variations (e.g., a norwegian study shows variations between 36-65% for polyps) • should reach a high (>85%) accuracy threshold to be acceptable • Current technology may potentially enable automated algorithmic assisted examinations • Introduce a digital “third eye” (with high accuracy and real-time processing) Standard endoscopy: Live Polyp Detection ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 78

73. Video Capsule (PillCam) § Standard colonoscopy: § expensive § does not scale § intrusive ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 79

74. Video Capsule (PillCam) § Standard colonoscopy: § expensive § does not scale § intrusive § Wireless Video Capsule endoscopy: § better scale § less intrusive § possible to combine examinations!? § watch hours of video § less expensive? (detection might lead to an endoscopy) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 80

75. A complete System ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 81

76. System Overview ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 82

77. Medical Knowledge Transfer (Data Collection) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 83

78. • Need more data and therefore tools to efficiently annotate and tag data Available GI Datasets Name Contain Annotation Size Type Usage CVC-ClinicDB Polyps GT masks 12000 images (several versions) Trad. ©, by permission ETIS-Larib Polyp DB Polyps, Normal GT masks 1500 images Trad. ©, by permission ASU-Mayo Clinic DB Polyps, Normal GT masks 20 videos Trad. ©, by permission Colonoscopy Videos DB Various Lesions Sorted 76 videos Trad. Academic Capsule Endoscopy DB Various Lesions and Findings Sorted 3170 images, 47 videos VCE Academic, by request GastroAtlas Various Lesions and Findings Sorted, Text annotations 4449 videos Trad. Academic WEO Atlas Various Lesions and Findings Sorted, Text annotations ? Trad. Academic GASTROLAB Various Lesions and Findings Sorted, Text annotations ? Trad. Academic Atlas of GE Various Lesions Sorted, Text annotations 669 images Trad. ©, by permission KID Various Lesions Sorted 2500 + 47 videos Trad. ©, by permission ASU-Mayo dataset: POLYPS • 20 videos • 10 with polyps, 10 without • 8-64 seconds long • varying resolution • ~18.000 frames/images • image mask of polyp (ground truth) • (currently) restricted use • Datasets are rare • Often small amount of data • Often missing annotation • Few are truly publicly available and accessible • Quality/resolution is bad • Imbalanced • Not enough “Normality” • … CVC: POLYPS • CVC-356 – 356 polyp images, 1350 normal frames • CVC-612 – 612 polyp images, 1350 normal frames • CVC-968 – 968 polyp images, 1350 normal frames • CVC-12K – 10025 polyp images, 1929 normal frames • image mask of polyp (ground truth) • (currently) restricted use ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 84

79. • Which image is not from the same class? … and it gets worse … • Making a mistake between cats and dogs may not matter, but a misclassification here may have lethal consequences Why Can’t CS People Do the Annotation!? PylorusZ-line Z-line Z-line Z-line Z-line ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 85

80. Available time of the clinicians? ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 86

81. • Simple and efficient • Web-based • Assisted object tracking Video Annotation Subsystem "Expert Driven Semi-Supervised Elucidation Tool for Medical Endoscopic Videos" Zeno Albisser, et. al. Proceedings of MMSys, Portland, OR, USA, March 2015 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 87

82. • For large collection of images • VV / Kvasir dataset • Fully cleaned • Feature extraction mechanisms • Different unsupervised clustering algorithms • Hierarchical image collection visualization • Open source: ClusterTag https://bitbucket.org/mpg_projects/clustertag ClusterTag: Image Clustering and Tagging Tool "ClusterTag: Interactive Visualization, Clustering and Tagging Tool for Big Image Collections" Konstantin Pogorelov, et. al. Proceedings of ICMR, Bucharest, Romania, June 2017 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 88

83. • Multi-Class Image Dataset for Computer Aided GI Disease Detection • GI endoscopy images • Some images contain the position and configuration of the endoscope (scope guide) • 8 different anomalies and anatomical landmarks • v1: 500 images per class, 6 pre-extracted global features • v2: 1000 images per class • New information added in the future: http://datasets.simula.no/kvasir/ The Kvasir Dataset "Kvasir: A Multi-Class Image-Dataset for Computer Aided Gastrointestinal Disease Detection" Konstantin Pogorelov, et al. Proceedings of MMSYS, Taiwan, June 2017 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 89

84. • Bowel Preparation Quality Video • 21 GI endoscopy videos of colon • Some frames contain the position and configuration of the endoscope (scope guide) • 4 classes showing four-score BBPS- defined bowel-preparation quality • 0 - very dirty • … • 3 - very clean • http://datasets.simula.no/nerthus/ The Nerthus Dataset "Nerthus: A Bowel Preparation Quality Video Dataset" Konstantin Pogorelov, et al. Proceedings of MMSYS, Taiwan, June 2017 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 90

85. • Still need even more efficient tools and data of entire procedures 1. “Annotation” during examination 2. Video with bookmarks 3. Annotate bookmarks 4. Automatically annotate neighboring frames using object tracking – and verify Next version of the annotation tool ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 91

86. GI Anomaly Detection System ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 92

87. • Common approaches • Handcrafted features • Convolutional neural network • Generative Adversarial Networks • Easy to extend with new diseases • Easy to extend with new algorithms • Easy to train • Results are explainable? • Disease Localization? • Real-time? Requirements Detection and Automatic Analysis subsystem ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 93

88. Performance (accuracy and speed) ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 94

89. § Mayo dataset (18781 images/frames) § masks for all polyps • GF: • recall 98.50%, precision 93.88%, fps ~300 • CNN: • Modified Inception v3: recall 95.86%, precision 80.78%, fps: ~30 • Inception v3 + WEKA: recall: 88.87%, precision: 89.16%, fps: ~30 ASU Mayo Dataset: Polyp Detection ”EIR - Efficient Computer Aided Diagnosis Framework for Gastrointestinal Endoscopies" Michael Riegler, et. al. Proceedings of CBMI, Bucharest, Romania, June 2016 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 95

90. • Resource consumption and processing performance of GF: • Neural networks (also including GPU support)? • tests so far: ~30 fps (same GPU as above) • but adding layers, more networks, … !?? (newer GPU) • Inception v3 TFL: 66 fps, plain CNN: ~40-45 fps • GAN: ~12 fps (for 160x160) ASU Mayo Dataset: Polyp Detection ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 96

91. • Process only frames containing polyps • Performs image enhancement • Detects curve-shaped objects and local maximums • Builds energy map and selects 4 possible locations • Localization performance: • recall 31.83 %, • precision 32.07% • ~30 fps • later better GPU: ~75 fps (detection: 300 fps ; localization 100 fps) ASU Mayo Dataset: First Try for Polyp Localization ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 97

92. • Vestre Viken (VV) multi-disease dataset (250 images per class) • GF: • recall 90.60 % • precision 91.40% • fps ~30 • CNN: • recall: 87.20% • precision: 87.90% • fps: ~30 VV Dataset: Multi-Disease Detection ""Efficient disease detection in gastrointestinal videos - global features versus neural networks" Konstantin Pogorelov, et. al. Multimedia Tools and Applications, 2017 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 98

93. • GF • CNN VV Dataset: Multi-Disease Detection ""Efficient disease detection in gastrointestinal videos - global features versus neural networks" Konstantin Pogorelov, et. al. Multimedia Tools and Applications, 2017 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 99

94. • 7 different algorithms • Convolutional neural networks (CNN) (2) – trained from scratch • 3-layers • 6-layers • Transfer learning (1) – retrained Inception v3 • Global features (4) • 2 global features (JCD, Tamura) • 6 global features (JCD, Tamura, Color Layout, Edge Histogram, Auto Color Correlogram and PHOG) • 2 different algorithms (Random forest and logistic model tree) • 2 baselines • Random Forrest with one global feature • Majority class • 2-folded cross validation Kvasir Dataset v1: Multi-Disease Detection ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 100

95. Kvasir Dataset v1: Multi-Disease Detection ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 101

96. Kvasir Dataset v1: Multi-Disease Detection DyedandLiftedPolypDyedResectionMargin ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 102

97. Kvasir Dataset v1: Multi-Disease Detection CecumPylorus ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 103

98. • Using same GF and some new deep features, i.e., • Pre-trained ImageNet dataset Inception v3 • ResNet50 models • Used different ML classifications; • random tree (RT) • random forest (RF) • logistic model tree (LMR) – performed best • Uses weights of 1000 pre-defined concepts as features • Top layer input as features vector (16384 for Inception v3 and 2048 for ResNet50) Kvasir Dataset v1 à v2: Multi-Disease Detection Pretrained model Output or top- layer input weights WEKA for classification Team Approaches F1 FPS SCL-UMD Global-features and deep-features extraction, Inception-V3 and VGGNet CNN models, followed by machine-learning-based classification using RT, RF, SVM and LMR classifiers 0.848 1.3 FAST-NU-DS Global and local features combined followed by data size reduction by applying K-means clustering and than using logistic regression model for the classification 0.767 2.3 ITEC-AAU Two different custom Inception-like CNN models 0.755 1.4 HKBU A manifold learning method (bidirectional marginal Fisher analysis) learning a compact representation of the data, then machine-learning-based multi-class support vector machine is used for the classification 0.703 2.2 SIMULA GF-features extraction, ResNet50 and Inception-V3 CNN models and followed by machine-learning-based classification using RT, RF and LMR classifiers 0.826 46.0 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 104

99. • 16 classes of anomalies and landmarks • Very varying dataset sizes for the different classes • Combination of retrained networks Kvasir Dataset v2 à v3: Multi-Disease Detection ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 105

100. Compared: • Handcrafted global features (GF-D) sing LIRE • Retrained and fine tuned existing DL architectures (RT-D) • Generative adversarial network (GAN) • Combined various datasets captured by different equipment in different hospitals. • With our best working GAN-based approach, we reached detection specificity of 94% and accuracy of 90.9% with only 356 training and 6,000 test samples • The localization specificity and accuracy for the same training set are 98.4% and 94.6% respectively. The Next Level: Comparing Handcrafted and Deep Learning Features – Cross Datasets ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 106

101. • 7 different algorithms • Convolutional neural networks (CNN) (2) – trained from scratch • 3-layers • 6-layers • Transfer learning (1) – retrained Inception v3 • Global features (4) • 2 global features (JCD, Tamura) • 6 global features (JCD, Tamura, Color Layout, Edge Histogram, Auto Color Correlogram and PHOG) • 2 different algorithms (Random forest and logistic model tree) • 2 baselines • Random Forrest with one global feature • Majority class • 2-folded cross validation Nerthus Dataset: Bowel Cleanness Level ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 107

102. Nerthus Dataset: Bowel Cleanness Level ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 108

103. Nerthus Dataset: Bowel Cleanness Level ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 109

104. Preprocessing ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 111

105. • Too little data • Blurry images due to camera motion • Objects too close to camera • Under or over scene lighting • Flares • Artificial objects and natural “contaminations” • Low resolution of capsular endoscopes • … Data Challenges: Preprocessing ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 112

106. Data Enhancements for CNN Training ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 113

107. Data Enhancements for CNN Training ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 114

108. • Artifacts in the images can influence the algorithm • Understanding of what the algorithm reacts to is crucial Borders and Overlays ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 115

109. • Results on Kvasir + CVC-986 • Accuracy improved for almost all models with some preprocessing Borders and Overlays ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 116

110. • Replacing artifacts in the video/image • Different methods • From simple to more advanced • Some difference but marginal • Future: Does it have to look good? • Support the algorithm not the human perception • Different from usual GAN use GAN inpainting of Navigation Box ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 117

111. Automatic Detection of Angiectasia Video Capsule Endoscopy ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 118

112. • Angiectasia is a vascular lesions that can cause of GI bleedings • Medical specialists reach a detection accuracy of about 69% • Medical systems should reach an 85% threshold to be acceptable in clinical use Angiectasia Detection ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 119

113. Angiectasia Detection: Varying Difficulty ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 120

114. • By far, GANs give the best detection: • sensitivity: 98% • specificity: 100% • BUT, sloooooow… • Several approaches are better than the average doctor (69%) • Most of the approaches have a too low detection rate, but still better than the baseline • Compromise between accuracy and speed Detection Compared ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 121

115. Detection Feedback ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 122

116. Detection Subsystem Outputs • Visualize the output of the system to the medical doctors • Simple and easy to understand (most important) • Easy to integrate in hospitals • Live support • Useable for automatic reports, etc. ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 123

117. • Polyps • Input: Camera or Video files • Output: Live stream and Performance reports • Full HD • Real-time: 30 FPS Real-time Detection Feedback ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 124

118. Increasing Understanding & Assisting Administrative Work ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 125

119. Reporting of Endoscopies Importance of Endoscopy Reporting • Critical in communication between patients and healthcare providers. • Sometimes only evidence of the performed procedure. • Important in measuring the quality of endoscopies. Current State Considered Poor • Inconsistent descriptions of abnormalities • Poor adoption of existing standards • Time consuming (up to 15 minutes or more) • Boring and lessens job satisfaction ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 126

120. Mimir: Reporting of Endoscopies Goals of Mimir • Give an easy to understand way of interpreting the output of a neural network. • Allows for deeper analysis of why the model produces a given result. • Class discriminatory visualizations based on selected class and layer. • Tools for uploading and managing various models. • Provide a tool for the automatic generation of modifiable medical reports. • Produced Visualizations • Visualizations produced using the grad-CAM technique. • Takes key attributes from saliency and class activation maps. ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 127

123. ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 130

124. ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 131

125. So, all problems solved!!?? ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 132

126. • Improve detection, localization and system performance (retrieval, machine learning, features, search, real-time, distributed computing, scale, visualization, neural networks, user interaction, object tracking, …) 1. Exploiting domain expert knowledge – build datasets 2. Integration of various data, multi-modality – new sensors 3. Explainable AI 4. Automated report system 5. Full system integration 6. Patient context information 7. Visualization, decision support 8. Integration of data from various sources / systems 9. Other areas in medicine 10. … Many more… Many Open Challenges… "Multimedia and Medicine: Teammates for Better Disease Detection and Survival" Michael Riegler, et. al. Proceedings ACM MM, Amsterdam, The Netherlands, October 2016 ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 133

127. • We have given several case-specific examples, but in general, they are common for MMIS • Doctors want to use all the data for general support: analysis, diagnostics, reporting, teaching, statistics, similarity search / comparisons, … • Currently, … • more and more high quality data is recorded / produced • data analysis methods are (only) promising • multi modal data analysis is not very common • good visualization tools exist, but not used (e.g., AR, VR, …) • some tools are missing • many (other) areas produce separate (isolated) methods • … • but, we need a complete integrated system! Ø Our multimedia community is needed Summary ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 134

128. The End… Multimedia Medicine ACM Multimedia 2018 Tutorial The Importance of Medical Multimedia 135

The Importance of Medical Multimedia

The Importance of Medical Multimedia

Recommended

More Related Content

What's hot

What's hot(12)

Similar to The Importance of Medical Multimedia

Similar to The Importance of Medical Multimedia(20)

More from klschoef

More from klschoef(6)

Recently uploaded

Recently uploaded(20)

The Importance of Medical Multimedia