SlideShare a Scribd company logo
1 of 14
Download to read offline
Detecting Semantics in Endoscopic
Videos with Deep Neural Networks
Klaus Schöffmann
Institut für Informationstechnologie
Klagenfurt University, Austria
Jörg Keckstein
Medizinische Fakultät
Ulm University, Germany
• Grant/Research Support
• European Regional Development Fund (ERDF) – 20214
• Carinthian Economic Promotion Fund (KWF) and Lakeside Labs
3520/26336/38165
Disclosures
Klaus Schöffmann, Klagenfurt University EEC2018 2
• Currently, there are many ways of delivering video signals to the
operation room, in order to perform a surgery or inspection
• Endoscopes in different endoscopic surgeries
• Capsules for minimally-invasive inspections of the GI-tract
• Microscopes with mounted cameras in case of microscopic surgery
• Video streams for robot-assisted surgery
Multitude of Videos in Medicine
Klaus Schöffmann, Klagenfurt University EEC2018 3
Video Streams/Recordings – added value?
Klaus Schöffmann, Klagenfurt University EEC2018 4
Post-surgical use
• Documentation
• Education and training
• Treatment planning
• Case re-visitation
• Legal …
Problems
• Manual analysis
time-consuming/tedious
• Multiple hours-long
surgeries per day:
huge video archives
What is needed
• Capable systems
aiding surgeons during
or post surgery
• Finding content of
interest and making
archives manageable
How to proceed
• Automatic content
analysis: detecting
relevant scenes
• Machine learning of
relevant scenes from
many examples
Patient
names Suturing
Cutting
Injection
Coagulation?
? ?
? ?
What semantics may we want to find?
Klaus Schöffmann, Klagenfurt University EEC2018 5
?
? ?
? ?
Image/Video Archive
Instruments
Anatomy
Pathology
Surgical Actions
Irrelevant Scenes
Technical Errors
Surgery Phases
1. Access
2. Dissection
3. Clipping
4. Cutting
5. Separation
Recognizing Surgical Actions in Laparoscopic Gynecology
Klaus Schöffmann, Klagenfurt University EEC2018 6
Dissection (blunt) Coagulation Cutting (cold)
Cutting (high frequency) Sling (Hysterectomy) Injection
SuturingSuction & Irrigation
Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz,
Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for
4 automatic content analysis problems in the domain of laparoscopic gynecology. In
Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM,
New York, NY, USA, 357-362.
31.000 images
111 surgeries
Demo: Surgical Action Recognition
Klaus Schöffmann, Klagenfurt University EEC2018 7
Petscharnig, S., & Schöffmann, K. (2017). Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications, 1-19.
Quantitative Evaluation: LapGyn4 (4-part Dataset)
Klaus Schöffmann, Klagenfurt University EEC2018 8
Surgical Actions (~31K samples) Anatomical Structures (~3K samples)
Very high performance using modern
machine learning standard:
• >95% recognition accuracy
• >91% recall
Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic
gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362.
Quantitative Evaluation: LapGyn4 (4-part Dataset)
Klaus Schöffmann, Klagenfurt University EEC2018 9
Suturing on Anatomy (~1K samples)Instrument Count (~22K samples)
Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic
gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362.
Comparatively poor performance:
• 80% accuracy
• 62% recall
• Not enough samples (~1.000 images)
• Visual context perhaps too similar
Still good performance:
• >92% accuracy
• >84% recall
• best in classifying zero instruments (acc: 96%)
Towards Endometriosis Recognition – former approach
Klaus Schöffmann, Klagenfurt University EEC2018 10
• Tool for supporting experts in
annotating Endometriosis
• Annotating affected regions
via ENZIAN/rASRM score
• Problems
• Too many examples
needed for every single
ENZIAN/rASRM category
required for machine learning
• Too much effort for few expert
annotators
A. Leibetseder, B. Münzer, K. Schoeffmann and J. Keckstein, "Endometriosis Annotation in Endoscopic Videos," 2017 IEEE International Symposium on Multimedia (ISM), Taichung, 2017, pp. 364-365.
Future work: Region-based Endometriosis Detection
Klaus Schöffmann, Klagenfurt University EEC2018 11
Endometriosis/Adhesion Database (expert annotations)
• Purpose: supporting surgeons
by suggesting suspicious regions
• Goals: localized detection using 3 categories of diseases:
adhesions, endometriosis suspicion, endometriosis
• Methodology
• Creating endometriosis/adhesion database
• Training and evaluating suitable machine learning models for classification
Machine
Learning
(e.g. R-CNN)
No Endometriosis
Adhesions
Endometriosis
Prediction
Model Endometriosis Suspicion
Automatic Smoke Detection (Real-Time Support)
Klaus Schöffmann, Klagenfurt University EEC2018 12
Detection Runtime
+ Machine Learning (acc: 94%)
– Saturation Analysis (acc: 81%)
+ Saturation Analysis (12ms)
– Machine Learning (150ms)
Andreas Leibetseder, Manfred J. Primus, Stefan Petscharnig, and Klaus Schoeffmann. “Image-based Smoke Detection in Laparoscopic Videos“.
Proceedings of Computer Assisted and Robotic Endoscopy and Clinical Image-Based Procedures: 4th International Workshop, CARE 2017, and
6th International Workshop, CLIP 2017, held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 14, 2017, pp. 70-87
Leibetseder A., Primus M.J., Schoeffmann K. (2018) Automatic Smoke Classification
in Endoscopic Video. In: Schoeffmann K. et al. (eds) MultiMedia Modeling. MMM
2018. Lecture Notes in Computer Science, vol 10705. Springer, Cham
• High potential in full video documentation
• Enables and aids additional usage scenarios
• Post-operative and real-time support
• Facilitating education and training
• Technical skill assessment
• Improving case re-visitations
• Aiding medical staff during interventions
• Needs a system to organize and analyze videos
• To detect important semantics/scenes
• Machine learning can help a lot with this, but also it requires:
• Many and diverse training examples
• Computational power
• Experts for creating correct training examples
Conclusions
Klaus Schöffmann, Klagenfurt University EEC2018 13
Klaus Schöffmann, Klagenfurt University EEC2018 14
Thank You!
Q/A
Assoc.-Prof. PD Dr. Klaus Schöffmann
Institut für Informationstechnologie
Alpen-Adria-Universität Klagenfurt
www.EndoscopicVideo.com

More Related Content

What's hot

FoCAS Newsletter Issue 1: Septemeber 2013
FoCAS Newsletter Issue 1: Septemeber 2013FoCAS Newsletter Issue 1: Septemeber 2013
FoCAS Newsletter Issue 1: Septemeber 2013
FoCAS Initiative
 
Presentation
PresentationPresentation
Presentation
Videoguy
 

What's hot (13)

Medical image analysis and big data evaluation infrastructures
Medical image analysis and big data evaluation infrastructuresMedical image analysis and big data evaluation infrastructures
Medical image analysis and big data evaluation infrastructures
 
BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis
BreastScreening: On the Use of Multi-Modality in Medical Imaging DiagnosisBreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis
BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis
 
Telepathology
TelepathologyTelepathology
Telepathology
 
Digital pathology
Digital pathologyDigital pathology
Digital pathology
 
Medical Image Analysis and Its Application
Medical Image Analysis and Its ApplicationMedical Image Analysis and Its Application
Medical Image Analysis and Its Application
 
Hand Pose Tracking for Clinical Applications
Hand Pose Tracking for Clinical ApplicationsHand Pose Tracking for Clinical Applications
Hand Pose Tracking for Clinical Applications
 
FoCAS Newsletter Issue 1: Septemeber 2013
FoCAS Newsletter Issue 1: Septemeber 2013FoCAS Newsletter Issue 1: Septemeber 2013
FoCAS Newsletter Issue 1: Septemeber 2013
 
Cloud Services for Education - HNSciCloud applied to the UP2U project
Cloud Services for Education - HNSciCloud applied to the UP2U projectCloud Services for Education - HNSciCloud applied to the UP2U project
Cloud Services for Education - HNSciCloud applied to the UP2U project
 
Nessos
NessosNessos
Nessos
 
Presentation
PresentationPresentation
Presentation
 
Dcb1419 f1 ultrasound
Dcb1419   f1 ultrasoundDcb1419   f1 ultrasound
Dcb1419 f1 ultrasound
 
140123 Workshop bioprinting Sirris
140123 Workshop bioprinting Sirris140123 Workshop bioprinting Sirris
140123 Workshop bioprinting Sirris
 
thesis
thesisthesis
thesis
 

Similar to Detecting Semantics in Endoscopic Videos with Deep Neural Networks

LuCIFEx: Lung Cancer Identification through Feature Extraction
LuCIFEx: Lung Cancer Identification through Feature ExtractionLuCIFEx: Lung Cancer Identification through Feature Extraction
LuCIFEx: Lung Cancer Identification through Feature Extraction
NECST Lab @ Politecnico di Milano
 
MYREN and Telemedicine
MYREN and TelemedicineMYREN and Telemedicine
MYREN and Telemedicine
Salem Omar
 
Utilization of virtual microscopy in a cooperative group setting
Utilization of virtual microscopy in a cooperative group settingUtilization of virtual microscopy in a cooperative group setting
Utilization of virtual microscopy in a cooperative group setting
BIT002
 
MicroscopyCongressAgenda
MicroscopyCongressAgendaMicroscopyCongressAgenda
MicroscopyCongressAgenda
Jon Clarke
 
Monte Carlo And Ct Interface For Medical Treatment Plans
Monte Carlo And Ct Interface For Medical Treatment PlansMonte Carlo And Ct Interface For Medical Treatment Plans
Monte Carlo And Ct Interface For Medical Treatment Plans
fondas vakalis
 

Similar to Detecting Semantics in Endoscopic Videos with Deep Neural Networks (20)

AI-powered Medical Imaging Analysis for Precision Medicine
AI-powered Medical Imaging Analysis for Precision MedicineAI-powered Medical Imaging Analysis for Precision Medicine
AI-powered Medical Imaging Analysis for Precision Medicine
 
The Evolution and Impact of Medical Science Journals in Advancing Healthcare
The Evolution and Impact of Medical Science Journals in Advancing HealthcareThe Evolution and Impact of Medical Science Journals in Advancing Healthcare
The Evolution and Impact of Medical Science Journals in Advancing Healthcare
 
journals on medical
journals on medicaljournals on medical
journals on medical
 
LuCIFEx: Lung Cancer Identification through Feature Extraction
LuCIFEx: Lung Cancer Identification through Feature ExtractionLuCIFEx: Lung Cancer Identification through Feature Extraction
LuCIFEx: Lung Cancer Identification through Feature Extraction
 
H2O World - Machine Learning to Save Lives - Taposh Dutta Roy
H2O World - Machine Learning to Save Lives - Taposh Dutta RoyH2O World - Machine Learning to Save Lives - Taposh Dutta Roy
H2O World - Machine Learning to Save Lives - Taposh Dutta Roy
 
MyCT 3rd Edition e-Newsletter
MyCT 3rd Edition e-NewsletterMyCT 3rd Edition e-Newsletter
MyCT 3rd Edition e-Newsletter
 
Medical Multimedia Information Systems (ACMMM17 Tutorial)
Medical Multimedia Information Systems (ACMMM17 Tutorial) Medical Multimedia Information Systems (ACMMM17 Tutorial)
Medical Multimedia Information Systems (ACMMM17 Tutorial)
 
MYREN and Telemedicine
MYREN and TelemedicineMYREN and Telemedicine
MYREN and Telemedicine
 
Digital Pathology streamlines tissue diagnostics and helps protect patient sa...
Digital Pathology streamlines tissue diagnostics and helps protect patient sa...Digital Pathology streamlines tissue diagnostics and helps protect patient sa...
Digital Pathology streamlines tissue diagnostics and helps protect patient sa...
 
Utilization of virtual microscopy in a cooperative group setting
Utilization of virtual microscopy in a cooperative group settingUtilization of virtual microscopy in a cooperative group setting
Utilization of virtual microscopy in a cooperative group setting
 
Telepathology and artificial inteligence in India and beyond
Telepathology and artificial inteligence in India and beyondTelepathology and artificial inteligence in India and beyond
Telepathology and artificial inteligence in India and beyond
 
MicroscopyCongressAgenda
MicroscopyCongressAgendaMicroscopyCongressAgenda
MicroscopyCongressAgenda
 
Multimedia tools propel Web-based telepathology
Multimedia tools propel Web-based telepathologyMultimedia tools propel Web-based telepathology
Multimedia tools propel Web-based telepathology
 
Medical Deep Learning: Clinical, Technical, & Regulatory Challenges and How t...
Medical Deep Learning: Clinical, Technical, & Regulatory Challenges and How t...Medical Deep Learning: Clinical, Technical, & Regulatory Challenges and How t...
Medical Deep Learning: Clinical, Technical, & Regulatory Challenges and How t...
 
TCIA Update
TCIA UpdateTCIA Update
TCIA Update
 
Virkki presentation VTT SmartHealth Ecosystem Event 12.6.2019
Virkki presentation VTT SmartHealth Ecosystem Event 12.6.2019Virkki presentation VTT SmartHealth Ecosystem Event 12.6.2019
Virkki presentation VTT SmartHealth Ecosystem Event 12.6.2019
 
Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...
Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...
Relevant Content Detection in Cataract Surgery Videos (Invited Talk 1 at IPTA...
 
2015-fall
2015-fall2015-fall
2015-fall
 
Monte Carlo And Ct Interface For Medical Treatment Plans
Monte Carlo And Ct Interface For Medical Treatment PlansMonte Carlo And Ct Interface For Medical Treatment Plans
Monte Carlo And Ct Interface For Medical Treatment Plans
 
Curriculum vitae
Curriculum vitaeCurriculum vitae
Curriculum vitae
 

More from klschoef

More from klschoef (7)

Medical Video Processing (Tutorial)
Medical Video Processing (Tutorial)Medical Video Processing (Tutorial)
Medical Video Processing (Tutorial)
 
Video Browser Showdown (VBS) 2012-2019
Video Browser Showdown (VBS) 2012-2019Video Browser Showdown (VBS) 2012-2019
Video Browser Showdown (VBS) 2012-2019
 
Video Browser Showdown 2018 - Results
Video Browser Showdown 2018 - ResultsVideo Browser Showdown 2018 - Results
Video Browser Showdown 2018 - Results
 
Video Browser Showdown 2018
Video Browser Showdown 2018Video Browser Showdown 2018
Video Browser Showdown 2018
 
ICME 2016 - Tutorial on Interactive Search in Video & Lifelog Repositories
ICME 2016 - Tutorial on Interactive Search in Video & Lifelog RepositoriesICME 2016 - Tutorial on Interactive Search in Video & Lifelog Repositories
ICME 2016 - Tutorial on Interactive Search in Video & Lifelog Repositories
 
Interactive Video Search - Tutorial at ACM Multimedia 2015
Interactive Video Search - Tutorial at ACM Multimedia 2015Interactive Video Search - Tutorial at ACM Multimedia 2015
Interactive Video Search - Tutorial at ACM Multimedia 2015
 
Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014)
Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014)Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014)
Video Browsing - The Need for Interactive Video Search (Talk at CBMI 2014)
 

Recently uploaded

Sonia Journal club presentation (2).pptx
Sonia Journal club presentation (2).pptxSonia Journal club presentation (2).pptx
Sonia Journal club presentation (2).pptx
palsonia139
 
CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...
CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...
CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...
ocean4396
 

Recently uploaded (20)

Making Patient-Centric Immunotherapy a Reality in Lung Cancer: Best Practices...
Making Patient-Centric Immunotherapy a Reality in Lung Cancer: Best Practices...Making Patient-Centric Immunotherapy a Reality in Lung Cancer: Best Practices...
Making Patient-Centric Immunotherapy a Reality in Lung Cancer: Best Practices...
 
Premium ℂall Girls In Mira Road👉 Dail ℂALL ME: 📞9004268417 📲 ℂall Richa VIP ℂ...
Premium ℂall Girls In Mira Road👉 Dail ℂALL ME: 📞9004268417 📲 ℂall Richa VIP ℂ...Premium ℂall Girls In Mira Road👉 Dail ℂALL ME: 📞9004268417 📲 ℂall Richa VIP ℂ...
Premium ℂall Girls In Mira Road👉 Dail ℂALL ME: 📞9004268417 📲 ℂall Richa VIP ℂ...
 
TEST BANK For Lewis's Medical Surgical Nursing in Canada, 4th Edition by Jane...
TEST BANK For Lewis's Medical Surgical Nursing in Canada, 4th Edition by Jane...TEST BANK For Lewis's Medical Surgical Nursing in Canada, 4th Edition by Jane...
TEST BANK For Lewis's Medical Surgical Nursing in Canada, 4th Edition by Jane...
 
Dermatome and myotome test & pathology.pdf
Dermatome and myotome test & pathology.pdfDermatome and myotome test & pathology.pdf
Dermatome and myotome test & pathology.pdf
 
5Cladba ADBB 5cladba buy 6cl adbb powder 5cl ADBB precursor materials
5Cladba ADBB 5cladba buy 6cl adbb powder 5cl ADBB precursor materials5Cladba ADBB 5cladba buy 6cl adbb powder 5cl ADBB precursor materials
5Cladba ADBB 5cladba buy 6cl adbb powder 5cl ADBB precursor materials
 
Mgr university bsc nursing adult health previous question paper with answers
Mgr university  bsc nursing adult health previous question paper with answersMgr university  bsc nursing adult health previous question paper with answers
Mgr university bsc nursing adult health previous question paper with answers
 
ANAPHYLAXIS BY DR.SOHAN BISWAS,MBBS,DNB(INTERNAL MEDICINE) RESIDENT.pptx
ANAPHYLAXIS BY DR.SOHAN BISWAS,MBBS,DNB(INTERNAL MEDICINE) RESIDENT.pptxANAPHYLAXIS BY DR.SOHAN BISWAS,MBBS,DNB(INTERNAL MEDICINE) RESIDENT.pptx
ANAPHYLAXIS BY DR.SOHAN BISWAS,MBBS,DNB(INTERNAL MEDICINE) RESIDENT.pptx
 
Let's Talk About It: Ovarian Cancer (The Emotional Toll of Treatment Decision...
Let's Talk About It: Ovarian Cancer (The Emotional Toll of Treatment Decision...Let's Talk About It: Ovarian Cancer (The Emotional Toll of Treatment Decision...
Let's Talk About It: Ovarian Cancer (The Emotional Toll of Treatment Decision...
 
Our Hottest 💘 Surat ℂall Girls Serviℂe 💘Pasodara📱 8527049040📱450+ ℂall Girl C...
Our Hottest 💘 Surat ℂall Girls Serviℂe 💘Pasodara📱 8527049040📱450+ ℂall Girl C...Our Hottest 💘 Surat ℂall Girls Serviℂe 💘Pasodara📱 8527049040📱450+ ℂall Girl C...
Our Hottest 💘 Surat ℂall Girls Serviℂe 💘Pasodara📱 8527049040📱450+ ℂall Girl C...
 
Renal Replacement Therapy in Acute Kidney Injury -time modality -Dr Ayman Se...
Renal Replacement Therapy in Acute Kidney Injury -time  modality -Dr Ayman Se...Renal Replacement Therapy in Acute Kidney Injury -time  modality -Dr Ayman Se...
Renal Replacement Therapy in Acute Kidney Injury -time modality -Dr Ayman Se...
 
TEST BANK For Huether and McCance's Understanding Pathophysiology, Canadian 2...
TEST BANK For Huether and McCance's Understanding Pathophysiology, Canadian 2...TEST BANK For Huether and McCance's Understanding Pathophysiology, Canadian 2...
TEST BANK For Huether and McCance's Understanding Pathophysiology, Canadian 2...
 
Evidence-based practiceEBP) in physiotherapy
Evidence-based practiceEBP) in physiotherapyEvidence-based practiceEBP) in physiotherapy
Evidence-based practiceEBP) in physiotherapy
 
Unlocking Holistic Wellness: Addressing Depression, Mental Well-Being, and St...
Unlocking Holistic Wellness: Addressing Depression, Mental Well-Being, and St...Unlocking Holistic Wellness: Addressing Depression, Mental Well-Being, and St...
Unlocking Holistic Wellness: Addressing Depression, Mental Well-Being, and St...
 
Sonia Journal club presentation (2).pptx
Sonia Journal club presentation (2).pptxSonia Journal club presentation (2).pptx
Sonia Journal club presentation (2).pptx
 
CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...
CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...
CAS 110-63-4 BDO Liquid 1,4-Butanediol 1 4 BDO Warehouse Supply For Excellent...
 
ROSE CASE SPINAL SBRT BY DR KANHU CHARAN PATRO
ROSE  CASE SPINAL SBRT BY DR KANHU CHARAN PATROROSE  CASE SPINAL SBRT BY DR KANHU CHARAN PATRO
ROSE CASE SPINAL SBRT BY DR KANHU CHARAN PATRO
 
Tips and tricks to pass the cardiovascular station for PACES exam
Tips and tricks to pass the cardiovascular station for PACES examTips and tricks to pass the cardiovascular station for PACES exam
Tips and tricks to pass the cardiovascular station for PACES exam
 
CONGENITAL HYPERTROPHIC PYLORIC STENOSIS by Dr M.KARTHIK EMMANUEL
CONGENITAL HYPERTROPHIC PYLORIC STENOSIS  by Dr M.KARTHIK EMMANUELCONGENITAL HYPERTROPHIC PYLORIC STENOSIS  by Dr M.KARTHIK EMMANUEL
CONGENITAL HYPERTROPHIC PYLORIC STENOSIS by Dr M.KARTHIK EMMANUEL
 
VVIP Whitefield ℂall Girls 6350482085 Heat-flaring { Bangalore } Worthy Girl ...
VVIP Whitefield ℂall Girls 6350482085 Heat-flaring { Bangalore } Worthy Girl ...VVIP Whitefield ℂall Girls 6350482085 Heat-flaring { Bangalore } Worthy Girl ...
VVIP Whitefield ℂall Girls 6350482085 Heat-flaring { Bangalore } Worthy Girl ...
 
Cardiovascular Physiology - Regulation of Cardiac Pumping
Cardiovascular Physiology - Regulation of Cardiac PumpingCardiovascular Physiology - Regulation of Cardiac Pumping
Cardiovascular Physiology - Regulation of Cardiac Pumping
 

Detecting Semantics in Endoscopic Videos with Deep Neural Networks

  • 1. Detecting Semantics in Endoscopic Videos with Deep Neural Networks Klaus Schöffmann Institut für Informationstechnologie Klagenfurt University, Austria Jörg Keckstein Medizinische Fakultät Ulm University, Germany
  • 2. • Grant/Research Support • European Regional Development Fund (ERDF) – 20214 • Carinthian Economic Promotion Fund (KWF) and Lakeside Labs 3520/26336/38165 Disclosures Klaus Schöffmann, Klagenfurt University EEC2018 2
  • 3. • Currently, there are many ways of delivering video signals to the operation room, in order to perform a surgery or inspection • Endoscopes in different endoscopic surgeries • Capsules for minimally-invasive inspections of the GI-tract • Microscopes with mounted cameras in case of microscopic surgery • Video streams for robot-assisted surgery Multitude of Videos in Medicine Klaus Schöffmann, Klagenfurt University EEC2018 3
  • 4. Video Streams/Recordings – added value? Klaus Schöffmann, Klagenfurt University EEC2018 4 Post-surgical use • Documentation • Education and training • Treatment planning • Case re-visitation • Legal … Problems • Manual analysis time-consuming/tedious • Multiple hours-long surgeries per day: huge video archives What is needed • Capable systems aiding surgeons during or post surgery • Finding content of interest and making archives manageable How to proceed • Automatic content analysis: detecting relevant scenes • Machine learning of relevant scenes from many examples Patient names Suturing Cutting Injection Coagulation? ? ? ? ?
  • 5. What semantics may we want to find? Klaus Schöffmann, Klagenfurt University EEC2018 5 ? ? ? ? ? Image/Video Archive Instruments Anatomy Pathology Surgical Actions Irrelevant Scenes Technical Errors Surgery Phases 1. Access 2. Dissection 3. Clipping 4. Cutting 5. Separation
  • 6. Recognizing Surgical Actions in Laparoscopic Gynecology Klaus Schöffmann, Klagenfurt University EEC2018 6 Dissection (blunt) Coagulation Cutting (cold) Cutting (high frequency) Sling (Hysterectomy) Injection SuturingSuction & Irrigation Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362. 31.000 images 111 surgeries
  • 7. Demo: Surgical Action Recognition Klaus Schöffmann, Klagenfurt University EEC2018 7 Petscharnig, S., & Schöffmann, K. (2017). Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications, 1-19.
  • 8. Quantitative Evaluation: LapGyn4 (4-part Dataset) Klaus Schöffmann, Klagenfurt University EEC2018 8 Surgical Actions (~31K samples) Anatomical Structures (~3K samples) Very high performance using modern machine learning standard: • >95% recognition accuracy • >91% recall Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362.
  • 9. Quantitative Evaluation: LapGyn4 (4-part Dataset) Klaus Schöffmann, Klagenfurt University EEC2018 9 Suturing on Anatomy (~1K samples)Instrument Count (~22K samples) Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sabrina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. 2018. Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 357-362. Comparatively poor performance: • 80% accuracy • 62% recall • Not enough samples (~1.000 images) • Visual context perhaps too similar Still good performance: • >92% accuracy • >84% recall • best in classifying zero instruments (acc: 96%)
  • 10. Towards Endometriosis Recognition – former approach Klaus Schöffmann, Klagenfurt University EEC2018 10 • Tool for supporting experts in annotating Endometriosis • Annotating affected regions via ENZIAN/rASRM score • Problems • Too many examples needed for every single ENZIAN/rASRM category required for machine learning • Too much effort for few expert annotators A. Leibetseder, B. Münzer, K. Schoeffmann and J. Keckstein, "Endometriosis Annotation in Endoscopic Videos," 2017 IEEE International Symposium on Multimedia (ISM), Taichung, 2017, pp. 364-365.
  • 11. Future work: Region-based Endometriosis Detection Klaus Schöffmann, Klagenfurt University EEC2018 11 Endometriosis/Adhesion Database (expert annotations) • Purpose: supporting surgeons by suggesting suspicious regions • Goals: localized detection using 3 categories of diseases: adhesions, endometriosis suspicion, endometriosis • Methodology • Creating endometriosis/adhesion database • Training and evaluating suitable machine learning models for classification Machine Learning (e.g. R-CNN) No Endometriosis Adhesions Endometriosis Prediction Model Endometriosis Suspicion
  • 12. Automatic Smoke Detection (Real-Time Support) Klaus Schöffmann, Klagenfurt University EEC2018 12 Detection Runtime + Machine Learning (acc: 94%) – Saturation Analysis (acc: 81%) + Saturation Analysis (12ms) – Machine Learning (150ms) Andreas Leibetseder, Manfred J. Primus, Stefan Petscharnig, and Klaus Schoeffmann. “Image-based Smoke Detection in Laparoscopic Videos“. Proceedings of Computer Assisted and Robotic Endoscopy and Clinical Image-Based Procedures: 4th International Workshop, CARE 2017, and 6th International Workshop, CLIP 2017, held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 14, 2017, pp. 70-87 Leibetseder A., Primus M.J., Schoeffmann K. (2018) Automatic Smoke Classification in Endoscopic Video. In: Schoeffmann K. et al. (eds) MultiMedia Modeling. MMM 2018. Lecture Notes in Computer Science, vol 10705. Springer, Cham
  • 13. • High potential in full video documentation • Enables and aids additional usage scenarios • Post-operative and real-time support • Facilitating education and training • Technical skill assessment • Improving case re-visitations • Aiding medical staff during interventions • Needs a system to organize and analyze videos • To detect important semantics/scenes • Machine learning can help a lot with this, but also it requires: • Many and diverse training examples • Computational power • Experts for creating correct training examples Conclusions Klaus Schöffmann, Klagenfurt University EEC2018 13
  • 14. Klaus Schöffmann, Klagenfurt University EEC2018 14 Thank You! Q/A Assoc.-Prof. PD Dr. Klaus Schöffmann Institut für Informationstechnologie Alpen-Adria-Universität Klagenfurt www.EndoscopicVideo.com