1.
Multimedia for Medicine:
The Medico Task
Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Carsten Griwodz,
Thomas de Lange, Kristin Ranheim Randel, Sigrun Losada Eskeland,
Duc-Tien Dang-Nguyen, Mathias Lux, Concetto Spampinato
Email: michael@simula.no, konstantin@simula.no

2.
Medico
 Bringing together IT and medicine, focusing on detecting
abnormalities, diseases and anatomical landmarks in images captured
by medical devices in the gastrointestinal tract
 Goals of the task: Help to improve the health care system using
multimedia methods to reach the next level of multimedia-assisted
computer diagnosis, detection and interpretation of abnormalities
 Attract more researchers to medical use cases

3.
GI Tract Challenges
 Many types of diseases can affect the human digestive system
 Screening of the gastrointestinal (GI) tract using different
types of traditional endoscopy
– is costly (colonoscopy: US - $1100/patient, $10 billion dollars)
– consumes valuable medical personnel time (1-2 hours)
– does not scale to large populations
– is intrusive to the patient
 Current developments in technology may potentially enable
automatic algorithmic screening and assisted examinations
 a true interdisciplinary activity with high chances of societal impact

4.
Colorectal Cancer Mortality 2012
 Most common cancer for men in Norway
 Second most common cancer for women in Norway

5.
Live Automatic Detection
 System to assist doctors during live
traditional endoscopy procedures
 Second pair of eyes
 Support for inexperienced doctors
 Automatic tagging of lesions
 Automated reports generation
 Better procedure documentation
[1] van Rijn, J. C., Reitsma, J. B., Stoker, J., Bossuyt, P. M., van Deventer, S. J., and Dekker, E.
Polyp miss rate determined by tandem colonoscopy: a systematic review.
The American journal of gastroenterology 101, 2 (2006)

6.
Wireless Video Capsule (Capsular Endoscopy)
 better scale
 less intrusive
 possible to combine
examinations!?
 less expensive?
(detection might lead to an endoscopy)
 expensive
 does not scale
 intrusive

7.
Our Goals
 A complete system for:
– Live Traditional Endoscopy
– Capsular Endoscopy
 Automatic detection of different abnormalities in the
digestive system
– HD sources
– Real-time and faster
– High recall and precision
– Automated reports generation

8.
EIR Overview

9.
Common Challenges
 Blurry images due to camera motion
 Objects too close to camera
 Under or over scene lighting
 Flares
 Artificial objects and natural "contaminants“
 Low resolution of Capsular Endoscopes
 No proper support for medical reports generation

10.
Common Challenges

11.
Subtasks
 Detection
 Detection of different diseases and landmarks
 Use as few images in the training dataset as possible
 Efficient detection
 Solve the classification problem as fast as possible
 Efficient detection (fare evaluation, the same hardware)
 Evaluate detection algorithms on the same hardware
 Report generation (Experimental)
 Automatically create a text-report for a medical doctor for three
video cases

12.
Dataset
 Hard to find annotated data allowed to use (even) for research
 Two data sets have been published at the MMSys 2017 open
dataset track: Kvasir and Nerthus (bowel preparation quality)
 Kvasir: the open source dataset consist of 8,000 annotated GI tract
images in 8 different classes: 500 in dev. + 500 in test set per class

13.
Participants and submissions
 12 teams registered and requested the data
 5 teams successfully submitted the results
 Austria, China, India, Italy, Norway, Pakistan, US…
 46 submissions in total
 Detection – Successful
 Efficient detection – Partially Successful
 Efficient detection – No-Show
 Report generation – No-Go

14.
The Approaches
 Machine Learning
– Manifold learning
– Support vector machine
– Random forest and random tree
– Logistic model tree
– Regression-based
– Unsupervised clustering
 Convolutional Neural Networks
– Direct classification
– Transfer learning
 Features Extraction
– Deep features
– Global and Local features

15.
Metrics
 Multi-class generalization of Matthews correlation coefficient (R_K
statistic)
– Correlation coefficient between the observed and predicted
classifications
– Perfect for unbalanced multi-class case
– Maximum value +1 for perfect prediction
– Minimum value between -1 and 0 depending on the true
distribution
• Frames per second (FPS)

16.
The Results – Best Performing
Team Training
set size
Multi-class
MCC
F1 FPS
SCL-UMD 3200 0.827 0.848 1.3
FAST-NU-DS 4000 0.736 0.767 2.3
ITEC-AAU 3600 0.724 0.755 1.4
HKBU 1000 0.663 0.703 2.2
SIMULA 4000 0.802 0.826 46.0
Random - -0.001 0.124 -
ZeroR - 0 0.125 -

17.
The Results – Best Submission Confusion Matrix
Predicted class
Actual class
polyps
normal-
cecum
normal-z-
line
normal-
pylorus
esophagitis
dyed-
resection-
margins
dyed-
lifted-
polyps
ulcerative
-colitis
polyps 448 13 1 0 0 0 23 33
normal-cecum 22 478 0 0 0 0 0 27
normal-z-line 0 0 427 8 202 0 0 0
normal-pylorus 4 0 5 480 5 0 0 0
esophagitis 0 0 67 10 293 0 0 2
dyed-resection-
margins 0 0 0 0 0 406 55 1
dyed-lifted-
polyps 2 0 0 0 0 94 421 0
ulcerative-colitis 24 9 0 2 0 0 1 437

18.
The Results – Smallest Training Set
Team Training
set size
Multi-class
MCC
F1
ITEC-AAU 400 0.607 0.649
FAST-NU-DS 732 0.649 0.689
HKBU 800 0.648 0.692
SCL-UMD 3200 0.827 0.848
SIMULA 4000 0.802 0.826
Random - -0.001 0.124
ZeroR - 0 0.125

19.
Conclusions
1. Good classification performance achieved
2. Small amount of training data would not stop us!
3. Real-time is still challenging
4. Combined approaches are required for medical image
analysis
5. Medical task is interesting for the community

20.
Future of the task
1. Localization/segmentation of findings and lesions
2. Exploiting domain expert knowledge – more data!
3. Integration of various data, multi-modality – new sensors,
doctors’ records, audio recordings, patient context
information
4. Automated reporting

21.
Thank You! Questions?
All the data is released publicly and available at:
http://datasets.simula.no/kvasir/