SlideShare a Scribd company logo
1 of 20
Download to read offline
Ground truth generation in medical imaging:
 a crowdsourcing-based iterative approach

           Antonio Foncubierta-Rodríguez
                          Henning Müller
Introduction

•   Medical image production grows rapidly in
    scientific and clinical environment
•   If images are easily accessed, they can be
    reused:
     •   Clinical decision support
     •   Young physician training
     •   Relevant document retrieval for researchers
•   Modality classification improves retrieval and
    accessibility of images
Motivation and dataset

•   ImageCLEF dataset:
     •   Over 300,000 images from open access
         biomedical literature
     •   Over 30 modalities hierarchically defined
•   Manual classification is expensive and time
    consuming
•   How can this be done in a more efficient way?
Conventional                                  Diagnostic
                                                     Ultrasound
  Tables, forms
                                                        MRI

 Program Listing                                        CT

                                                     2D X-RAY
Statistical figures,
   graphs and                                       Angiography
      charts
                                                        PET
                                                                     Radiology


     System
    overviews                                         SPECT

                                                      Infrared
   Flowcharts
                                                     Combined




                          Graph
Gene sequence                            Skin
                                                       Gross
                                       Organs
                                                                    light
                                                                   Visible




Chromatography,
                                                     Endoscopy
                                                                                                  Classification Hierarchy




     gel

                                                       EEG
    Chemical
    structure                                        ECG, EKG
                                                                                       Compound

                                                                    waves
                                                                   Signals,




                                                       EMG
     Symbol
                                                     Light Micr.

                                     Transmission     Electron
 Math formulae                        Microscope       Micr.

                                                    Fluorescence

                                        Phase
                                                    Interference
                                       contrast
                                                                     Microscopy




                                                     Dark field
                        Non

                       photos
                       clinical




                                                        2D

                                                        3D
                         sketches
                                                                     Reconstructions




                        Hand-drawn
Image examples




COMPOUND            GENERIC        GENERIC
                       Table          Figures/Charts




DIAGNOSTIC          DIAGNOSTIC     DIAGNOSTIC
   Radiology           Radiology      Microscopy
       Ultrasound          CT             Fluorescence
Iterative workflow

•     Avoid manual classification as much as possible
•     Iterative approach:
    1. Create a small training set
      •   Manual classification into 34 categories

    2. Use an automatic tool that learns from training set
    3. Evaluate results
      •   Manual classification into right/wrong categories

    4. Improve training set
    5. Repeat from 2
Crowdsourcing in medical imaging

•   Crowdsourcing reduces time and cost for
    annotation
•   Medical image annotation is often done by
     •   Medical doctors
     •   Domain experts
•   Can unknown users provide valid annotations?
     •   Quality?
     •   Speed?
User Groups

•   Experiments were performed with three different
    user groups:

                        1
                       MD

                   18 known
                    experts

           2470 contributors from
            open crowdsourcing
Crowdsourcing platform

•   Crowdflower platform was chosen for the
    experiments
     •   Integrated interface for job design
     •   Complete set of management tools: gold
         creation, internal interface, statistics, raw data
     •   Hub feature: jobs can be announced in several
         crowdsourcing pools:
     •       Amazon Mturk
     •       Get Paid
     •       Zoombucks
Experiment: Initial training set generation

•   Initial training set
    generation
     •    1,000 images
     •    Limited to 18
          known experts
     •    Aim: test the
          crowdsourcing
          interface
Experiment: Automated classification verification

•   300,000 images
•   Binary task: approve
    or refuse classification
•   Aim: evaluate speed
    and difficulty of
    verification task
Experiments: trustability

•   Trustability experiments
     •   Aim: compare user groups expected accuracy
     •   3,415 images were classified by the Medical
         Doctor
     •   The two user groups were required to reclassify
         images
     •   Random subset of 1,661 images used as gold
         standard
     •   Feedback on wrong classification was given to the
         known experts for detecting ambiguities
     •   Feedback on 847 of the gold images was muted for
         the crowd
Results: user self assessment

•   Users were required to answer how sure they
    were of their choice
•   Allows discarding untrusted data from trusted
    sources
•   Confidence rate
     •   Medical doctor: 100 %
     •   Known experts group: 95.04 %
     •   Crowd group: 85.56 %
Results: tasks completed per user

Open crowdsourcing    Internal interface
Results: MD and known experts

•   Agreement
     •   Broad category: 88.76 %
     •   Diagnostic subcategory: 97.40 %
     •   Microscopy: 89.06 %
     •   Radiology: 90.91 %
     •   Reconstructions: 100 %
     •   Visible light photography: 79.41 %

     •   Conventional subcategory: 76 %
•   Speed
     •   MD: 85 judgements per hour
     •   Experts: 66 judgements per hour and user
Results: MD and Crowd

•   Agreement
     •   Broad category: 85.53 %
     •   Diagnostic subcategory: 85.15 %
     •   Microscopy: 70.89 %
     •   Radiology: 64.01 %
     •   Reconstructions: 0 %
     •   Visible light photography: 58.89 %

     •   Conventional subcategory: 75.91 %
•   Speed
     •   MD: 85 judgements per hour
     •   Crowd: 25 judgements per hour and user
Results: Automatic classification verification

•   Verification by experts
•   1,000 images were verified
•   Agreement among annotators: 100%
•   Speed:
     •   Users answered twice as fast
Conclusions

•   Iterative approach reduces amount of manual
    work
     •   Only a small subset is fully manually annotated
     •   Automatic classification verification is faster
•   Significant differences among user groups
     •   Faster crowd annotations due to the number of
         contributors
     •   Poorer crowd annotations in the most specific
         classes
•   Comparable performance among user groups
     •   Broad categories
Future work

•   Experiments can be redesigned to fit the crowd
    behaviour:
     •   A smaller number of (good) contributors has
         previously led to CAD-comparable performance
     •   Selection of contributors:
     •      Historical performance on the platform?
     •      Selection/Training phase within the job
Thanks for your attention!

            Antonio Foncubierta-Rodríguez and Henning Müller.
   “Ground truth generation in medical imaging: A crowdsourcing based
    iterative approach”,in Workshop on Crowdsourcing for Multimedia,
                    ACM Multimedia, Nara, Japan, 2012


                      Contact: antonio.foncubierta@hevs.ch

More Related Content

What's hot

L08 Image Quality
L08 Image QualityL08 Image Quality
L08 Image Quality
lidgor
 
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
CSCJournals
 

What's hot (20)

303 306
303 306303 306
303 306
 
IRJET- Brain Tumor Detection and Classification with Feed Forward Back Propag...
IRJET- Brain Tumor Detection and Classification with Feed Forward Back Propag...IRJET- Brain Tumor Detection and Classification with Feed Forward Back Propag...
IRJET- Brain Tumor Detection and Classification with Feed Forward Back Propag...
 
Development and Implementation of VLSI Reconfigurable Architecture for Gabor ...
Development and Implementation of VLSI Reconfigurable Architecture for Gabor ...Development and Implementation of VLSI Reconfigurable Architecture for Gabor ...
Development and Implementation of VLSI Reconfigurable Architecture for Gabor ...
 
New Noise Reduction Technique for Medical Ultrasound Imaging using Gabor Filt...
New Noise Reduction Technique for Medical Ultrasound Imaging using Gabor Filt...New Noise Reduction Technique for Medical Ultrasound Imaging using Gabor Filt...
New Noise Reduction Technique for Medical Ultrasound Imaging using Gabor Filt...
 
237 240
237 240237 240
237 240
 
A Review of Various Retinal Microaneurysm Detection Methods For Grading Of Di...
A Review of Various Retinal Microaneurysm Detection Methods For Grading Of Di...A Review of Various Retinal Microaneurysm Detection Methods For Grading Of Di...
A Review of Various Retinal Microaneurysm Detection Methods For Grading Of Di...
 
Altaire Performance Paper
Altaire Performance PaperAltaire Performance Paper
Altaire Performance Paper
 
L08 Image Quality
L08 Image QualityL08 Image Quality
L08 Image Quality
 
ENHANCED SYSTEM FOR COMPUTER-AIDED DETECTION OF MRI BRAIN TUMORS
ENHANCED SYSTEM FOR COMPUTER-AIDED DETECTION OF MRI BRAIN TUMORSENHANCED SYSTEM FOR COMPUTER-AIDED DETECTION OF MRI BRAIN TUMORS
ENHANCED SYSTEM FOR COMPUTER-AIDED DETECTION OF MRI BRAIN TUMORS
 
D45012128
D45012128D45012128
D45012128
 
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
A Wavelet Based Automatic Segmentation of Brain Tumor in CT Images Using Opti...
 
Detection of exudates draft
Detection of exudates draftDetection of exudates draft
Detection of exudates draft
 
Implementation of Lower Leg Bone Fracture Detection from X Ray Images
Implementation of Lower Leg Bone Fracture Detection from X Ray ImagesImplementation of Lower Leg Bone Fracture Detection from X Ray Images
Implementation of Lower Leg Bone Fracture Detection from X Ray Images
 
Viva201393(1).pptxbaru
Viva201393(1).pptxbaruViva201393(1).pptxbaru
Viva201393(1).pptxbaru
 
Iq3116211626
Iq3116211626Iq3116211626
Iq3116211626
 
Automatic target detection in
Automatic target detection inAutomatic target detection in
Automatic target detection in
 
An Image-Based Bone fracture Detection Using AForge Library
An Image-Based Bone fracture Detection Using AForge LibraryAn Image-Based Bone fracture Detection Using AForge Library
An Image-Based Bone fracture Detection Using AForge Library
 
Detection of exudates draft
Detection of exudates draftDetection of exudates draft
Detection of exudates draft
 
Pellucid radlive modules
Pellucid radlive modulesPellucid radlive modules
Pellucid radlive modules
 
Efficient Brain Tumor Detection Using Wavelet Transform
Efficient Brain Tumor Detection Using Wavelet TransformEfficient Brain Tumor Detection Using Wavelet Transform
Efficient Brain Tumor Detection Using Wavelet Transform
 

Viewers also liked

Tema 4 prácticas- myriam oliva díaz-palacios 3ºb
Tema 4 prácticas- myriam oliva díaz-palacios 3ºbTema 4 prácticas- myriam oliva díaz-palacios 3ºb
Tema 4 prácticas- myriam oliva díaz-palacios 3ºb
Olivovy
 
Guia GastronóMica
Guia GastronóMicaGuia GastronóMica
Guia GastronóMica
aguadediego
 
Brexpiprazole Final Presentation
Brexpiprazole Final PresentationBrexpiprazole Final Presentation
Brexpiprazole Final Presentation
Nina Chacko
 

Viewers also liked (20)

wengermayer business consulting company presentation personalberatung german
wengermayer business consulting company presentation personalberatung germanwengermayer business consulting company presentation personalberatung german
wengermayer business consulting company presentation personalberatung german
 
Expocision mundo de sofia-Nixon t.1001
Expocision mundo de sofia-Nixon t.1001Expocision mundo de sofia-Nixon t.1001
Expocision mundo de sofia-Nixon t.1001
 
WEB 1.0 , 2.0 Y 3.0
WEB 1.0 , 2.0 Y 3.0WEB 1.0 , 2.0 Y 3.0
WEB 1.0 , 2.0 Y 3.0
 
Presentació del pla d'acció per al desenvolupament del tecmed contract manufa...
Presentació del pla d'acció per al desenvolupament del tecmed contract manufa...Presentació del pla d'acció per al desenvolupament del tecmed contract manufa...
Presentació del pla d'acció per al desenvolupament del tecmed contract manufa...
 
Libro azul salusi
Libro azul salusiLibro azul salusi
Libro azul salusi
 
Yeastar My pbx u200_datasheet_en
Yeastar My pbx u200_datasheet_enYeastar My pbx u200_datasheet_en
Yeastar My pbx u200_datasheet_en
 
Tema 4 prácticas- myriam oliva díaz-palacios 3ºb
Tema 4 prácticas- myriam oliva díaz-palacios 3ºbTema 4 prácticas- myriam oliva díaz-palacios 3ºb
Tema 4 prácticas- myriam oliva díaz-palacios 3ºb
 
Iveco aifo parts
Iveco aifo parts Iveco aifo parts
Iveco aifo parts
 
Halloween fun mr speedy plumbing los angeles ca
Halloween fun   mr speedy plumbing los angeles caHalloween fun   mr speedy plumbing los angeles ca
Halloween fun mr speedy plumbing los angeles ca
 
Guia GastronóMica
Guia GastronóMicaGuia GastronóMica
Guia GastronóMica
 
Coaching on court
Coaching on courtCoaching on court
Coaching on court
 
Animales de la suerte con cuentas y abalorios
Animales de la suerte con cuentas y abaloriosAnimales de la suerte con cuentas y abalorios
Animales de la suerte con cuentas y abalorios
 
Brexpiprazole Final Presentation
Brexpiprazole Final PresentationBrexpiprazole Final Presentation
Brexpiprazole Final Presentation
 
Comercio electronico b2 b
Comercio electronico b2 bComercio electronico b2 b
Comercio electronico b2 b
 
Axudas conselleria economia emprego e industria modif comercio 250116
Axudas conselleria economia emprego e industria modif comercio 250116Axudas conselleria economia emprego e industria modif comercio 250116
Axudas conselleria economia emprego e industria modif comercio 250116
 
Suggestions For Road Safety Spanish
Suggestions  For  Road  Safety  SpanishSuggestions  For  Road  Safety  Spanish
Suggestions For Road Safety Spanish
 
Dem Installateur das Leben einfacher machen
Dem Installateur das Leben einfacher machenDem Installateur das Leben einfacher machen
Dem Installateur das Leben einfacher machen
 
Aprendizaje Automático con Python
Aprendizaje Automático con PythonAprendizaje Automático con Python
Aprendizaje Automático con Python
 
SanCor Cooperativas Unidas Limitada
SanCor Cooperativas Unidas Limitada SanCor Cooperativas Unidas Limitada
SanCor Cooperativas Unidas Limitada
 
INMUNOLOGÍA
INMUNOLOGÍA INMUNOLOGÍA
INMUNOLOGÍA
 

Similar to Ground truth generation in medical imaging: a crowdsourcing-based iterative approach

BMES_Poster_Microscope_Comparison_Meyer
BMES_Poster_Microscope_Comparison_MeyerBMES_Poster_Microscope_Comparison_Meyer
BMES_Poster_Microscope_Comparison_Meyer
Andrew Meyer
 
BMES_Abstract_Microscope_Comparison_Meyer
BMES_Abstract_Microscope_Comparison_MeyerBMES_Abstract_Microscope_Comparison_Meyer
BMES_Abstract_Microscope_Comparison_Meyer
Andrew Meyer
 
Design and development of pulmonary tuberculosis diagnosing system using image
Design and development of pulmonary tuberculosis diagnosing system using imageDesign and development of pulmonary tuberculosis diagnosing system using image
Design and development of pulmonary tuberculosis diagnosing system using image
IAEME Publication
 
A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...
A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...
A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...
NIET Journal of Engineering & Technology (NIETJET)
 

Similar to Ground truth generation in medical imaging: a crowdsourcing-based iterative approach (20)

BMES_Poster_Microscope_Comparison_Meyer
BMES_Poster_Microscope_Comparison_MeyerBMES_Poster_Microscope_Comparison_Meyer
BMES_Poster_Microscope_Comparison_Meyer
 
AUTOMATIC TARGET DETECTION IN HYPERSPECTRAL IMAGES USING NEURAL NETWORK
AUTOMATIC TARGET DETECTION IN HYPERSPECTRAL IMAGES USING NEURAL NETWORKAUTOMATIC TARGET DETECTION IN HYPERSPECTRAL IMAGES USING NEURAL NETWORK
AUTOMATIC TARGET DETECTION IN HYPERSPECTRAL IMAGES USING NEURAL NETWORK
 
Digitalimageprocessing
DigitalimageprocessingDigitalimageprocessing
Digitalimageprocessing
 
Review of Classification algorithms for Brain MRI images
Review of Classification algorithms for Brain MRI imagesReview of Classification algorithms for Brain MRI images
Review of Classification algorithms for Brain MRI images
 
BMES_Abstract_Microscope_Comparison_Meyer
BMES_Abstract_Microscope_Comparison_MeyerBMES_Abstract_Microscope_Comparison_Meyer
BMES_Abstract_Microscope_Comparison_Meyer
 
osteo.pptx
osteo.pptxosteo.pptx
osteo.pptx
 
Design and development of pulmonary tuberculosis diagnosing system using image
Design and development of pulmonary tuberculosis diagnosing system using imageDesign and development of pulmonary tuberculosis diagnosing system using image
Design and development of pulmonary tuberculosis diagnosing system using image
 
Pathomics Based Biomarkers, Tools, and Methods
Pathomics Based Biomarkers, Tools, and MethodsPathomics Based Biomarkers, Tools, and Methods
Pathomics Based Biomarkers, Tools, and Methods
 
Analysis of Efficient Wavelet Based Volumetric Image Compression
Analysis of Efficient Wavelet Based Volumetric Image CompressionAnalysis of Efficient Wavelet Based Volumetric Image Compression
Analysis of Efficient Wavelet Based Volumetric Image Compression
 
spect .pptx
spect .pptxspect .pptx
spect .pptx
 
Automated_EEG_artifact_elimination_by_applying_mac.pdf
Automated_EEG_artifact_elimination_by_applying_mac.pdfAutomated_EEG_artifact_elimination_by_applying_mac.pdf
Automated_EEG_artifact_elimination_by_applying_mac.pdf
 
Medical_Image_Processing-首节课.pptx
Medical_Image_Processing-首节课.pptxMedical_Image_Processing-首节课.pptx
Medical_Image_Processing-首节课.pptx
 
diagnostic technique.pptx
diagnostic technique.pptxdiagnostic technique.pptx
diagnostic technique.pptx
 
Ca4201508513
Ca4201508513Ca4201508513
Ca4201508513
 
Early stage detection of skin cancer via terahertz spectral profiling and 3D ...
Early stage detection of skin cancer via terahertz spectral profiling and 3D ...Early stage detection of skin cancer via terahertz spectral profiling and 3D ...
Early stage detection of skin cancer via terahertz spectral profiling and 3D ...
 
Analysis and Classification of Skin Lesions Using 3D Volume Reconstruction
Analysis and Classification of Skin Lesions Using 3D Volume ReconstructionAnalysis and Classification of Skin Lesions Using 3D Volume Reconstruction
Analysis and Classification of Skin Lesions Using 3D Volume Reconstruction
 
Medical Image Processing in Nuclear Medicine and Bone Arthroplasty
Medical Image Processing in Nuclear Medicine and Bone ArthroplastyMedical Image Processing in Nuclear Medicine and Bone Arthroplasty
Medical Image Processing in Nuclear Medicine and Bone Arthroplasty
 
A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...
A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...
A Proposed Framework to De-noise Medical Images Based on Convolution Neural N...
 
APPLICATION OF 1D CNN IN ECG CLASSIFICATION
APPLICATION OF 1D CNN IN ECG CLASSIFICATIONAPPLICATION OF 1D CNN IN ECG CLASSIFICATION
APPLICATION OF 1D CNN IN ECG CLASSIFICATION
 
Vevo LAZR-X Brochure mkt02906 v1.0
Vevo LAZR-X Brochure mkt02906 v1.0Vevo LAZR-X Brochure mkt02906 v1.0
Vevo LAZR-X Brochure mkt02906 v1.0
 

More from Institute of Information Systems (HES-SO)

More from Institute of Information Systems (HES-SO) (20)

MIE20232.pptx
MIE20232.pptxMIE20232.pptx
MIE20232.pptx
 
Classification of noisy free-text prostate cancer pathology reports using nat...
Classification of noisy free-text prostate cancer pathology reports using nat...Classification of noisy free-text prostate cancer pathology reports using nat...
Classification of noisy free-text prostate cancer pathology reports using nat...
 
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Machine learning assisted citation screening for Systematic Reviews - Anjani ...Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
 
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Exploiting biomedical literature to mine out a large multimodal dataset of ra...Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
 
L'IoT dans les usines. Quels avantages ?
L'IoT dans les usines. Quels avantages ?L'IoT dans les usines. Quels avantages ?
L'IoT dans les usines. Quels avantages ?
 
Studying Public Medical Images from Open Access Literature and Social Network...
Studying Public Medical Images from Open Access Literature and Social Network...Studying Public Medical Images from Open Access Literature and Social Network...
Studying Public Medical Images from Open Access Literature and Social Network...
 
Risques opérationnels et le système de contrôle interne : les limites d’un te...
Risques opérationnels et le système de contrôle interne : les limites d’un te...Risques opérationnels et le système de contrôle interne : les limites d’un te...
Risques opérationnels et le système de contrôle interne : les limites d’un te...
 
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
 
Le système de contrôle interne : Présentation générale, enjeux et méthodes
Le système de contrôle interne : Présentation générale, enjeux et méthodesLe système de contrôle interne : Présentation générale, enjeux et méthodes
Le système de contrôle interne : Présentation générale, enjeux et méthodes
 
Crowdsourcing-based Mobile Application for Wheelchair Accessibility
Crowdsourcing-based Mobile Application for Wheelchair AccessibilityCrowdsourcing-based Mobile Application for Wheelchair Accessibility
Crowdsourcing-based Mobile Application for Wheelchair Accessibility
 
Quelle(s) valeur(s) pour le leadership stratégique ?
Quelle(s) valeur(s) pour le leadership stratégique ?Quelle(s) valeur(s) pour le leadership stratégique ?
Quelle(s) valeur(s) pour le leadership stratégique ?
 
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
 
Challenges in medical imaging and the VISCERAL model
Challenges in medical imaging and the VISCERAL modelChallenges in medical imaging and the VISCERAL model
Challenges in medical imaging and the VISCERAL model
 
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
NOSE: une approche Smart-City pour les zones périphériques et extra-urbainesNOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
 
Medical image analysis and big data evaluation infrastructures
Medical image analysis and big data evaluation infrastructuresMedical image analysis and big data evaluation infrastructures
Medical image analysis and big data evaluation infrastructures
 
Medical image analysis, retrieval and evaluation infrastructures
Medical image analysis, retrieval and evaluation infrastructuresMedical image analysis, retrieval and evaluation infrastructures
Medical image analysis, retrieval and evaluation infrastructures
 
How to detect soft falls on devices
How to detect soft falls on devicesHow to detect soft falls on devices
How to detect soft falls on devices
 
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSISFUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
 
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLSMOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
 
Enhanced Students Laboratory The GET project
Enhanced Students Laboratory The GET projectEnhanced Students Laboratory The GET project
Enhanced Students Laboratory The GET project
 

Recently uploaded

Recently uploaded (20)

Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 

Ground truth generation in medical imaging: a crowdsourcing-based iterative approach

  • 1. Ground truth generation in medical imaging: a crowdsourcing-based iterative approach Antonio Foncubierta-Rodríguez Henning Müller
  • 2. Introduction • Medical image production grows rapidly in scientific and clinical environment • If images are easily accessed, they can be reused: • Clinical decision support • Young physician training • Relevant document retrieval for researchers • Modality classification improves retrieval and accessibility of images
  • 3. Motivation and dataset • ImageCLEF dataset: • Over 300,000 images from open access biomedical literature • Over 30 modalities hierarchically defined • Manual classification is expensive and time consuming • How can this be done in a more efficient way?
  • 4. Conventional Diagnostic Ultrasound Tables, forms MRI Program Listing CT 2D X-RAY Statistical figures, graphs and Angiography charts PET Radiology System overviews SPECT Infrared Flowcharts Combined Graph Gene sequence Skin Gross Organs light Visible Chromatography, Endoscopy Classification Hierarchy gel EEG Chemical structure ECG, EKG Compound waves Signals, EMG Symbol Light Micr. Transmission Electron Math formulae Microscope Micr. Fluorescence Phase Interference contrast Microscopy Dark field Non photos clinical 2D 3D sketches Reconstructions Hand-drawn
  • 5. Image examples COMPOUND GENERIC GENERIC Table Figures/Charts DIAGNOSTIC DIAGNOSTIC DIAGNOSTIC Radiology Radiology Microscopy Ultrasound CT Fluorescence
  • 6. Iterative workflow • Avoid manual classification as much as possible • Iterative approach: 1. Create a small training set • Manual classification into 34 categories 2. Use an automatic tool that learns from training set 3. Evaluate results • Manual classification into right/wrong categories 4. Improve training set 5. Repeat from 2
  • 7. Crowdsourcing in medical imaging • Crowdsourcing reduces time and cost for annotation • Medical image annotation is often done by • Medical doctors • Domain experts • Can unknown users provide valid annotations? • Quality? • Speed?
  • 8. User Groups • Experiments were performed with three different user groups: 1 MD 18 known experts 2470 contributors from open crowdsourcing
  • 9. Crowdsourcing platform • Crowdflower platform was chosen for the experiments • Integrated interface for job design • Complete set of management tools: gold creation, internal interface, statistics, raw data • Hub feature: jobs can be announced in several crowdsourcing pools: • Amazon Mturk • Get Paid • Zoombucks
  • 10. Experiment: Initial training set generation • Initial training set generation • 1,000 images • Limited to 18 known experts • Aim: test the crowdsourcing interface
  • 11. Experiment: Automated classification verification • 300,000 images • Binary task: approve or refuse classification • Aim: evaluate speed and difficulty of verification task
  • 12. Experiments: trustability • Trustability experiments • Aim: compare user groups expected accuracy • 3,415 images were classified by the Medical Doctor • The two user groups were required to reclassify images • Random subset of 1,661 images used as gold standard • Feedback on wrong classification was given to the known experts for detecting ambiguities • Feedback on 847 of the gold images was muted for the crowd
  • 13. Results: user self assessment • Users were required to answer how sure they were of their choice • Allows discarding untrusted data from trusted sources • Confidence rate • Medical doctor: 100 % • Known experts group: 95.04 % • Crowd group: 85.56 %
  • 14. Results: tasks completed per user Open crowdsourcing Internal interface
  • 15. Results: MD and known experts • Agreement • Broad category: 88.76 % • Diagnostic subcategory: 97.40 % • Microscopy: 89.06 % • Radiology: 90.91 % • Reconstructions: 100 % • Visible light photography: 79.41 % • Conventional subcategory: 76 % • Speed • MD: 85 judgements per hour • Experts: 66 judgements per hour and user
  • 16. Results: MD and Crowd • Agreement • Broad category: 85.53 % • Diagnostic subcategory: 85.15 % • Microscopy: 70.89 % • Radiology: 64.01 % • Reconstructions: 0 % • Visible light photography: 58.89 % • Conventional subcategory: 75.91 % • Speed • MD: 85 judgements per hour • Crowd: 25 judgements per hour and user
  • 17. Results: Automatic classification verification • Verification by experts • 1,000 images were verified • Agreement among annotators: 100% • Speed: • Users answered twice as fast
  • 18. Conclusions • Iterative approach reduces amount of manual work • Only a small subset is fully manually annotated • Automatic classification verification is faster • Significant differences among user groups • Faster crowd annotations due to the number of contributors • Poorer crowd annotations in the most specific classes • Comparable performance among user groups • Broad categories
  • 19. Future work • Experiments can be redesigned to fit the crowd behaviour: • A smaller number of (good) contributors has previously led to CAD-comparable performance • Selection of contributors: • Historical performance on the platform? • Selection/Training phase within the job
  • 20. Thanks for your attention! Antonio Foncubierta-Rodríguez and Henning Müller. “Ground truth generation in medical imaging: A crowdsourcing based iterative approach”,in Workshop on Crowdsourcing for Multimedia, ACM Multimedia, Nara, Japan, 2012 Contact: antonio.foncubierta@hevs.ch