SlideShare a Scribd company logo
1 of 23
Download to read offline
Challenges in medical imaging and
the VISCERAL model
Henning Müller
HES-SO &
Martinos Center
Overview
• Systematic evaluations
– Information retrieval, industrial challenges
• ImageCLEF
– 2003-2016
• Challengesin medical imaging and Open Science
– Conference and platforms (Kaggle, Topcoder, …)
• VISCERAL
– “Moving the algorithms to the data and not the data
to the algorithms”
• Conclusions
Systematic evaluations
• 1960-: the Cranfieldtests
– Test collection, tasks, ground truth
– Automatic indexing better than manual terms
• 1992-: TREC – Text Retrieval Conference
– At NIST, Gaithersburg
– Many different tasks over the years
• 1999-: CLEF, TRECVid as offspring of TREC
• Industrial performance benchmarks
– TPC (1988), common transaction processing frame
– Supercomputer benchmark (1993), common criteria
– …
Cleverdon, C. W. (1960). ASLIB Cranfield research project
on the comparative efficiency of indexing systems.ASLIB
Proceedings, XII, 421-431.
• Benchmark on multimodal imageretrieval
– Run since 2003, medical task since 2004
– Part of the Cross Language Evaluation Forum (CLEF)
• Many tasks related to medical image retrieval
– Image classification (modality, body part, …)
– Image-based retrieval
– Case-based retrieval (finding similar cases)
– Compound figure separation
– Caption prediction
– …
• Many old databases remain available, imageclef.org
Henning Müller, Paul Clough, Thomas
Deselaers, Barbara Caputo, ImageCLEF –
Experimental evaluation of visual information
retrieval, Springer, 2010.
ImageCLEF experiences
• Creating a community is important to have a good
participation (many groups register to access data)
– Workshop to discuss results, evolution of tasks over
the years (attracting postgraduate students)
• Impact of data sets can be high (see also TREC)
– Overview articles are frequently cited, best
participant algorithms as well
• Large data causes problems in some countries
• Hard to make groups collaborate
– Evaluate system components
• Little interactive evaluation of systems
• Not everything is fully reproducible
Open Science
• Initiatives to share data, tasks and tools
– Not only experts, really everyone
– More efficient way to do science, no reimplementation
– NIH, some journals push for open data, open access
• Data papers and executable papers
– Full reproducibility
• Which is otherwise often not given!!
• Challengesas an important way to bring many
people into the loop of data science
– http://www.challenge.gov/
– Kaggle, TopCoder, …
Open Science is the practice of science in such a way
that others can collaborate and contribute, where
research data, lab notes and other research processes
are freely available, under terms that enable reuse,
redistribution and reproduction of the research and its underlying data and methods.
Challenges in Medical Imaging
• Grand Challenges in Medical Imaging
– http://grand-challenge.org/
– Including challenges and details on impact & why
• 2007: MICCAI workshop on liver segmentation
– Including common data
• Now most conferences organize many challenge
sessions in addition to workshops
– MICCAI, ISBI, SPIE, ICPR, …
• Problem: still, most publications are on closed data
sets, impossible to verify, small, ...
– What if all were available on a secure infrastructure?
Platforms for ML challenges
• Kaggle
– Much influence on machine learning challenges
– Big commercial factor, giving price money and
also hiring good talent
– Download data, submit results
• TopCoder
– Use of code instead of results list
– 79,900,000 of price money distributed
– Almost 1 million members
• Many other exist in specific domains
– Sage Bionetworks in the biomedical field
Challenges with challenges
• Get a large number of participants and different
techniques, as there are many burdens
– Only one can win in the end, price money
• Same conditions for all (computation, bandwidth)
• How to distribute very large data sets?
• How to deal with confidential/restricteddata?
– Medical, commercial data, forbidden data sets
• How to deal with quickly changing data?
– Data of cell phone providers, Internet companies
• Reproducibility
– Optimizations on test data, particularly with prices
cloud
VISCERAL model
• VISCERAL – Visual Concept Extraction
Challenge in Radiology
• “Bringing the algorithms to the data”
– Have the data centrally stored, in our case in the
cloud (which can be HIPAA compliant)
• Three types of challenges
– Anatomy segmentations (3x), 20 organs
– Retrieval challenge (2x), finding similar cases
– Lesion detection challenge (2x), 5 organs
• Provide large data sets that are well annotated
and can be shared long term
– Challenging with IRB approval in three countries
Test
Resources available
Test DataTraining Data
Participants Organiser
Participant
Virtual
MachinesRegistration
System
Annotation
Management System
Analysis
System
Annotators
(Radiologists)
Locally Installed
Annotation
Clients
Microsoft
Azure
Cloud
Test Data
Silver corpus (example trachea)
• Executable code of all participants
– Run it on new data, do label fusion
Dice 0.85 Dice 0.71 Dice 0.84 Dice 0.83
Participant segmentations
Dice 0.92
Silver Corpus
Docker vs. Virtual MachinesContainers
Bins/Libs
VM
VS.
Evaluation as a Service (EaaS)
• Evaluationvia APIs, code, cloud, …
• Workshop in Sierre in March 2015
• Many aspects, viewpoints,
interests
• White paper published
– ArXiv
• All comments are
welcome!!
Cloud-based evaluation
• Workshop at Martinos
Center, Boston, MA, November 2015
• How to run benchmarks on very large data sets in
the cloud (reproducibility, motivation)
• Many different stakeholders
– Scientists, infrastructure providers, companies,
funding organizations
• Sustainability is a major challenge
• Interest of singe persons vs. interests of a domain
– Give credit to creators of data and tools
• Nature Scientific data
Coding4Cancer & others
• Challenge on cancer prediction (breast, lung)
• Price money for the challenges
– Make code open source to be eligible
• Commercial medical imaging challenges
– Zebra Medical Vision
• Large data sets available for research
• Use of their infrastructure, only, using Docker
– RadLogic
• Plug-in concept for algorithms of scientists
What is needed now?
• Long-term vision of how medical data analysis will
develop and how data & tools can be shared
– Moon shot initiative on cancer (Biden)
• International research infrastructure
– Public-private partnerships to make them
sustainable, still sharing costs is not clear
– Leaving data where produced, moving the code
• Incentives to share data and task environments
– Those doing the major work should receive credit
– More work on those preparing data & tasks
• Annotate data, standard formats, support to others
Conclusions
• Open Science is developingquickly
– Potential advantages for all
• The medical domain is complicated as data
require protection (the more, the bigger)
– Particularly for genomics
– No duplications limit data exposure
• Translational aspects also need to be taken into
account (transfer code towards products)
– Executable “papers” and available data should help
– Objective performance comparison
• Challengeswill be part of this ecosystem
Contact
• More information canbe found at
– http://www.imageclef.org/
– http://visceral.eu/
– http://medgift.hevs.ch/
– http://publications.hevs.ch/
• Contact:
– Henning.mueller@hevs.ch

More Related Content

What's hot

Digital Pathology at John Hopkins
Digital Pathology at John HopkinsDigital Pathology at John Hopkins
Digital Pathology at John HopkinsWilliam Baird
 
Digital webinar master deck final
Digital webinar master deck finalDigital webinar master deck final
Digital webinar master deck finalPistoia Alliance
 
The XNAT imaging informatics platform
The XNAT imaging informatics platformThe XNAT imaging informatics platform
The XNAT imaging informatics platformimgcommcall
 
Introduction to Data Management
Introduction to Data ManagementIntroduction to Data Management
Introduction to Data ManagementAmanda Whitmire
 
Digital pathology and its importance as an omics data layer
Digital pathology and its importance as an omics data layerDigital pathology and its importance as an omics data layer
Digital pathology and its importance as an omics data layerYves Sucaet
 
Data management (1)
Data management (1)Data management (1)
Data management (1)SM Lalon
 
Open and Collaborative Software for Digital Pathology
Open and Collaborative Software for Digital Pathology Open and Collaborative Software for Digital Pathology
Open and Collaborative Software for Digital Pathology William Baird
 
Data, Data Everywhere: What's A Publisher to Do?
Data, Data Everywhere: What's  A Publisher to Do?Data, Data Everywhere: What's  A Publisher to Do?
Data, Data Everywhere: What's A Publisher to Do?Anita de Waard
 
Machine Learning in Modern Medicine with Erin LeDell at Stanford Med
Machine Learning in Modern Medicine with Erin LeDell at Stanford MedMachine Learning in Modern Medicine with Erin LeDell at Stanford Med
Machine Learning in Modern Medicine with Erin LeDell at Stanford MedSri Ambati
 
Current and emerging scientific data curation practices
Current and emerging scientific data curation practicesCurrent and emerging scientific data curation practices
Current and emerging scientific data curation practicesMichael Day
 
Survey of research data management practices up2010digschol2011
Survey of research data management practices up2010digschol2011Survey of research data management practices up2010digschol2011
Survey of research data management practices up2010digschol2011heila1
 
IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...
IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...
IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...Amanda Whitmire
 
DCC 101: Preservation
DCC 101: PreservationDCC 101: Preservation
DCC 101: PreservationMichael Day
 
Institutional Data Management Blueprint
Institutional Data Management BlueprintInstitutional Data Management Blueprint
Institutional Data Management BlueprintJisc
 
Introduction to research data management; Lecture 01 for GRAD521
Introduction to research data management; Lecture 01 for GRAD521Introduction to research data management; Lecture 01 for GRAD521
Introduction to research data management; Lecture 01 for GRAD521Amanda Whitmire
 
Nataly Zhukova - Conceptual Model for Routine Measurements Analyses in Seman...
Nataly Zhukova - Conceptual Model for Routine Measurements Analyses  in Seman...Nataly Zhukova - Conceptual Model for Routine Measurements Analyses  in Seman...
Nataly Zhukova - Conceptual Model for Routine Measurements Analyses in Seman...AIST
 

What's hot (20)

Digital Pathology at John Hopkins
Digital Pathology at John HopkinsDigital Pathology at John Hopkins
Digital Pathology at John Hopkins
 
Digital webinar master deck final
Digital webinar master deck finalDigital webinar master deck final
Digital webinar master deck final
 
The XNAT imaging informatics platform
The XNAT imaging informatics platformThe XNAT imaging informatics platform
The XNAT imaging informatics platform
 
Introduction to Data Management
Introduction to Data ManagementIntroduction to Data Management
Introduction to Data Management
 
Digital pathology and its importance as an omics data layer
Digital pathology and its importance as an omics data layerDigital pathology and its importance as an omics data layer
Digital pathology and its importance as an omics data layer
 
Data management (1)
Data management (1)Data management (1)
Data management (1)
 
Open and Collaborative Software for Digital Pathology
Open and Collaborative Software for Digital Pathology Open and Collaborative Software for Digital Pathology
Open and Collaborative Software for Digital Pathology
 
2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...
2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...
2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...
 
Data, Data Everywhere: What's A Publisher to Do?
Data, Data Everywhere: What's  A Publisher to Do?Data, Data Everywhere: What's  A Publisher to Do?
Data, Data Everywhere: What's A Publisher to Do?
 
Rdm slides march 2014
Rdm slides march 2014Rdm slides march 2014
Rdm slides march 2014
 
Machine Learning in Modern Medicine with Erin LeDell at Stanford Med
Machine Learning in Modern Medicine with Erin LeDell at Stanford MedMachine Learning in Modern Medicine with Erin LeDell at Stanford Med
Machine Learning in Modern Medicine with Erin LeDell at Stanford Med
 
Current and emerging scientific data curation practices
Current and emerging scientific data curation practicesCurrent and emerging scientific data curation practices
Current and emerging scientific data curation practices
 
Research-KS-Jun2015
Research-KS-Jun2015Research-KS-Jun2015
Research-KS-Jun2015
 
data curation issues
data curation issuesdata curation issues
data curation issues
 
Survey of research data management practices up2010digschol2011
Survey of research data management practices up2010digschol2011Survey of research data management practices up2010digschol2011
Survey of research data management practices up2010digschol2011
 
IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...
IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...
IDCC Workshop: Analysing DMPs to inform research data services: lessons from ...
 
DCC 101: Preservation
DCC 101: PreservationDCC 101: Preservation
DCC 101: Preservation
 
Institutional Data Management Blueprint
Institutional Data Management BlueprintInstitutional Data Management Blueprint
Institutional Data Management Blueprint
 
Introduction to research data management; Lecture 01 for GRAD521
Introduction to research data management; Lecture 01 for GRAD521Introduction to research data management; Lecture 01 for GRAD521
Introduction to research data management; Lecture 01 for GRAD521
 
Nataly Zhukova - Conceptual Model for Routine Measurements Analyses in Seman...
Nataly Zhukova - Conceptual Model for Routine Measurements Analyses  in Seman...Nataly Zhukova - Conceptual Model for Routine Measurements Analyses  in Seman...
Nataly Zhukova - Conceptual Model for Routine Measurements Analyses in Seman...
 

Similar to Challenges in medical imaging and the VISCERAL model

Real-time applications of Data Science.pptx
Real-time applications  of Data Science.pptxReal-time applications  of Data Science.pptx
Real-time applications of Data Science.pptxshalini s
 
Large Scale Data Mining using Genetics-Based Machine Learning
Large Scale Data Mining using Genetics-Based Machine LearningLarge Scale Data Mining using Genetics-Based Machine Learning
Large Scale Data Mining using Genetics-Based Machine Learningjaumebp
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxssuser1a4f0f
 
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker, Inc.
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxwahiba ben abdessalem
 
Data_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdfData_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdfvishal choudhary
 
Automating Data Science over a Human Genomics Knowledge Base
Automating Data Science over a Human Genomics Knowledge BaseAutomating Data Science over a Human Genomics Knowledge Base
Automating Data Science over a Human Genomics Knowledge BaseVaticle
 
Pistoia alliance debates analytics 15-09-2015 16.00
Pistoia alliance debates   analytics 15-09-2015 16.00Pistoia alliance debates   analytics 15-09-2015 16.00
Pistoia alliance debates analytics 15-09-2015 16.00Pistoia Alliance
 
Australia's Environmental Predictive Capability
Australia's Environmental Predictive CapabilityAustralia's Environmental Predictive Capability
Australia's Environmental Predictive CapabilityTERN Australia
 
Graham Pryor
Graham PryorGraham Pryor
Graham PryorEduserv
 
Data discovery and sharing at UCLH
Data discovery and sharing at UCLHData discovery and sharing at UCLH
Data discovery and sharing at UCLHJisc
 
High Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run TimeHigh Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run TimeGeoffrey Fox
 
Workflow-Driven Geoinformatics Applications and Training in the Big Data Era
Workflow-Driven Geoinformatics Applications and Training in the Big Data EraWorkflow-Driven Geoinformatics Applications and Training in the Big Data Era
Workflow-Driven Geoinformatics Applications and Training in the Big Data EraIlkay Altintas, Ph.D.
 
Big data's impact on healthcare
Big data's impact on healthcareBig data's impact on healthcare
Big data's impact on healthcareRené Kuipers
 
Brisbane Health-y Data: RedCap
Brisbane Health-y Data: RedCapBrisbane Health-y Data: RedCap
Brisbane Health-y Data: RedCapARDC
 
Introduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia ResearchIntroduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia ResearchDavid De Roure
 
Considerations and challenges in building an end to-end microbiome workflow
Considerations and challenges in building an end to-end microbiome workflowConsiderations and challenges in building an end to-end microbiome workflow
Considerations and challenges in building an end to-end microbiome workflowEagle Genomics
 
Share and Reuse: how data sharing can take your research to the next level
Share and Reuse: how data sharing can take your research to the next levelShare and Reuse: how data sharing can take your research to the next level
Share and Reuse: how data sharing can take your research to the next levelKrzysztof Gorgolewski
 
How to overcome obstacles to data publication: Issues, requirements, and good...
How to overcome obstacles to data publication: Issues, requirements, and good...How to overcome obstacles to data publication: Issues, requirements, and good...
How to overcome obstacles to data publication: Issues, requirements, and good...ariadnenetwork
 
Running Mixed Workloads on Kubernetes at IHME
Running Mixed Workloads on Kubernetes at IHMERunning Mixed Workloads on Kubernetes at IHME
Running Mixed Workloads on Kubernetes at IHMETyrone Grandison
 

Similar to Challenges in medical imaging and the VISCERAL model (20)

Real-time applications of Data Science.pptx
Real-time applications  of Data Science.pptxReal-time applications  of Data Science.pptx
Real-time applications of Data Science.pptx
 
Large Scale Data Mining using Genetics-Based Machine Learning
Large Scale Data Mining using Genetics-Based Machine LearningLarge Scale Data Mining using Genetics-Based Machine Learning
Large Scale Data Mining using Genetics-Based Machine Learning
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptx
 
Docker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce HoffDocker in Open Science Data Analysis Challenges by Bruce Hoff
Docker in Open Science Data Analysis Challenges by Bruce Hoff
 
Data_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptxData_Science_Applications_&_Use_Cases.pptx
Data_Science_Applications_&_Use_Cases.pptx
 
Data_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdfData_Science_Applications_&_Use_Cases.pdf
Data_Science_Applications_&_Use_Cases.pdf
 
Automating Data Science over a Human Genomics Knowledge Base
Automating Data Science over a Human Genomics Knowledge BaseAutomating Data Science over a Human Genomics Knowledge Base
Automating Data Science over a Human Genomics Knowledge Base
 
Pistoia alliance debates analytics 15-09-2015 16.00
Pistoia alliance debates   analytics 15-09-2015 16.00Pistoia alliance debates   analytics 15-09-2015 16.00
Pistoia alliance debates analytics 15-09-2015 16.00
 
Australia's Environmental Predictive Capability
Australia's Environmental Predictive CapabilityAustralia's Environmental Predictive Capability
Australia's Environmental Predictive Capability
 
Graham Pryor
Graham PryorGraham Pryor
Graham Pryor
 
Data discovery and sharing at UCLH
Data discovery and sharing at UCLHData discovery and sharing at UCLH
Data discovery and sharing at UCLH
 
High Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run TimeHigh Performance Data Analytics and a Java Grande Run Time
High Performance Data Analytics and a Java Grande Run Time
 
Workflow-Driven Geoinformatics Applications and Training in the Big Data Era
Workflow-Driven Geoinformatics Applications and Training in the Big Data EraWorkflow-Driven Geoinformatics Applications and Training in the Big Data Era
Workflow-Driven Geoinformatics Applications and Training in the Big Data Era
 
Big data's impact on healthcare
Big data's impact on healthcareBig data's impact on healthcare
Big data's impact on healthcare
 
Brisbane Health-y Data: RedCap
Brisbane Health-y Data: RedCapBrisbane Health-y Data: RedCap
Brisbane Health-y Data: RedCap
 
Introduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia ResearchIntroduction to Big Data and its Potential for Dementia Research
Introduction to Big Data and its Potential for Dementia Research
 
Considerations and challenges in building an end to-end microbiome workflow
Considerations and challenges in building an end to-end microbiome workflowConsiderations and challenges in building an end to-end microbiome workflow
Considerations and challenges in building an end to-end microbiome workflow
 
Share and Reuse: how data sharing can take your research to the next level
Share and Reuse: how data sharing can take your research to the next levelShare and Reuse: how data sharing can take your research to the next level
Share and Reuse: how data sharing can take your research to the next level
 
How to overcome obstacles to data publication: Issues, requirements, and good...
How to overcome obstacles to data publication: Issues, requirements, and good...How to overcome obstacles to data publication: Issues, requirements, and good...
How to overcome obstacles to data publication: Issues, requirements, and good...
 
Running Mixed Workloads on Kubernetes at IHME
Running Mixed Workloads on Kubernetes at IHMERunning Mixed Workloads on Kubernetes at IHME
Running Mixed Workloads on Kubernetes at IHME
 

More from Institute of Information Systems (HES-SO)

Classification of noisy free-text prostate cancer pathology reports using nat...
Classification of noisy free-text prostate cancer pathology reports using nat...Classification of noisy free-text prostate cancer pathology reports using nat...
Classification of noisy free-text prostate cancer pathology reports using nat...Institute of Information Systems (HES-SO)
 
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Machine learning assisted citation screening for Systematic Reviews - Anjani ...Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Machine learning assisted citation screening for Systematic Reviews - Anjani ...Institute of Information Systems (HES-SO)
 
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Exploiting biomedical literature to mine out a large multimodal dataset of ra...Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Exploiting biomedical literature to mine out a large multimodal dataset of ra...Institute of Information Systems (HES-SO)
 
Studying Public Medical Images from Open Access Literature and Social Network...
Studying Public Medical Images from Open Access Literature and Social Network...Studying Public Medical Images from Open Access Literature and Social Network...
Studying Public Medical Images from Open Access Literature and Social Network...Institute of Information Systems (HES-SO)
 
Risques opérationnels et le système de contrôle interne : les limites d’un te...
Risques opérationnels et le système de contrôle interne : les limites d’un te...Risques opérationnels et le système de contrôle interne : les limites d’un te...
Risques opérationnels et le système de contrôle interne : les limites d’un te...Institute of Information Systems (HES-SO)
 
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...Institute of Information Systems (HES-SO)
 
Le système de contrôle interne : Présentation générale, enjeux et méthodes
Le système de contrôle interne : Présentation générale, enjeux et méthodesLe système de contrôle interne : Présentation générale, enjeux et méthodes
Le système de contrôle interne : Présentation générale, enjeux et méthodesInstitute of Information Systems (HES-SO)
 
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...Institute of Information Systems (HES-SO)
 
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
NOSE: une approche Smart-City pour les zones périphériques et extra-urbainesNOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
NOSE: une approche Smart-City pour les zones périphériques et extra-urbainesInstitute of Information Systems (HES-SO)
 

More from Institute of Information Systems (HES-SO) (20)

MIE20232.pptx
MIE20232.pptxMIE20232.pptx
MIE20232.pptx
 
Classification of noisy free-text prostate cancer pathology reports using nat...
Classification of noisy free-text prostate cancer pathology reports using nat...Classification of noisy free-text prostate cancer pathology reports using nat...
Classification of noisy free-text prostate cancer pathology reports using nat...
 
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Machine learning assisted citation screening for Systematic Reviews - Anjani ...Machine learning assisted citation screening for Systematic Reviews - Anjani ...
Machine learning assisted citation screening for Systematic Reviews - Anjani ...
 
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Exploiting biomedical literature to mine out a large multimodal dataset of ra...Exploiting biomedical literature to mine out a large multimodal dataset of ra...
Exploiting biomedical literature to mine out a large multimodal dataset of ra...
 
L'IoT dans les usines. Quels avantages ?
L'IoT dans les usines. Quels avantages ?L'IoT dans les usines. Quels avantages ?
L'IoT dans les usines. Quels avantages ?
 
Studying Public Medical Images from Open Access Literature and Social Network...
Studying Public Medical Images from Open Access Literature and Social Network...Studying Public Medical Images from Open Access Literature and Social Network...
Studying Public Medical Images from Open Access Literature and Social Network...
 
Risques opérationnels et le système de contrôle interne : les limites d’un te...
Risques opérationnels et le système de contrôle interne : les limites d’un te...Risques opérationnels et le système de contrôle interne : les limites d’un te...
Risques opérationnels et le système de contrôle interne : les limites d’un te...
 
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
Le contrôle interne dans les administrations publiques tient-il toutes ses pr...
 
Le système de contrôle interne : Présentation générale, enjeux et méthodes
Le système de contrôle interne : Présentation générale, enjeux et méthodesLe système de contrôle interne : Présentation générale, enjeux et méthodes
Le système de contrôle interne : Présentation générale, enjeux et méthodes
 
Crowdsourcing-based Mobile Application for Wheelchair Accessibility
Crowdsourcing-based Mobile Application for Wheelchair AccessibilityCrowdsourcing-based Mobile Application for Wheelchair Accessibility
Crowdsourcing-based Mobile Application for Wheelchair Accessibility
 
Quelle(s) valeur(s) pour le leadership stratégique ?
Quelle(s) valeur(s) pour le leadership stratégique ?Quelle(s) valeur(s) pour le leadership stratégique ?
Quelle(s) valeur(s) pour le leadership stratégique ?
 
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence ...
 
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
NOSE: une approche Smart-City pour les zones périphériques et extra-urbainesNOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
NOSE: une approche Smart-City pour les zones périphériques et extra-urbaines
 
How to detect soft falls on devices
How to detect soft falls on devicesHow to detect soft falls on devices
How to detect soft falls on devices
 
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSISFUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
 
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLSMOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
MOBILE COLLECTION AND DISSEMINATION OF SENIORS’ SKILLS
 
Enhanced Students Laboratory The GET project
Enhanced Students Laboratory The GET projectEnhanced Students Laboratory The GET project
Enhanced Students Laboratory The GET project
 
Solar production prediction based on non linear meteo source adaptation
Solar production prediction based on non linear meteo source adaptationSolar production prediction based on non linear meteo source adaptation
Solar production prediction based on non linear meteo source adaptation
 
Exploring the New Trends of Chinese Tourists in Switzerland
Exploring the New Trends of Chinese Tourists in SwitzerlandExploring the New Trends of Chinese Tourists in Switzerland
Exploring the New Trends of Chinese Tourists in Switzerland
 
Social Media Data analyzis and Semantics for Tourism Understanding
Social Media Data analyzis and Semantics for Tourism UnderstandingSocial Media Data analyzis and Semantics for Tourism Understanding
Social Media Data analyzis and Semantics for Tourism Understanding
 

Recently uploaded

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfFIDO Alliance
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsUXDXConf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfChristopherTHyatt
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKUXDXConf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureFemke de Vroome
 
THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in  GERMANY for 2024: IPTVreelTHE BEST IPTV in  GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreelreely ones
 

Recently uploaded (20)

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in  GERMANY for 2024: IPTVreelTHE BEST IPTV in  GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreel
 

Challenges in medical imaging and the VISCERAL model

  • 1. Challenges in medical imaging and the VISCERAL model Henning Müller HES-SO & Martinos Center
  • 2. Overview • Systematic evaluations – Information retrieval, industrial challenges • ImageCLEF – 2003-2016 • Challengesin medical imaging and Open Science – Conference and platforms (Kaggle, Topcoder, …) • VISCERAL – “Moving the algorithms to the data and not the data to the algorithms” • Conclusions
  • 3. Systematic evaluations • 1960-: the Cranfieldtests – Test collection, tasks, ground truth – Automatic indexing better than manual terms • 1992-: TREC – Text Retrieval Conference – At NIST, Gaithersburg – Many different tasks over the years • 1999-: CLEF, TRECVid as offspring of TREC • Industrial performance benchmarks – TPC (1988), common transaction processing frame – Supercomputer benchmark (1993), common criteria – … Cleverdon, C. W. (1960). ASLIB Cranfield research project on the comparative efficiency of indexing systems.ASLIB Proceedings, XII, 421-431.
  • 4. • Benchmark on multimodal imageretrieval – Run since 2003, medical task since 2004 – Part of the Cross Language Evaluation Forum (CLEF) • Many tasks related to medical image retrieval – Image classification (modality, body part, …) – Image-based retrieval – Case-based retrieval (finding similar cases) – Compound figure separation – Caption prediction – … • Many old databases remain available, imageclef.org Henning Müller, Paul Clough, Thomas Deselaers, Barbara Caputo, ImageCLEF – Experimental evaluation of visual information retrieval, Springer, 2010.
  • 5. ImageCLEF experiences • Creating a community is important to have a good participation (many groups register to access data) – Workshop to discuss results, evolution of tasks over the years (attracting postgraduate students) • Impact of data sets can be high (see also TREC) – Overview articles are frequently cited, best participant algorithms as well • Large data causes problems in some countries • Hard to make groups collaborate – Evaluate system components • Little interactive evaluation of systems • Not everything is fully reproducible
  • 6. Open Science • Initiatives to share data, tasks and tools – Not only experts, really everyone – More efficient way to do science, no reimplementation – NIH, some journals push for open data, open access • Data papers and executable papers – Full reproducibility • Which is otherwise often not given!! • Challengesas an important way to bring many people into the loop of data science – http://www.challenge.gov/ – Kaggle, TopCoder, … Open Science is the practice of science in such a way that others can collaborate and contribute, where research data, lab notes and other research processes are freely available, under terms that enable reuse, redistribution and reproduction of the research and its underlying data and methods.
  • 7. Challenges in Medical Imaging • Grand Challenges in Medical Imaging – http://grand-challenge.org/ – Including challenges and details on impact & why • 2007: MICCAI workshop on liver segmentation – Including common data • Now most conferences organize many challenge sessions in addition to workshops – MICCAI, ISBI, SPIE, ICPR, … • Problem: still, most publications are on closed data sets, impossible to verify, small, ... – What if all were available on a secure infrastructure?
  • 8. Platforms for ML challenges • Kaggle – Much influence on machine learning challenges – Big commercial factor, giving price money and also hiring good talent – Download data, submit results • TopCoder – Use of code instead of results list – 79,900,000 of price money distributed – Almost 1 million members • Many other exist in specific domains – Sage Bionetworks in the biomedical field
  • 9. Challenges with challenges • Get a large number of participants and different techniques, as there are many burdens – Only one can win in the end, price money • Same conditions for all (computation, bandwidth) • How to distribute very large data sets? • How to deal with confidential/restricteddata? – Medical, commercial data, forbidden data sets • How to deal with quickly changing data? – Data of cell phone providers, Internet companies • Reproducibility – Optimizations on test data, particularly with prices
  • 10. cloud
  • 11. VISCERAL model • VISCERAL – Visual Concept Extraction Challenge in Radiology • “Bringing the algorithms to the data” – Have the data centrally stored, in our case in the cloud (which can be HIPAA compliant) • Three types of challenges – Anatomy segmentations (3x), 20 organs – Retrieval challenge (2x), finding similar cases – Lesion detection challenge (2x), 5 organs • Provide large data sets that are well annotated and can be shared long term – Challenging with IRB approval in three countries
  • 12. Test
  • 14.
  • 15. Test DataTraining Data Participants Organiser Participant Virtual MachinesRegistration System Annotation Management System Analysis System Annotators (Radiologists) Locally Installed Annotation Clients Microsoft Azure Cloud Test Data
  • 16. Silver corpus (example trachea) • Executable code of all participants – Run it on new data, do label fusion Dice 0.85 Dice 0.71 Dice 0.84 Dice 0.83 Participant segmentations Dice 0.92 Silver Corpus
  • 17. Docker vs. Virtual MachinesContainers Bins/Libs VM VS.
  • 18. Evaluation as a Service (EaaS) • Evaluationvia APIs, code, cloud, … • Workshop in Sierre in March 2015 • Many aspects, viewpoints, interests • White paper published – ArXiv • All comments are welcome!!
  • 19. Cloud-based evaluation • Workshop at Martinos Center, Boston, MA, November 2015 • How to run benchmarks on very large data sets in the cloud (reproducibility, motivation) • Many different stakeholders – Scientists, infrastructure providers, companies, funding organizations • Sustainability is a major challenge • Interest of singe persons vs. interests of a domain – Give credit to creators of data and tools • Nature Scientific data
  • 20. Coding4Cancer & others • Challenge on cancer prediction (breast, lung) • Price money for the challenges – Make code open source to be eligible • Commercial medical imaging challenges – Zebra Medical Vision • Large data sets available for research • Use of their infrastructure, only, using Docker – RadLogic • Plug-in concept for algorithms of scientists
  • 21. What is needed now? • Long-term vision of how medical data analysis will develop and how data & tools can be shared – Moon shot initiative on cancer (Biden) • International research infrastructure – Public-private partnerships to make them sustainable, still sharing costs is not clear – Leaving data where produced, moving the code • Incentives to share data and task environments – Those doing the major work should receive credit – More work on those preparing data & tasks • Annotate data, standard formats, support to others
  • 22. Conclusions • Open Science is developingquickly – Potential advantages for all • The medical domain is complicated as data require protection (the more, the bigger) – Particularly for genomics – No duplications limit data exposure • Translational aspects also need to be taken into account (transfer code towards products) – Executable “papers” and available data should help – Objective performance comparison • Challengeswill be part of this ecosystem
  • 23. Contact • More information canbe found at – http://www.imageclef.org/ – http://visceral.eu/ – http://medgift.hevs.ch/ – http://publications.hevs.ch/ • Contact: – Henning.mueller@hevs.ch