Medical 3D data retrieval


Published on

The aim of the 3DOR Workshop series is to stimulate researchers from different fields to present state-of-the-art work in the field. 3DOR 2013 took place as the 6th workshop in this series on May 11, 2013 in Girona (Spain), in conjunction with Eurographics 2013. Prof. Henning Muller presented the keynote talk about Medical 3D data retrieval.

Published in: Technology, Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Medical 3D data retrieval

  1. 1. Medical 3D data retrievalHenning Müller
  2. 2. Where we are2
  3. 3. Who I am •  Medical informatics studies in Heidelberg, Germany (1992-1997)•  Exchange with Daimler Benz research, USA•  PhD in image processing, image retrieval,Geneva, Switzerland (1998-2002)•  Exchange with Monash University, Melbourne, AUS•  Postdoc, assistant prof. in Geneva Universityhospitals in medical informatics (2002-)•  Professor in Computer Science at the HES-SO, Sierre, Switzerland (2007-)3
  4. 4. What we do4
  5. 5. Why we are doing this•  Much imaging data is produced•  Imaging data is very complex•  And getting more complex•  Imaging is essential in diagnosis and treatment planning•  Images out of their contextloose most of their sense•  Clinical data is necessary•  Diagnoses are often not precise•  Evidence-based medicine case-based reasoning5
  6. 6. Topics today63D texture analysisRadiology image retrieval3D organ detection retrieval
  7. 7. Diagnosis aid7
  8. 8. 83D texture
  9. 9. 3D texture vs. 3D objects•  Not the outershape of structuresbut the inner structure•  Hard to visualize, often in several views•  Borders are important because texture needsto be compared inside the same objects•  Exact place can be important, shape of organs as well•  Variations among subjects are very high•  Particularly for healthy tissue, linked to other factors 9
  10. 10. Viewing 3D texture•  Multi-planar rendering•  3D views•  Semi-transparency•  MDs need to get used to it10
  11. 11. A short video11
  12. 12. 4D texture, pulmonary embolism•  4D dual energy CT•  Other 4D data include time components•  Goal is to measure tissue perfusion•  Find whether and which parts of the lung are blocked•  Emergency radiology has very little time for decisions•  Do things that physicians can not do (quantifying)12Material Attenuation Coefficient vs keV0.111010040 50 60 70 80 90 100 110 120 130 140Photon Energy (keV)m(E)(cm2/mg)IodineWater80 keV 140 keV
  13. 13. Texture characteristics•  Localized characterizations of scales and localorientations are important for textures •  Often choices for characterization are arbitrary•  Riesz transform implements nth order directionalderivatives at multiple scales•  Linear combinations of the filters allows to create texturetemplates to detect specific textures13local orientationscale
  14. 14. Combining filters for detection14
  15. 15. Texture templates15Scale and orientation modeling
  16. 16. Templates for lung textures16Signatures allow for visually checking themodeled information
  17. 17. 3D texture templates17•  Vertical planes•  3D checkerboard•  3D wiggledcheckerboard
  18. 18. 18Radiology image retrieval
  19. 19. Khresmoi application•  Mixing multilingual data from many resourcesand semantic information for medical retrieval•  LinkedLifeData19
  20. 20. The informed patient20
  21. 21. Identifying user requirements•  Surveys among several radiologists•  Also GPs and patients•  Observing diagnosis processes•  Analyzing search log files•  Eye tracking on a radiology viewing station•  What are information needs and what aretasks that are hard and where help is needed?•  Test the developed systems in user studies•  Analyze feedback•  Record the system use for understanding problems21
  22. 22. Eye tracking22
  23. 23. Recording user tests23
  24. 24. Data used for ParaDISE•  Scientific data of the biomedical literature •  600’000 articles and 1.6 mio figures of the open accessliterature (4 mio images if separating compound figures)•  Public data source but only 2D data•  Clinical data form the Vienna Medical Universityimage archive•  5TB of data of two consecutive months•  Radiology reports for each case•  Private data source, so access only with password•  Link medical cases with similar cases from theliterature based on image data and text 24
  25. 25. Connecting different data levels25EHR, PACS
  26. 26. Context is important (25 yo vs. 88 yo)!26
  27. 27. ParaDISE architecture27
  28. 28. Interfaces for search: radiology28
  29. 29. Searching text and images29
  30. 30. Playful interfaces30
  31. 31. Semantics in radiology•  Map text to ontologies such as RadLex•  LinkedLifeData is a huge knowledge source•  Permits synonyms, hypernyms, several languages, etc.•  Use structure and links•  Anatomic regions are linked with modalities•  Specific findings (such as fibrosis or micro nodules) arelinked with anatomic regions and also with modalities•  Use visual information and semantics to extractinformation from the images•  Detecting modality and anatomic region from the images31
  32. 32. Visual words in 3D•  Visual features can depend strongly on theapplication domain•  Many benchmarks show that using a large feature setand then reducing it is often performing best•  Early vs. late fusion is not absolutely clear•  Modeling the feature space based on what isactually present in the images•  Use techniques known in text retrieval•  Removing stop words•  Latent semantic analysis, synonyms, etc.32
  33. 33. Visual words and region detection33
  34. 34. 343D organ detection and retrieval
  35. 35. Goals•  Organize benchmarks on image analysis invery large data sets 10 TB•  Identifying anatomic structures•  Finding similar cases•  Many challenges•  10 TB can not be downloaded and also sending harddisks to participants is not easy•  How can we obtain a solid ground truth to comparealgorithms on a large scale•  Scalability is a major factor, so efficiency of algorithms35
  36. 36. Cloud-based benchmarking36Test
  37. 37. Creation of the database37
  38. 38. Annotated data (20-55 organs)38
  39. 39. Gold and silver corpus•  Training data will be fully annotated manually•  Part of the test data will also be annotatedmanually (gold corpus)•  Does not scale to 10 TB•  Based on results and evaluation a larger silvercorpus will be created•  Based on high quality systems on gold corpus viamajority voting•  Annotate organs where there are strong differencesbetween participating systems•  Maximize the gain of manual annotation39
  40. 40. Participation is open!!•  Data will be released this summer (August 1)••  Registration system is almost finished•  Participants will receive a virtual machine theycan configure to their needs in the Azure cloud•  Linux, Windows, …•  First phase on training data (can be downloaded ifreally needed), access via Azure API•  Then the organizers will take over the virtualmachine and run things on test data•  Detailed protocol is being defined40
  41. 41. MICCAI workshop••  Linked to the MICCAI conference•  Medical Image Computing for Computer Assisted Intervention•  Scientific part of the workshop and part relatedto VISCERAL•  Big data in medical image analysis•  Discuss the challenges and orient the work of theproject towards real challenges•  Community effort 41
  42. 42. Conclusions•  Medical imaging offers many interestingopportunities in multidimensional data analysis•  Challenges remain such as data confidentiality,3D/4D visualization, small regions of interest•  Images should never be regarded out of their context•  Application-driven approaches can really helpphysicians and get support from them•  Creating (large) data sets for developing toolsrequires much effort•  Open data sets are really important to advance sciece42
  43. 43. Contact and more information•  More information can be found at •••••  Contact:•  Henning.mueller@hevs.ch43