BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis
•Download as PPTX, PDF•
0 likes•78 views
In this work, we describe the field research, design, and comparative deployment of a multimodal medical imaging user interface for breast screening. The main contributions described here are threefold: 1) The design of an advanced visual interface for multimodal diagnosis of breast cancer (BreastScreening); 2) Insights from the field comparison of Single-Modality vs Multi-Modality screening of breast cancer diagnosis with 31 clinicians and 566 images; and 3) The visualization of the two main types of breast lesions in the following image modalities: (i) MammoGraphy (MG) in both Craniocaudal (CC) and Mediolateral oblique (MLO) views; (ii) UltraSound (US); and (iii) Magnetic Resonance Imaging (MRI).
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis
Similar to BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis (20)
Recently uploaded
Recently uploaded (20)
BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis
- 1. AVI 2020 International Conference on Advanced Visual Interfaces
- 2. BreastScreening On the Use of Multi-Modality in Medical Imaging Diagnosis
- 3. Authors Francisco M. Calisto ISR-Lisboa Instituto Superior Técnico Universidade de Lisboa Nuno Nunes ITI Instituto Superior Técnico Universidade de Lisboa Jacinto C. Nascimento ISR-Lisboa Instituto Superior Técnico Universidade de Lisboa
- 4. Motivation Why Cancer? Why Medical Imaging? 1.
- 5. 9.6 million deaths in 2018 5 Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A. and Jemal, A., 2018. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians, 68(6), pp.394-424.
- 6. 30% - 50%of cancers can currently be prevented by avoiding risk factors and implementing existing evidence-based prevention strategies. Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A. and Jemal, A., 2018. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians, 68(6), pp.394-424.
- 7. Early Detection Cancer mortality can be reduced if cases are detected and treated early.
- 8. Early Diagnosis » Care Components » Awareness » Accessing » Clinical Components » Evaluation » Diagnosis » Staging » Treatment Access There are two components of Early Detection Screening Screening aims to identify individuals with abnormalities suggestive of a specific cancer or pre-cancer.
- 9. HCI in Healthcare Diagnostic systems have also been studied under the HCI field. 2.
- 10. HCI in Healthcare » Supporting the image search UX through novel UIs [1, 2] » Medical imaging technologies to support radiologists [3] » Systems that assist radiologists in image interpretation [4, 5] 1 Koutsabasis, P. and Domouzis, C.K., 2016, June. Mid-air browsing and selection in image collections. In Proceedings of the International Working Conference on Advanced Visual Interfaces (pp. 21-27). 2 Lee, B., Srinivasan, A., Stasko, J., Tory, M. and Setlur, V., 2018, May. Multimodal interaction for data visualization. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (pp. 1-3). 3 Woźniak, P., Romanowski, A., Yantaç, A.E. and Fjeld, M., 2014, October. Notes from the front lines: lessons learnt from designing for improving medical imaging data sharing. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (pp. 381-390). 4 Cai, C.J., Winter, S., Steiner, D., Wilcox, L. and Terry, M., 2019. " Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction, 3 (CSCW), pp.1-24. 5 Oram, L., MacLean, K., Kruchten, P. and Forster, B., 2014, June. Crafting diversity in radiology image stack scrolling: control and annotations. In Proceedings of the 2014 conference on Designing interactive systems (pp. 567-576).
- 11. Medical Imaging How to improve and support medical imaging? 3.
- 12. Healthcare Systems Machines that can learn with new data from clinicians’ experience.
- 13. Healthcare Systems But first we need to create a solution to generate information to the algorithms.
- 15. Multimodality Annotating System + Magnetic Resonance Imaging (MRI) UltraSound (US) MammoGraphy (MG) +
- 19. Multimodality Annotating System > Magnetic Resonance Imaging (MRI) UltraSound (US) MammoGraphy (MG) > Lesions
- 20. Multimodality Annotating System Interaction A system for a multimodal interaction with MG, US and MRI medical images on a multi- screen and multi- environment. Visualization The indistinct visualization of cluttered breast lesions. Big Data A platform for generating and managing big data on medical images.
- 21. Methods Human-centered study to understand clinician’s needs. 4.
- 22. 31 clinicians contributed to our study
- 23. 6 institutions of healthcare in Portugal.
- 24. Six Institutions » 8 clinicians Hospital Fernando Fonseca » 12 clinicians IPO-Lisboa » 1 clinician Hospital de Santa Maria » 8 clinicians IPO-Coimbra » 1 clinician Madeira Medical Center » 1 clinician SAMS
- 25. > 16 hours of observationsand interviews
- 26. 566 images acquiredfrom Hospital Fernando Fonseca
- 27. Design Goals The technical design challenges lead to a set of design issues. 5.
- 28. Design Issues » Medical imaging structure trade-offs » Radiology room temporal awareness [6] » Image segmentation support » Radiologists system trust 6 Nascimento, J.C. and Carneiro, G., 2019. One shot segmentation: unifying rigid detection and non-rigid segmentation using elastic regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence.
- 29. Five Design Goals » DIM Design around and for Medical Imaging » TAS Temporal Awareness Support » ISS Image Segmentation Support » SMS Several Modalities Support » GTO Growing Trust Overview
- 30. Design around and for Medical Imaging Taking into account the heterogeneous nature of medical imaging to leverage its contextual richness.
- 31. Temporal Awareness Support Observing how the radiology workflow events, treatments, and problems progressed over time.
- 32. Image Segmentation Support The overview of image details allowing a more accurate diagnostic.
- 33. Several Modalities Support Enable the view and the process of diagnostic imaging studies.
- 34. Growing Trust Overview Allowing an efficient triangulation via visualizations, image processing between medical images and available features.
- 35. BreastScreening A medical imaging visualization proof-of-concept to be evaluated in a realistic clinical scenario. 6.
- 36. User Interface
- 37. User Interface
- 38. User Interface
- 39. Procedures Evaluation of BreastScreening in real-world conditions. 7.
- 40. Cond. C1 Single-Modality Two Conditions Cond. C2 Multi-Modality 40
- 41. Three Patients » P1 Low (BI-RADS < 2) » P2 High (BI-RADS > 3) » P3 Medium (1 > BI-RADS > 4)
- 42. BREAST SEVERITY BI-RADS Meaning 0 Needs more information (more exams or waiting for more exams) 1 Negative 2 Benign 3 Probably Benign 4 Suspicious 5 Highly suggestive of malignancy 6 Known biopsy-proven malignancy
- 43. Quantitative Analysis Four relations emerged from our analysis. 8.
- 44. Quantitative Analysis » SUS Scores vs SUS Questions » Intern » Junior » Middle » Senior » NASA-TLX » Single-Modality vs Multi-Modality » Time vs Number of Clicks » P1 » P2 » P3 » BI-RADS Classification
- 45. SUS Scores vs SUS Questions Participants adopting the Multi-Modality condition obtained higher SUS scores than those using the Single-Modality condition.
- 46. Workload In general, the workload improved for Mental Demand, Physical Demand, Temporal Demand, Performance, Effort and Frustration while using the Multi- Modality setup.
- 47. Time vs Number of Clicks The time per image was reduced. The number of clicks was also improved. Which are directly related to the number of annotations on the lesion.
- 49. Qualitative Analysis Clinicians were invited to give some feedback about the UI during the open interviews. 9.
- 50. “ The system will be a great asset for us.
- 51. “ I would like to frequently use your system on my daily practice.
- 52. Conclusion The conclusions of our work. 10.
- 53. Contributions » Identifying the main clinical workflow issues, the interaction cognitive load challenges and the opportunities; » Establishing a set of design goals for medical imaging design; » The design, reflections and in-situ evaluation of BreastScreening supporting the clinical translation; » The impact evidence of Multi-Modality in diagnosing and severity classification of breast lesions;
- 54. Contributions The system can lead to more efficient and accurate clinical diagnosis.
- 55. Information Francisco Maria Calisto E-Mail: francisco.calisto@tecnico.ulisboa.pt Academic Webpage: web.tecnico.ulisboa.pt/francisco.calisto Lab Webpage: welcome.isr.tecnico.ulisboa.pt
- 56. Thank you!
- 57. “The paradigm shift of the ImageNet thinking is that while a lot of people are paying attention to models, let's pay attention to data. Data will redefine how we think about models. - Li Fei-Fei
- 58. Q&A
Editor's Notes
- At the International Conference on Advanced Visual Interfaces
- We are presenting the paper tiltled as BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis, a multimodality platform for annotating medical images
- Authors...
- Why cancer? Why medical imaging?
- Cancer is a leading cause of death worldwide, accounting for an estimated 9.6 million deaths in 2018.
- Between 30–50% of cancers can currently be prevented by avoiding risk factors and implementing existing evidence-based prevention strategies.
- The cancer burden can also be reduced through early detection of cancer and management of patients who develop cancer. Cancer mortality can be reduced if cases are detected and treated early.
- There are two components of early detection. Early diagnosis and screening. On early diagnosis, we have care and clinical components, as well as treatment access. Pair to this, screening identifies patients abnormalities. HCI plays a fundamental step on supporting early diagnosis and screening.
- Several diagnostic systems has been studied under the HCI field.
- There are a large amount of publications in the field of HCI. However, our solution is different from previously works. Namely, three issues highlight our proposal. First, the use of multimodal images in breast diagnosis. Second, the possibility to provide radiologists to have the annotations in all modalities. Third, we also provide a rich sample of medical images and a dataset of annotations.
- How to improve and support medical imaging?
- The dataset is extremely important since recently there is a bunch of machine learning algorithms that use rich annotation datasets. In this case, we are able to provide new data to the machine learning algorithms.
- Specifically, we created a solution to generate information to the algorithms.
- Our workflow is as follows. First, we present the medical images to clinicians. Then, we provide facility to image visualization. And finally, clinicians are able to provide medical annotations in all modalities.
- You may wonder: why multimodality? Because there are two situations: (1) when we have adipose tissue, we just need MG; however, (2) when the tissue is dense, MG is not enough for a proper diagnosis, so it is imperative the use of a multimodality strategy. In this case, for dense breast, MG, US and MRI. Sometimes, clinicians perform a kind of loop for the inspection of lesions. It is why is very important to have some similar proposal as we do in this work.
- There are two types of lesions...
- There are masses, with small contours of the tissue.
- And there are microcalcifications which are little points on the tissue.
- So we want to collect this information. These are some examples of mass on MG, US and MRI. And we aim to locate and to store this information in our dataset.
- Our multimodality system provides interaction, easy visualization, as well as big data generation and management. The interaction is made in terms of multimodality. The visualization is accomplish with cluttered breast lesions. Last, we obtain a rich multimodal dataset with annotations.
- In our work, we propose...
- A human-centered design and participatory evaluation with 31 clinicians
- Six institutions of healthcare in Portugal.
- The clinicians are distributed according with the following: 8 clinicians from Hospital Fernando Fonseca; 12 clinicians from IPO-Lisboa; 1 clinician from Hospital de Santa Maria; 8 clinicians from IPO-Coimbra; 1 clinician from Madeira Medical Center; and 1 clinician from SAMS.
- The recruited specialists are in advanced career positions and were observed and interviewed in a semi-structured fashion. Each session took approximately 30 minutes. Making a total of more than 16 hours of observations and interviews.
- Screening of 566 acquired images from Hospital Fernando Fonseca. However, we have more than 4000 images now.
- The technical design challenges lead to a set of design issues.
- Combining the clinical context and the technical design challenges lead to a set of design issues, including: medical imaging structure trade-offs, radiology room temporal awareness, image segmentation, and radiologists system trust.
- Based on these design issues, we define five design goals: (1) Design around and for Medical Imaging; (2) Temporal Awareness Support; (3) Image Segmentation Support; (4) Several Modalities Support; and (5) Growing Trust Overview. Now let me explain each of them.
- First, Design around and for Medical Imaging, we are taking into account the heterogeneous nature of medical imaging to leverage its contextual richness.
- Second, Temporal Awareness Support where we want to observe how the radiology workflow events, treatments, and problems progressed over time.
- Fourth, Image Segmentation Support is the overview of image details allowing a more accurate diagnostic. Namely, reducing the number of false-positives classification (BI-RADS) of the lesion, as well as improving the number of clicks when performing the lesion delineation, i.e., segmentation.
- Fifth, Several Modalities Support to enable the view and the process of diagnostic imaging studies, including MG, US and MRI medical imaging modalities.
- Six and final, Growing Trust Overview to allow an efficient triangulation via visualizations, image processing between medical images and available features, i.e., annotations of masses and calcifications.
- To validate the proposed design goals, we created BreastScreening, as a medical imaging visualization proof-of-concept to be evaluated in a realistic clinical scenario. In our design explorations, we sought to integrate several image modalities and visualization to support insight.
- The User Interface (UI) consists of two main components...
- 4. List of Patient Views. where we have a 4.5 Study List of patients that the interface includes on the top left corner.
- And also the 5. Medical Imaging Diagnosis Views. In this case, we have the 5.1. Viewports where clinicians can interact with MG, US and MRI using a set of features according to the 5.2. Toolbars. Finally, clinicians can change the respective modality on the 5.3. Modality Selection freely.
- The procedures are as follows...
- You may think that the Multi-Modality may overload the workflow of clinicians. To provide evidence of the contrary, we intent to test two conditions: Cond. C1 - Single-Modality, and Cond. C2 - Multi-Modality. We want to demonstrate experimentally that Multi-Modality is as fast as Single-Modality and as accurate. This means that clinicians do not have to work more while dealing with Multi-Modality.
- We collected complete imaging exams for three patients (P1, P2 and P3) on all possible modalities (MG, US and MRI).
- BI-RADS
- Four relations emerged from our analysis.
- Here we have used: a) differences between SUS Scores and SUS Questions among clinical experience (i.e., Intern, Junior, Middle, and Senior); b) the workload measurements of both Single-Modality and Multi-Modality views; c) the relation between Time and Number of Clicks, clustering by Patient (i.e., P1, P2 and P3); and d) the accuracy.
- Participants adopting the Multi-Modality condition obtained higher SUS scores than those using the Single-Modality condition.
- In general, the workload improved while using the Multi-Modality setup.
- Despite the overall amount of time has increased, the time per image was reduced. The number of clicks was also improved. Therefore, we can argue that in the same amount of time, clinicians are providing more clicks. Which is directly related to the number of annotations in the lesion.
- For the BI-RADS classification, it is clear that the Multi-Modality performs better, since the most severe BI-RADS exhibits the smaller mean and variance in the most of the cases. Also note that for the most problematic patient (in this case P2 scored with BI-RADS = 5) the multi-modal largely outperforms the Single-Modality setting.
- Clinicians were invited to give some feedback about the UI during the open interviews. We received several positive comments regarding our BreastScreening system.
- At the end, several clinicians (19/31) answered that the assistant will be an asset of an immense importance for the current RR situation: “The system will be a great asset for us” (C6).
- Another positive answer was the one related to the frequency of use (28/31) for this new assistant regarding the current system used by the clinicians on the daily practice: “I would like to frequently use your system on my daily practice” (C1).
- The conclusions of our work.
- We Identify the main clinical workflow issues, the interaction cognitive load challenges and the opportunities. Our work is establishing a set of design goals for medical imaging design. Also, we provide the design, reflections and in-situ evaluation of BreastScreening supporting the clinical translation. And finaly, we show the impact evidence of Multi-Modality in diagnosing and severity classification of breast lesions with 31 radiologists in six different clinical institutions.
- Our results show that the system can lead to more efficient and accurate clinical diagnosis.
- Francisco Maria Calisto
- Thank you!
- “The paradigm shift of the ImageNet thinking is that while a lot of people are paying attention to models, let's pay attention to data. Data will redefine how we think about models.” - Li Fei-Fei
- Q&A time.