The document proposes an open-source framework for evaluating and comparing nonlinear image registration algorithms in a standardized way. It describes the need for such a framework given challenges like the lack of direct ground truths and gold standard methods. The framework includes public data sets, synthetic deformations, registration algorithms, and quantitative measures. A sample evaluation using this framework to evaluate an intra-subject thorax CT registration algorithm is also presented and discussed.

1. A Framework for Comparison and Evaluation of Nonlinear Intra-Subject Image Registration Algorithms Martin Urschler, Stefan Kluckner, Horst Bischof Institute for Computer Graphics and Vision, Graz, University of Technology, Austria

2. Motivation Validation very difficult! Lack of direct ground truth Lack of gold standard methods - Highly ill-posed problem Large space of possible solutions We claim: Standardizing Evaluation Protocols at least as important as Developing Novel Methods! -> Put framework to discussion: - Open-source community effort - Instantiate framework by showing sample evaluation Presented algorithm at last years MICCAI. Nonlinear Image Registration

3. Contents Problem Definition Related Work & Similar Efforts Open Science Nonlinear Registration Evaluation Framework Sample Evaluation – Intra-subject Thorax CT Conclusion & Outlook

4. The Nonlinear Registration Problem „ Find a deformable mapping h(x) aligning moving and fixed image such that a defined similarity criterium is minimized.“ h(x) Base Illustration taken from ITK Software Guide. SSD, NCC, Mutual Information, … B-Spline, Thin-Plate Spline, Def Field, … Gradient Descent, BFGS, … Intensity-Based, Feature-Based, …

5. Medical Applications of Nonlinear Image Registration Angiography Studies Anatomy & Function Perfusion /Ventilation Correction of Motion Artifacts Studies of Shape Variation Segmentation by Atlas Registration Surgical Planning … Lung Perfusion for Pulmonary Embolism Detection [Wildberger et al.] Validation Study: Atlas-based Brain Volume Segmentation from MRI Images [Ng02] – taken from ITK Documentation Inter- & Intra-Modality Inter- & Intra-Subject No Ground Truth or Gold Standards Highly Ill-Posed How should we reach clinical acceptance?

6. Related Work on Evaluation (Frameworks) There exist some good ideas for evaluation in (Medical) Computer Vision! VALMET Segmentation Evaluation [Gerig et al] Community would benefit from open-source implementation! MICCAI 2007 Segmentation Challenge Workshop Multi-View 3D Reconstruction [Seitz et al] Stereo Reconstruction [Scharstein et al] Middlesbury Retrospective Evaluation of Intersubject Brain Registration [Hellier et al] Retrospective Rigid Registration Evaluation Project [West et al] NIREP (Inter-subject Brain-Registration Evaluation) [Christensen et al] Validation of Nonrigid Image Registration Using FEM [Schnabel et al] Segmentations are compared!

7. Open Science Evaluation Framework Algorithm Evaluation vs. Validation [Hellier et al] Specific Problems: Lacking ground truth, ill-posedness, Lack of gold standard General Problems: Noise, Partial Volume Effect, Interpolation, Numerics Modular Evaluation Framework Open-source, open-access, open-data, open protocols Building Blocks: Registration Algorithms Public domain data sets Synthetic Deformations Similarity Measures Python Framework as Glue

8. Open Science Evaluation Framework

9. Synthetic Deformations Used Simple synthetic transformation models Regular grid with random deformations & TPS Uniform periodic cosine transformation Tailored to breathing difference registration: Simulated Breathing Model Synthetic Airway Tree Movement from Manual Correspondences … Increase number of models Pool of synthetic transformations defined by pool of algorithms („bronze standard“ Glatard et al – MICCAI 2006) Community effort needed

10. Data Sets and Algorithms Public domain data sets from National Library of Medicine Dataset Collection (currently offline) MIDAS data collection project National Lung Cancer Archives (NCIA) ITK Algorithms Demons Symmetric Demons Level Set Motion Curvature Fast Block Matching (Workshop Contribution) Diffeomorphic Demons (Workshop Contribution)

11. Quantitative Measures Measures on Displacement Fields RMS of displacement field differences MAD of displacement field differences MAX of displacement field differences Jacobian determinant of displacement field Measures on fixed and warped moving image Clamped RMS intensity differences MAD intensity differences MAX intensity differences Normalized Mutual Information Edge Overlap IMHO only first group says something about registration performance!

12. A Sample Evaluation Purpose: Evaluate intra-modality thoracic CT registration subject to breathing differences 2 data sets 256^3 NLM NormalChestCTNoContrast NCIA LIDC 30047 Single choice of algorithm parameters 64 bit Opteron with 2.4GHz and 8GB RAM

13. A Sample Evaluation Simulated Data Difference to Original Standard Demons

14. A Sample Evaluation - Results Quantitative measures Large number (see paper) Quantities are single numbers Problematic? Standard Demons & Diffeomorphic Demons very well Algorithm Parameter Studies Visual Results (Problem Cases) Level Set motion -> artifacts Fast Block Matching -> Implementation Issues (Test Framework) Symmetric Demons

15. Conclusion Open Science Evaluation Framwork presented Intra-Modality Sample Evaluation shown Results should not be seen as final algorithm quality statements Framework has to grow… Useful for comparing & testing algorithms Useful for parameter studies A small step towards establishing standardized protocol to gain clinical acceptance…

16. Further Work Common Web Repository (Hosted by Kitware?) Upgrade by community effort More algorithms More quality measures Segmentation based -> Problems for general applicability More synthetic deformations, including noise models Extension to inter-modality, inter-subject problems Cooperate with NIREP? How to prevent „evaluation framework tuning“?

17. Thank you for your attention!