Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learning-based vertebra segmentation, identification and partioning


Published on

The spine is visualized on many CT and MR exams, including thorax and abdomen scans that were originally not intended for spine imaging. Because these often cover several but not all vertebrae, it is difficult to make strong assumptions for automatic analysis. Challenges are therefore the unknown number of target structures (vertebrae) in the image, their anatomical identification (which vertebrae are visible? must not assign the same label to two vertebrae) and that some biomarkers are related only to part of the vertebrae, often the vertebral body. This talk covers an instance segmentation approach for vertebra detection, segmentation, and anatomical identification, and a partitioning approach to separate vertebral body and arch based on thin-plate spline surfaces positioned by a convolutional neural network.

Published in: Health & Medicine
  • I'm So Sorry They Lied To You� ■■■
    Are you sure you want to  Yes  No
    Your message goes here

Learning-based vertebra segmentation, identification and partioning

  1. 1. Learning-based vertebra segmentation, identification and partitioning DDH Open, 3 April 2019 Nikolas Lessmann
  2. 2. Extrapulmonary diseases in lung cancer screening Cardiovascular disease Osteoporosis Lung cancer
  3. 3. Osteoporosis
  4. 4. Potential new biomarkers • Height loss compared to previous screening round? • Density loss over time? • … CT-biomarkers for osteoporosis Bone mineral density (usually done with dual energy X-ray absorptiometry or QCT) Relative height loss (compared with neighboring vertebrae)
  5. 5. • Visible in many different types of scans (different FOV, context) • Difficult to distinguish (anatomical label) • Difficult to separate from each other in low-resolution scans • How reliable are density measures in low-SNR scans? Challenges for automatic vertebra analysis in CT
  6. 6. Part 1 Vertebra segmentation
  7. 7. Vertebrae often need to be separated not only from the background, but also from each other Generic algorithm should not make assumptions about: • How many vertebrae are visible • Which vertebrae are visible • Their location within the image = instance segmentation An instance segmentation problem
  8. 8. Dedicated loss functions, e.g., arXiv:1708.02551 Detection & segmentation pipelines (e.g., Mask RCNN, based on region proposals) Recurrent attention (instance by instance segm.) Instance segmentation with CNNs arXiv:1703.06870 arXiv:1605.09410
  9. 9. • Only one class of objects (vertebra) • Size and shape are relatively consistent • Instances are located in a column, one next to the other A restricted instance segmentation problem
  10. 10. Combined detection and segmentation • Segment one vertebra • Look for the next in the neighborhood The network needs a memory, like recurrent networks, to remember which vertebrae have already been found Iterative instance segmentation
  11. 11. ROI (field of view of CNN) moves through the image volume
  12. 12. Patch-based iterative instance segmentation Image Memory Segmentation 1. Iteration
  13. 13. Patch-based iterative instance segmentation Image Memory Segmentation 2. Iteration
  14. 14. Patch-based iterative instance segmentation Image Memory Segmentation 3. Iteration
  15. 15. • Use prior knowledge that vertebrae are located next to each other • Search for spine, then follow the spine Traversal along the spine
  16. 16. I = Image patch M = Memory patch S = Segmentation mask L = Anatomical label Prediction of label 1-24, global probabilistic model to find most plausible labeling C = Completeness score Vertebrae may be visible only partially in the image, relevant for following automatic steps Network architecture
  17. 17. Label regression with probabilistic interpretation Prediction: 14.8 Interpretation: • 80% probability label 15 • 20% probability label 14 Maximum-likelihood approach Anatomical labeling Mean certainty per labeling sequence 0.24 0.56 0.89
  18. 18. • Some vertebrae are often only partially contained in the scan • For some applications, that is important to know – biomarkers computed based on incompletely visible vertebrae can be unreliable Completeness score
  19. 19. Sum of three error terms • Segmentation: Minimize FP and FN voxels • Labeling: L1 norm • Completeness: Binary crossentropy Loss function Weight map to improve segmentation accuracy at the boundary between vertebrae
  20. 20. • Adam optimizer with fixed learning rate + high momentum (0.999) • Single patch per iteration (no mini-batches)  each patch has ~2,000,000 voxels • Also patches without vertebral bone are presented to the network • Memory state derived from reference segmentation • Data augmentation: elastic deformations, random Gaussian noise and smoothing, random cropping Training details
  21. 21. • Manual and semi-automatic annotation of NLST (low-dose chest CT) scans • Several datasets from vertebra segmentation challenges Evaluation Lumbar spine CT Chest CT Lumbar spine MR (VB only)
  22. 22. Evaluation
  23. 23. Part 2 Vertebra partitioning
  24. 24. • We can do vertebra segmentation in CT • We can do vertebral body segmentation in MR • We cannot do vertebral body segmentation in CT… (New training set? Transfer learning? GANs?) Motivation
  25. 25. Alternative idea: Let a CNN fit a continuous surface that represents the boundary Surface fitting
  26. 26. • Splines are defined by a set of control points • Splines are smooth between control points Think of a stick or a thin metal sheet that is fixated at several points and resists bending due to its rigidity  CNN could predict location of control points Fitting a thin-plate spline with a CNN
  27. 27. How to backpropagate through this? Flowchart
  28. 28. Differentiable TPS surface? Surface “height” across grid Configuration of grid points w.r.t. control points Configuration of control points w.r.t. control points Control point “heights” (network predicition)
  29. 29. If we had vertebra and vertebral body segmentations of the same subjects (and maybe even the same scans), we could train a simpler approach … We have segmentation masks from • Different subjects • Different scans • Different modalities even How to train without matching segmentation pairs?
  30. 30. Segmentation masks do not match, but we can learn something from this data – statistical approach, capture plausible shapes • Do not use image intensities at all • Train convolutional autoencoder on vertebral body segmentation masks and use as out-of-distribution classifier Shape encoding with autoencoders
  31. 31. Flowchart Trained with vertebra segmentation masks from CT scans (around 600 NLST scans, > 6000 vertebrae) Previously trained with vertebral body segmentation masks from MR scans (around 300 UMCU scans, >1000 vertebral bodies)
  32. 32. Reconstruction error + several regularizing terms • Volume penalty: Volume Vertebra – Volume Vertebral Body • “Fights” the reconstruction loss by pushing the surface to the right (posterior) • Empty segmentation masks are perfectly reconstructed by the AE… • Curvature penalty: abs(det(Hessian)) • Prevents that surface is very wobbly (seems not really necessary) Loss function
  33. 33. Loss function
  34. 34. Manual annotation of vertebrae and vertebral bodies in 5 chest CT scans Results
  35. 35. Measurements of the height of the vertebral bodies and correlation with extracted volume Average of 3 measurements in 389 scans Results
  36. 36. We can… • Find and segment the vertebrae • Label them anatomically • Determine if they are completely or partially visible • Partition them into body and posterior elements Next step: Application (fracture prediction, fracture classification, …) Also possible: use partitioning results to train direct vertebral body segmentation Conclusions and future work
  37. 37. Thank you!