Thesis (English):
http://handle.ncl.edu.tw/11296/ndltd/08095156356149825986
General Information:
This is the slides for my MSc thesis defense, which introduces an algorithm that automatically analyzes cardiac magnetic resonance scans for extracting left ventricular cardiac parameters. These parameters are crucial for determining whether or not a patient suffers from certain forms of cardiovascular diseases, such as ischaemic heart disease and hypertrophy.
The performance of the algorithm in terms of segmentation accuracy (APD) outperforms all other similar reported algorithms (as of 2014) that also use the same CMR scan database* by at least 15%. This means more accurate cardiac parameters can be obtained using my proposed algorithm.
*Evaluation is done using the Cardiac MR Database provided by Sunnybrook Health Science Center, Toronto, Canada.
Less Is More: Utilizing Ballerina to Architect a Cloud Data Platform
Automated quantitative assessment of left ventricular functions by MR image segmentation
1. Automated left ventricle segmentation
in SAX CMR images using cost-volume filtering (CVF)
and novel myocardial contour processing framework
Graduate Institute of Communication Engineering
College of EECS, NTU
Oct 22, 2014
Speaker: An-Cheng Chang, MSc student
2. What is CMR segmentation and why?
• Tracing the chamber volume gives insight into
how well the heart functions.
• CMR segmentation is involved as a major part of
the analysis.
2
Left ventricle (LV)
Heart
Cardiac Magnetic Resonance (CMR) images
Epicardium
border
Endocardium
border
Long axis
Apex
Base
Long axis
3. Objective
Automatically trace the LV endo∙card∙ium border
3
Left ventricle (LV)
Endocardium
border
heartinner tissue
(it is not a trivial task)
4. Difficulties in CMR segmentation
• Endocardium border is often obscured by papillary muscles and trabeculae
carneae. (Fig A)
• Variation between individuals. Wide pathological variations. (Fig B)
• Low image quality: noises and distortions, e.g., field inhomogeneity, partial
volume effect. (Fig C and Fig D)
4
C. Artifacts
(field inhomogeneity)
B. Wide subject
variations
D. Artifacts
(partial volume)
A. Endocardium border
obscured by PMTC
Ground truth
Auto
5. Related works on CMR segmentation
Ngo et al., IEEE ICIP 2013
• Deep learning (pre-trained) + level set
• Algorithm initialize by cropping ROI
5
Cropped by operator
Pre-trained
deep learning
network
Cropped by operator
Hu et al., Magn Reson Imaging (Elsevier, 2013)
• GMM + dynamic programming
• Algorithm initializes by cropping ROI
Method
SR (%)
Mean(StD)
APD (mm)
Mean(StD)
DM
Mean(StD)
Hu 2013 91.1(9.4) 2.24(0.40) 0.89(0.03)
Ngo 2013 97.9(6.18) 2.08(0.40) 0.90(0.03)
Ours 94.1(6.1) 1.75(0.42) 0.91(0.03)
Ours (2014)
• CVF + contour processing
• Algorithm initializes by one click on the LV
6. Result highlight
6
Auto contours rejected: 0 out of 18
Mean error: 2.59mm (+0%)
EF underestimated by 4%
Subject SC-HYP-40; hypertrophic heart.
Auto contours rejected: 7 out of 18
Mean error: 3.23mm (+25%)
EF underestimated by 11%
With proposed CVF Without CVF
7. Endocardium delineation using
CVF and proposed myocardial contour processing
7
LV localization &
ROI refinement
Blood pool
classification
by CVF
Myocardial
contour
processing
3D+T
volume
Result
(BASE)
(APEX)
slice 1
slice 2
slice M
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
t=1 t=2 t=N
Polar transformation
Inverse polar transformation
Proposed method
8. Endocardium delineation using
CVF and proposed myocardial contour processing
8
LV localization &
ROI refinement
Blood pool
classification
by CVF
Myocardial
contour
processing
3D+T
volume
Result
(BASE)
(APEX)
slice 1
slice 2
slice M
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
t=1 t=2 t=N
Polar transformation
Proposed method
Inverse polar transformation
9. Blood pool classification by CVF
9
Signal intensity
Occurrence
LV
blood poolOthers
T
LV blood pool
Others
Binary label image:
Local variance map:
Cost slice for selecting ‘LV blood pool’
Cost slice for selecting ‘Others’
Proposed
cost volume
Refined
blood pool segment:
Cost aggregation &
Label selection
Polar transformation
Proposed
cost initialization
Cost-volume filtering
(CVF)
10. Cost-volume-filtering (CVF) based
image segmentation
• CVF is originally used for refining stereo matching results. Recently been
generalized for discrete labeling problems (Rhemann et al.; appears in PAMI 2013,
CVPR 2011).
• Method: Initializing cost(x, y, label) cost aggregation label selection.
• Cost initialization scheme depends on applications.
10
*Rhemann et al.; appears in PAMI 2013, CVPR 2011
(a) Stereo matching (depth from stereo) (b) Interactive image segmentation
Applications of CVF
11. Why CVF?
• Histogram-based labeling (Otsu’s method, GMM-based thresholding) ignores
spatial relationship.
11
LV blood pool
Others
LV
blood pool
Others
T
2D image 1D histogram 2D image
Spatial
information
lost
• CVF considers both spatial relationship and intensity similarity when outputting
labeled results.
-> Good for handling bias field caused by MR field inhomogeneity.
• We proposed a new cost initialization scheme for CVF to address the partial
volume effect in MR images.
• CVF is fast. Can be O(N) time and non-approximate.
12. Effectiveness of CVF-based segmentation
12
• The proposed cost initialization scheme addresses the partial volume issue.
• Compare the result between Fig B and Fig C.
13. 13
• Robust against MR field inhomogeneity
Robustness of CVF-based segmentation
.
PA
.
PB
PA: Brighter PB: Dimmer
Polar
transformation
Inverse polar
transformation
14. Principle of CVF:
using proposed CMR segmentation as an example
14
Cost(x, y, label)
Image
data
Cost
aggregation
Guide
image Labels(x,y)
Cost initialization
Image data
Original image
Cost volume
Cost(x, y, ’blood pool’)
Cost(x, y, ’others’)
High cost
Low cost
Binary label
image
Local
variance map
Label selection
(reduce dimension)
Filtered_cost(x, y, label)
LV blood pool
Others
15. Principle of CVF:
using proposed CMR segmentation as an example
15
Cost(x, y, label)
Image
data
Cost
aggregation
Guide
image
Cost initialization
Labels(x,y)
Label selection
(reduce dimension)
Filtered_cost(x, y, label)
Kernel
(Box filter)
× =
Cost slice Ci Guide image I Weighting W
16. Principle of CVF:
using proposed CMR segmentation as an example
16
Cost(x, y, label)
Image
data
Cost
aggregation
Guide
image
Cost initialization
Labels(x,y)
Label selection
(reduce dimension)
Filtered_cost(x, y, label)
Kernel
(Box filter)
× =
Cost slice Ci
The principle: cost is aggregated from similar* neighbors.
*similar in guide image I
Guide image I Weighting W
(shift-variant)
17. Principle of CVF:
using proposed CMR segmentation as an example
17
Cost(x, y, label)
Image
data
Cost
aggregation
Guide
image
Cost initialization
Labels(x,y)
Label selection
(reduce dimension)
Filtered_cost(x, y, label)
Guide image I Kernel
(Gaussian)
× =
Weighting W
The principle: cost is aggregated from similar* neighbors.
*similar in guide image I
Cost slice C1 for
‘blood pool’
18. Principle of CVF:
using proposed CMR segmentation as an example
18
Cost(x, y, label)
Image
data
Cost
aggregation
Guide
image
Cost initialization
Filtered_cost(x, y, label)
Labels(x,y)
Label selection
(reduce dimension)
Filtered cost slice C1’
Filtered cost slice C2’
Labels fSelect the label with the least cost
CVF-refined
Binary label image
Shift-
variant
filter W(I)
20. Endocardium delineation using
CVF and proposed myocardial contour processing
20
LV localization &
ROI refinement
Blood pool
classification by
CVF
Myocardial
contour
processing
3D+T
volume
Result
(BASE)
(APEX)
slice 1
slice 2
slice M
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
t=1 t=2 t=N
Polar transformation
Inverse polar transformation
Proposed method
21. Endocardial Contour Processing Framework
21
■ Corrected contour (assigned to set B)
■ Good contour (assigned to set G)
Contour based on Canny’s edge detector
Contour based on CVF
papillary muscletrabaculae
Inverse polar
transform
C(p)
I(p)
E(p)
Combine two raw contours I and C and return a regularized final contour
• I: based on CVF-labeled binary image; intensity similarity
• C: based on Canny’s edge detector; gradient information
22. Goodcontour
Correctcontour
Contour function generation and correction
22
1D contour function f
First-order derivative f’
Fluctuations
2D labeled image
Fluctuations are detected and
corrected by linear interpolation
Corrected contour
Fluctuations are a result of the
presence of papillary muscle and
trabeculae
23. Detecting the fluctuations
23
Box filter w2
Box filter w3
Impulse wδ
∗
∗
∗
𝑓′
𝑓′
𝑓′
Type 1
Type 2
Type 3
Contour function
1st order
derivative
ResponseFilter bank
24. Endocardial Contour Processing Framework
24
■ Corrected contour (assigned to set B)
■ Good contour (assigned to set G)
Contour based on Canny’s edge detector
Contour based on CVF
papillary muscletrabaculae
Inverse polar
transform
C(p)
I(p)
E(p)
Combine two raw contours I and C and return a regularized final contour
• I: based on CVF-labeled binary image; intensity similarity
• C: based on Canny’s edge detector; gradient information
25. Complementary contour generation
25
Original image
Non-maximum suppressed
edge response
Edge pruned
Complementary contour function C
Contour form CVF acts as a guide to pick up
Canny’s edge response
26. Endocardial Contour Processing Framework
26
■ Corrected contour (assigned to set B)
■ Good contour (assigned to set G)
Contour based on Canny’s edge detector
Contour based on CVF
papillary muscletrabaculae
Inverse polar
transform
C(p)
I(p) E(p)
min 𝒪 𝐸 =
𝑝 ∈ 𝐵
𝐸 𝑝 − 𝐶 𝑝 2
+
𝑝 ∈𝐺
𝐸 𝑝 − 𝐼 𝑝 2
+ 𝜆
𝑝
𝐸′′
𝑥 2
Combine C(p) and I(p) by minimizing the following objective function:
data term smoothness term
Least squares problem re-formulate to Ax=b and solve for x
27. Endocardial Contour Processing Framework
• The additional constraint will ensure the final contour to enclose the
blood pool
27
𝒪 𝐸 =
𝑝 ∈ 𝐵
𝐸 𝑝 − 𝐶 𝑝 2
+
𝑝 ∈𝐺
𝐸 𝑝 − 𝐼 𝑝 2
+ 𝜆
𝑝
𝐸′′
𝑥 2
subject to 𝐸 𝑥 > 𝐼(𝑥)
Auto Auto /w constraint By expert
28. 28
Inverse polar
transform
Review of the system processing flow
Lock down ROI
Polar mapping
Otsu’s
thresholding
After CVF
Myocardial contour processing
29. Test dataset:
Sunnybrook cardiac MR database
• The first (in 2009) publically accessible cardiac MR database
• Provides 45 MR datasets including one healthy plus three pathological cases
• Includes an evaluation tool & ground truth at end-diastole (ED) and end-
systole (ES)
29
Technical details:
• Acquisition protocol: SSFP MR SAX images are obtained during 10-15 second breath-holds with a
temporal resolution of 20 cardiac phases over the heart cycle, and scanned from the ED phase. Six to 12
SAX images were obtained from the atrioventricular ring to the apex
• MRI scanner: 1.5T GE Signa MRI.
(thickness=8mm, gap=8mm, FOV=320mm*320mm, matrix= 256*256)
33. CMR volume segmentation at ED and ES phase
Patient: SC-HF-I-01 (Heart failure with infarct)
33
-Red: Auto. - - - Purple: expert
34. CMR volume segmentation at ED and ES phase
Patient: SC-HYP-09 (Hypertrophy)
34
-Red: Auto. - - - Purple: expert
35. CMR volume segmentation at ED and ES phase
Patient: SC-HF-NI-04 (Non-ischemic heart failure)
35
-Red: Auto. - - - Purple: expert
36. Evaluation metrics
Average perpendicular distance (APD)
• The computed contour for a given slice is qualified if
APD < 5mm
Overlapping dice metric (DM)
DM =
2𝐴 𝑎𝑚
𝐴 𝑚 + 𝐴 𝑎
Success rate (SR)
• Number of qualified contours (APD < 5mm) in one
CMR scan.
36
SR=75%
37. Performance evaluation
with respect to pathological groups
SR (%) APD (mm) DM
Group Mean StD Mean StD Mean StD
SC-HF-I 94.2 7.7 1.542 0.296 0.93 0.02
SC-HF-NI 95.1 4.1 1.736 0.459 0.92 0.02
SC-HYP 93.6 6.5 1.900 0.420 0.88 0.03
SC-N 93.4 5.3 1.834 0.377 0.89 0.02
Overall 94.1 6.1 1.748 0.417 0.91 0.03
37
A total of 800 MR images from 45 patients are evaluated
38. Result comparison
Method
SR (%)
Mean(StD)
APD (mm)
Mean(StD)
DM
Mean(StD)
Huang 2011 (auto) 81.5(18.0) 2.19(0.44) 0.91(0.03)
Hu 2013 (auto) 91.1(9.4) 2.24(0.40) 0.89(0.03)
Constantinides
2012 (semi-auto)
91.0(8.0) 1.94(0.42) 0.89(0.04)
Constantinides
2012 (auto)
80.0(16.0) 2.44(0.56) 0.86(0.05)
Ngo 2013
(semi-auto)
97.9(6.18) 2.08(0.40) 0.90(0.03)
Ours (auto) 94.1(6.1) 1.75(0.42) 0.91(0.03)
38
Ours against others. All use the same 45-patient dataset.
39. Additional result comparison
Method
SR (%)
Mean(StD)
APD (mm)
Mean(StD)
DM
Mean(StD)
Jolly 20091 95.62(8.83) 2.26(0.59) 0.88(0.04)
Lu 20091 72.45(18.86) 2.07(0.61) 0.89(0.03)
Huang 20091 -- 2.10(044) 0.89(0.04)
Wijnhout 20091 86.47(11) 2.29(0.57) 0.89(0.03)
Constantinides
20091 92.28(--) 2.04(0.47) 0.89(0.04)
Marák 20091 -- 3.00(0.59) 0.86(0.04)
Feng 2013 92.8(9.2) 1.93(0.37) 0.86(0.04)
Ngo 2013 96.58(3.66) 2.22(0.46) 0.89(0.03)
Ours 96.31(4.85) 1.67(0.40) 0.91(0.03)
39
Ours against others. All use the same 15-patient ‘validation’ dataset.
1 Results reported in MICCAI LVSC 2009.
40. Evaluating ejection fraction
40
EF error
Group Mean StD
SC-HF-I -0.15 2.89
SC-HF-NI 2.27 4.64
SC-HYP 2.13 5.01
SC-N 2.39 5.70
Overall 1.61 4.72
y = 0.9547x + 0.3663
R² = 0.9421
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60 70 80 90
AUTOEF
MANUAL EF
AUTO VS. MANUAL EF
SC-HF-NI
SC-HF-I
SC-HYP
SC-N
Method R2 for EF
Cocosco 2008 0.90
Lu 2013 0.92
Lorenzo-Valdés 2004 0.92
Cordero-Grande 2011 0.92
Constantinides 2012 0.83
Ours 0.94
-10
-5
0
5
10
15
20
0 20 40 60 80 100
AUTO-MANUAL
(AUTO + MANUAL) /2
BLAND-ALTMAN PLOT FOR EF
Mean: 1.61
SD: 4.72
Measures the proportion of blood ejected with each cardiac cycle
• EF = (ED volume – ES volume)/ED volume
41. Conclusion
• Developed an algorithm that detects the left ventricular endocardial contour in
CMR images, with top-tier accuracy.
• Use cost-volume filtering (CVF) to combat MR inhomogeneity.
• Proposed a novel cost initialization scheme that handles partial volume effect.
• Proposed a contour processing framework, in which information from gradient and
intensity similarity are encoded along with a smoothness constraint
• Clinical aspect: highly correlated (R2 = 0.94) between auto and manual EF. No
systematic bias is observed.
• Future work includes incorporating inter-slice and inter-frame relationships to
increase detection rate.
41
LV localization
& ROI
refinement
Blood pool
classification
by CVF
Myocardial
contour
processing
3D+T
volume
Result
(BASE)
(APEX)
slice 1
slice 2
slice M
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
t=1 t=2 t=N
Polar transformation
Inverse polar transformation
Proposed method
42. Supplement A:
Automated localization of the left ventricle (LV)
43
• Find the area that covers LV blood pool and LV muscle
• Lock down the region of interest (ROI) and hands off the ROI to the rest
of the algorithm
• This will exclude the influence of nearby tissues when making initial
estimate of LV blood pool
Iteratively refining the region of interest (ROI)
44. (Fig A) Initialize a ROI inside the LV, then:
• (Fig B) Classify the pixels in the ROI (using Otsu’s method)
• (Fig C) Retain the primary connected component
• (Fig E) Compute LV blood pool’s convex hull
• (Fig D) Dilate the convex hull. This has become the new ROI
Repeat B->C->E->D until convergence
45
Step-by-step breakdown
46. 47
• Low contrast area can be recovered regardless of detected LV centroid
PB
.
PC
.
PC
.
PB
.
PB
.
PC
.
PC
PC
.
PB
.
PB
PB
PC
Polar transformation
vs.
vs.
Ours
Robustness of CVF-based segmentation
48. Extreme case of hypertrophy (SC-HYP-08)
49
-Red: automated. - - - Purple: expert-drawn
49. What brings down the success rate?
• Mostly in apical slices & extremely hypertrophic hearts.
• Severe partial volume effect in apical slices violates our assumption.
• In hypertrophic cases, endocardium border can be completely obscured
by papillary muscle and trabeculae carneae.
50
(A) (B)
Editor's Notes
“C” “MR”
照心臟MRI除了可以觀察病理特徵之外,還可以獲得其他額外資訊
A. 因為左心室內部不是平滑的 心肉柱會附著在心肌內壁 還有一個獨立的肌肉組織叫做乳狀肌非常靠近心肌內壁
B. 無法用複雜模型完美描述所有病例和個體的差異。愈少假設愈好
Deep learning and level set 先不管技術本身
CVF 用來解決partial volume所造成的像素分類錯誤
*Briefly describe the workflow*
The input is the CMR 4D volume, and we will process each image individually.
Each image will go through three main stages of the algorithm, and the endocardial contour that goes along the inner muscle wall of the left ventricle will be produced.
(First thing we need to do is to locate the left ventricle
Then we classify the picture elements (pixels) and identify which are inside the LV and which are not
In the third stage, we will process the contour of the left ventricle )
*Briefly describe the workflow*
The input is the CMR 4D volume, and we will process each image individually.
Each image will go through three main stages of the algorithm, and the endocardial contour that goes along the inner muscle wall of the left ventricle will be produced.
(First thing we need to do is to locate the left ventricle
Then we classify the picture elements (pixels) and identify which are inside the LV and which are not
In the third stage, we will process the contour of the left ventricle )
Right after we locate the left ventricle, we do polar coordinates mapping (why?) to transform the original image to what we call here a polar image. Next, we detect the blood pool using the decision boundary we just determined. However, this simple decision process does not give us the best segmentation result yet in many cases. So we will further refine it using a technique called cost-volume filtering process.
And the cost-volume filtering
*If asked what CVF is: it is a
Discrete labeling problems: spatially
Field inhomogeneity=gradual intensity roll-off. Blood in LV should have the same level of gray,
讓受到partial volume影響的地方 選擇兩個label的cost都相等
舉例:
Each candidate label
舉出CVF特性: not space-invariant
Each candidate label
Each candidate label
*Briefly describe the workflow*
The input is the CMR 4D volume, and we will process each image individually.
Each image will go through three main stages of the algorithm, and the endocardial contour that goes along the inner muscle wall of the left ventricle will be produced.
(First thing we need to do is to locate the left ventricle
Then we classify the picture elements (pixels) and identify which are inside the LV and which are not
In the third stage, we will process the contour of the left ventricle )
Detecting the blood pool in the left ventricle is only part of the segmentation. Because inside the muscle wall, there are some irregularities that can't be detected so far. These irregularities, by clinical practice, need to be included in the blood pool as well.
So we propose a contour processing framework to deal with this. Here is the
Detecting the blood pool in the left ventricle is only part of the segmentation. Because inside the muscle wall, there are some irregularities that can't be detected so far. These irregularities, by clinical practice, need to be included in the blood pool as well.
So we propose a contour processing framework to deal with this. Here is the
Detecting the blood pool in the left ventricle is only part of the segmentation. Because inside the muscle wall, there are some irregularities that can't be detected so far. These irregularities need to be included in the LV blood pool as well.
So we propose a contour processing framework,
Section closing remarks:
That was our proposed method for segmenting the LV blood pool. Next we will evaluate the performance of our method.
We use the Sunnybrook cardiac MR database to evaluate our algorithm. A lot researchers also evaluate their algorithms with this database, so we are able to directly compare against each other.
Visual examinations of the result
Using the algorithm we developed, we are able to closely match the expert’s delineation of endocardial contour.
Here are a few more examples
APD measures the error
That means it can be used in practical situations. It saves human time, it involves minimal human control, and the result is highly accurate.
Our method starts with locating the target, the left ventricle, in the CMR images.
The localization process starts with a small region at the center of the image, and eventually the region will grow to fully cover the left ventricle
This step is very crucial to the rest of the algorithm. The first reason is that we want to apply the algorithm on the right target. And the second reason is like this:
If we do a statistical analysis of the signal intensity, we find that each tissue type has its own intensity distribution. For example, The left ventricle blood pool is generally brighter than the left ventricle muscle wall; so you will see two distinctive hills here on the histogram. And its clear to us that we can minimize the decision error by putting a decision boundary here to classify these two tissues.
However, the outside tissues may interfere with the decision process. Now it won't be easy to find a good decision boundary that divide the blood pool and the muscle wall because now the distribution function of so many tissue types are mixed up together.
And that is the second reason why locating the left ventricle is important; by doing so we remove external tissues' influence on the decision process so we are able to get a better classification result that correctly labels the blood pool