Tissue Segmentation Methods Using 2D Histogram Matching in a Sequence of MR Brain Images_part2 (Extd.)

Tissue Segmentation Methods using
2D Histogram Matching in a Sequence
of MR Brain Images (Part 2 – Extd.)
Vladimir Kanchev, PhD
Radiocommunications and
Videotechnologies Department
TU Sofia, Sofia, Bulgaria
July 2017

Page  2
This Research is Reported in:
Kanchev, Vladimir and
Roumen Kountchev.
"Tissue Segmentation Methods Using
2D Histogram Matching in a Sequence
of MR Brain Images."
New Approaches in Intelligent Image
Analysis. Springer International
Publishing, 2016. 183-222.
(Chapter 6)

Page  3
Summary – Part 1
Points to remember:
 MRI data – what are their characteristics,
artefacts, etc.
 Transductive learning framework – how we
compute and apply our segmentation model
 2D histogram – how we construct it
 2D histogram matching – how we perform it

Page  4
Contents
1. Main idea and contributions
2. Introduction
3. Method description
4. Experimental results
5. Conclusions and future work

Page  6
Method Description
For each algorithm we:
 highlight motivation, solution, input and output data
 give a brief math description, staying at a high level
 give a text description of each substep and the
main properties
We aim to increase reproducibility and
understandability. See Chapter 6 in the book above
for more details.

Page  7
Method Description
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a 2D histogram
3.6 Segment using back projection

Page  8
Preprocess an MR Image Sequence
The problem: MR images originally are set into
different imaging planes and have
various types of tissues. We wan
to set all of them under equal
conditions before applying our
segmentation method.
The challenge: How can we do it fast and
accurately?

Page  9
The solution: We apply preprocessing
operations to all MR images from
all MR image subsequences.

Page  10
Operations:
1. Remove redundant tissues using their ground-
truth masks (Brainweb).
2. Transform into the coronal plane (IBSR18 and
Brainweb) and rotate, if it is necessary. Then
resample (Brainweb).
3. Perform gamma correction (optional).

Page  11
Our preprocessing operations do not include:
 noise reduction
 artefact correction
 other intensity normalization operations besides
gamma correction

Page  12
Input data:
 all MR images from the MR image sequences
Output data:
 preprocessed MR images – all the above MR
images from the sequences

Page  13
Original properties of input test MRI data:
 IBSR 18* – real MRI data, preprocessed (no noise
and artefacts), originally set in the axial plane
 IBSR 20* – real MRI data, non-preprocessed
(presence of noise and artefacts), originally set in
the coronal plane
 Brainweb** – artificially generated MRI data, non-
preprocessed (no noise and artefacts), originally
set in the axial plane
**http://brainweb.bic.mni.mcgill.ca/
*https://www.nitrc.org/projects/ibsr

Page  14
Remove Redundant Tissues
Remove skull and other unnecessary tissues
using labeled ground-truth masks and keep only
CSF, GM and WM tissues (Brainweb).
Bauer, S., Fejes, T., & Reyes, M. (2013). A
skull-stripping filter for ITK. Insight Journal,
2012.
* Bauer, S., Fejes, T., & Reyes, M. (2013). A skull-stripping
filter for ITK. Insight Journal, 2012.
*

Page  15
Set in the Coronal Plane
We set all MR images from the sequence and their
labeled masks
 from their original in the coronal plane
 rotate them to be set in upright position (when it is
necessary)

Page  16
Different Imaging Body Planes
*
* https://upload.wikimedia.org/wikipedia/commons/3/34/BodyPlanes.jpg

Page  17
MR Brain Images in Different Planes
*
axial sagittal coronal
* http://www.esa.int/spaceinimages/Images/2013/06/MRI_brain_scan2

Page  18
Set in the Coronal Plane
This operation:
 is fast and easy to implement
 does not produce artefacts
 provides a compact representation of separate
tissues in a MR image

Page  19
Resample MRI Data - Brainweb
A specific problem with MR image sequences from
Brainweb after their conversion to the coronal plane:
the resolution of labeled masks is
of simulated data sets is
We decided not to introduce pixels/voxels with new
intensity values, so we use interpolation with the
nearest neighbor to perform enlargement of the
simulated data sets.
]434,362,362[
]256,256,181[

Page  20
Resample MRI Data - Brainweb
Operations:
1. Interpolate (enlarge) the simulated datasets with
the nearest neighbor two times in three directions.
2. Center the interpolated simulated datasets in
relation to the labeled masks and remove
redundant pixels from the tissues of the
simulated datasets, which have no corresponding
mask pixels.
3. Reduce the labeled masks and the enlarged
simulated data sets to the original size of the
simulated data sets.

Page  21
Gamma Correction
We apply gamma correction to the intensity value
of each pixel from the MR image sequences.
The formula is:
, where
– normalization coefficient
– input image,
– output (preprocessed) image
– gamma coefficient
– intensity levels of the input image
* Gonzalez & Woods: Digital image processing.
3rd edn. (2007).
* Gonzalez & Woods: Digital image processing. 3rd edn. (2007)

io IcI .
c
i
I  1-L0,i I
 1-L0,oI

oI
L

Page  22
Gamma Correction
*
* Gonzalez & Woods: Digital image processing. 3rd edn. (2007).

Page  23
Gamma Correction
During preprocessing gamma correction:
 sets apart Gray Matter (GM) and White Matter
(WM) tissue distributions in a 2D histogram
 improves the compactness of cerebrospinal fluid
(CSF) tissue distribution in the 2D histogram
 is applied optionally

Page  24
Properties
In order to produce proper results, the
preprocessing should:
 be applied to each MR image sequence of the
given data sets with the same parameters alike
 use ground truth masks (Brainweb) to remove
skulp and other unnecessary tissues accurately
 use additional resampling of Brainweb datasets
due to the inconsistency of the size of simulated
MRI data and their labeled masks

Page  25
Contents

Page  26
Divide into MR Image Subsequences
The problem: Brain tissues (CSF, GM and WM
tissues) change gradually
their properties (area, intensity
distribution, etc.) in the separate
MR images along the MR image
sequence.
The challenge: How can we adapt our
segmentation method to it?

Page  27
The solution: We divide the MR image
sequence into a few MR image
subsequences using a similarity
distance between the
2D histograms of separate MR
images.

Page  28
Motivation
We also divide into subsequences since:
 consecutive MR images have greater correlation
 artefacts have local character – they appear
frequently in consecutive MR images
 2D histogram matching between similar 2D
histograms provides better results
 we can speed up the segmentation method by
parallelization

Page  29
Input data:
 MR image sequence
Output data:
 a few MR image subsequences (from the
sequence above)

Page  30
We evaluate the similarity between consecutive
MR images from the MRI sequence as follows:
 we use the corresponding normalized, non-
preprocessed 2D histograms
 we use a modification of the wave hedges
distance (Hedges, 1976) to evaluate the similarity
between the computed 2D histograms of the
consecutive MR images

Page  31

Page  32
Wave hedges distance between 2D
histograms (Hedges, 1976):
,
, – 2D histograms of two MR images
– the range of intensity levels of a 2D histogram
– indices of the current bin from the 2D histogram
We apply the wave hedges distance within a MR
image sequence, where is a 2D histogram of the
first (reference) MR image and – a 2D histogram of
the current MR image.
 
 






1
1
1
0 ,max
,
B
i
B
j ijij
ijij
c
FE
FE
FED
E F
 .,.cD
B
E
F
ji,

Page  33
When the similarity distance goes out of the interval
then , the current MR image becomes the reference
MR image and a new MR image subsequence starts.
 1.1,9.0
rc DD 

Page  34
Properties
After we apply the MR image division (IBSR20):
 we obtain longer MR image subsequences in the
middle and shorter at the end
 tissues in the middle have larger areas, well-
shaped and compact 2D histograms and perform
better matching
 tissues at the end have smaller areas and do not
have enough 2D histogram bins for the matching

Page  35
Contents

Page  36
Compute a 2D Histogram
The problem: We want a single type of 2D
histogram to describe existing
tissues and edges in a MR image
and the MR image itself.
The challenge: How can we construct the 2D
histogram in such a way to build
in a segmentation model?

Page  37
The solution: A 2D histogram is produced after
a summation of eight gray-level
co-occurrence matrices (GLCMs)
(see subsection 2.4 in the first
presentation).

Page  38
We use 2D histograms to describe:
 tissues – model 2D histograms
 edges – edges 2D histograms (an edges matrix)
 whole test MR image – test 2D histogram

Page  39
The summation of eight GLCMs of a given MR
image can be shown with the following formula:
𝑝 𝑘 𝑖, 𝑗 =
𝐶𝑖𝑗
𝑘
𝑁 𝑥∙𝑁 𝑦∙𝐾
,
𝐶𝑖𝑗
𝑘
– number of intensity transitions of and intensities
– number of directions for pixel pairs computation (8)
– a current direction
, – and resolution of the MR image
– provides the normalization of a 2D histogram
i j
xN yN
K
KNN yx
..
x y
k

Page  40
Properties
Properties of a 2D histogram (before
preprocessing):
 pixel pairs of separate tissues CSF, GM and WM
are situated on the main diagonal
 pixel pairs of inter-tissue edges CSF-GM, CSF-
WM and GM-WM stay far from the diagonal
 pixel pairs of edges between separate tissues and
background (Bckgr) stay on the first column and
row
 most of the Bckgr pixel pairs stay on bin in the
2D histogram
)1,1(

Page  41
2D Histogram
(a) (b)
A (non-normalized) 2D histogram before (a) and
after (b) the preprocessing

Page  42
preprocessed
2D histogram
benchmark
2D histogram

Page  43
We compute model 2D histograms as:
 we sum GLCMs of neighboring pixel pairs of CSF,
GM and WM tissues from the model MR images
 remove edges pixel pairs tissue-bckgr
We compute edges 2D histograms as:
 we sum GLCMs of neighboring pixel pairs of edges
classes CSF-GM, CSF-WM and GM-WM of the
model MR images

Page  44
We compute test 2D histogram as:
 we sum GLCMs of neighboring pixel pairs of the
test MR image
 remove edges pixel pairs tissues-bckgr from the
test 2D histogram
 remove computed edges pixel pairs of edges
classes CSF-GM, CSF-WM and GM-WM from the
test 2D histogram

Page  45
Model and Edges 2D Histograms

Page  46
Input data (for each MR image subsequence):
 first and last (model) MR image
 segmented ground-truth masks for CSF, GM and
WM tissues (for the first and the last MR image)
 other (test) MR images in the subsequence (w/o
ground truth masks)

Page  47
Output data (for each MR image subsequence):
 (preprocessed) model 2D histograms of CSF, GM
and WM tissues (of the first and the last MR
image)
 edges matrix of CSF-GM, CSF-WM and GM-WM
edges classes (of the first and the last MR image)
 (preprocessed) test 2D histograms

Page  48
Model and Edges 2D Histograms
CSF
model 2D
histogram
GM
model 2D
histogram
WM
model 2D
histogram
edges
matrix

Page  49
Compute a Test 2D Histogram
test 2D histogram

Page  50
Compute Model 2D Histograms
Compute and preprocess model 2D histograms:
1. Segment 6 MR images of separate tissues
(CSF, GM and WM) from the first and the last MR
image of a MR image subsequence using ground-
truth segmented masks.
2. Compute 6 2D histograms of the separate
tissues from the segmented 6 MR images from
1 above.
3. Set to 0 bins from the first row and column
(preprocessing) of each of the computed 6 2D
histograms from 2 above.

Page  51
Compute Model 2D Histograms
4. Set to 0 bins in the CSF model 2D histogram
with x and y coordinates above a certain threshold
– 100.
It concerns 2D histograms of CSF tissue
(IBSR 20), because they are more scattered.
5. Sum up the 2D histograms of the corresponding
tissues and normalize them through division by
2.
Finally, in the general case, we obtain 3 model
2D histograms of CSF, GM and WM tissues for the
given MRI subsequence.

Page  52
Compute Edges 2D Histograms
Compute and preprocess edges 2D histograms:
1. Segment 12 MR images of the tissue pairs
(CSF-GM, CSF-WM and GM-WM) and tissues
(CSF, GM and WM) for the first and the last MR
image from the MR image subsequence using the
ground-truth segmented masks.
2. Compute 12 2D histograms from the calculated
12 MR images from 1 above.
3. Set to 0 bins from the first column and row of
the computed 12 2D histograms from 2 above.

Page  53
Compute Edges 2D Histogram
4. Subtract from the 2D histograms of tissue
pairs the 2D histograms of their composite
single tissues; perform the subtraction separately
for tissues from the first and the final MR image.
5. Sum up the corresponding edges 2D
histograms (CSF-GM, CSF-WM, GM-WM) from
the first and the final MR image from 4 above.
Finally, in the general case, we obtain an edges
matrix of positions of edges bins from the three
edges 2D histograms for the MR image subsequence.

Page  54
Compute a Test 2D Histogram
Compute and preprocess a test 2D histogram:
1. Load a test MR image from the current MR image
subsequence.
2. Compute a test 2D histogram from the test MR
image.
3. Set to 0 bins from the first column and row
(preprocessing) of the test 2D histogram.
4. Select bins from the test 2D histogram which
coincide with the edges matrix and save them
for the final MR image segmentation. Then, set
the selected bins to 0.

Page  55
Properties
Properties of the output 2D histograms:
 a non-preprocessed and normalized 2D
histogram gives the stastistics of appearance of
pixel pairs in a MR image
 the number of bins in model (non-preprocessed)
2D histograms is proportional to the size of each
tissue
 the shape and distribution of 2D histograms
depend on the type of MRI data

Page  56
Contents

Page  57
Match a 2D Histogram
The problem: We have overlapping bin
distribution of separate tissues in
a test 2D histogram. How can we
label train and test bins from the
test 2D histogram using model 2D
histograms?
The challenge: Can we use a matching operation
to label the train set of bins?

Page  58
The solution: We perform 2D histogram
matching using a vector
(histogram) specification
between a separate model and
a given test 2D histogram.

Page  59
Basic operations of the 2D histogram matching:
1. Compute and preprocess model 2D histograms
of CSF, GM and WM tissues from the MRI
subsequence. Extract their model vectors.
2. Compute and preprocess a test 2D histogram,
extract segments and test vectors for each
tissue.
3. Specify the corresponding model and test
vectors.
4. Compute a train matrix for each tissue/segment
of the test 2D histogram.

Page  60
Motivation
We use a vector specification, since we have:
 a well-known theory of histogram specification
 less memory consumption and shorter execution
time
 available zig-zag ordering algorithms (JPEG
encoding, as in our case) for conversion into a
vector

Page  61
We perform a vector specification after a
truncation within a percentile interval:
 that should be the same value for model and test
vectors
 a shorter percentile interval would reduce the
influence of outliers but might leave unclassified
areas in the test 2D histogram
 a longer percentile interval produces overlapping
test segments and train matrices

Page  62
We cut segments of the test 2D histogram to get
corresponding model and test 2D histograms with:
 non-zero bins of similar positions
 similar number of non-zero bins

Page  63
Input data (for a given test MR image):
 a model 2D histogram of each tissue – CSF, GM
and WM
 a test 2D histogram
Output data (for a given test MR image):
 train matrices for all 3 tissues
 test segments for all 3 tissue
 parts at the start and end of the diagonal of the test
2D histogram

Page  64

Page  65
Compute a Model Vector
Operations for each model 2D histogram:
1. Set to 0 all bins, whose value is equal or less
than 2 (IBSR20).
2. Extract a model vector using zig-zag ordering
of non-zero bins from the model 2D histogram.
3. Calculate a normalized and a cumulative model
vector from the model vector.
4. Truncate elements from the cumulative model
vector outside the percentile interval; truncate
their corresponding elements from the normalized
vector; calculate threshold bins for the test
segment which correspond to the vector elements.

Page  66
Compute a Test Vector
Operations for each test 2D histogram:
1. Set to 0 all bins from the test 2D histogram that
do not correspond to the current tissue, using
the threshold bins. Thus, we calculate a test
segment.
2. Extract a test vector using zig-zag ordering of
non-zero bins from the test segment. Then, calcu-
late a normalized and a cumulative test vector.
3. Truncate elements from the cumulative test
vector outside the percentile interval; truncate
their corresponding elements from the normalized
test vector.

Page  67
Vector Specification
Operations for each pair of a test and a model
vector:
1. Compute a corresponding vector of indices
between elements of the cumulative test vector
and their closest element values (min L1
distance) of the cumulative model vector.
2. Transform elements from the normalized test
vector using elements from the normalized
model vector using the corresponding vector.
As a final result we obtain a resulting normalized
vector (specified vector) for the given tissue.

Page  68
Vector Specification

Page  69
Compute а Train Matrix
Operations for each resulting vector:
1. Calculate a difference vector as we subtract
from the resulting normalized vector (specified
vector) the normalized test vector.
2. Perform k-means clustering (L1 distance) on
element values of the difference vector.
3. Set to 0 elements of all clusters except the one
with the greatest mean value (centroid).
4. Back project the difference vector elements as a
train matrix.

Page  70
Compute a Train Matrix
CSF
WM
GM

Page  71
Properties
The properties of the output train matrices:
 most of the non-zero bins are concentrated in the
central areas of each tissue segment, only few in
the periphery
 the number of non-zero bins of each tissue is
proportional to its area in the MR image
 the number and position of non-zero bins depend
also on the used percentile interval for truncation

Page  72
Contents
3.5 Classify a test 2D histogram

Page  73
Classify a Test 2D Histogram
The problem: We have already labeled train
bins in the test 2D histogram
(partial classification). What
about other bins?
The challenge: How can we classify robustly the
other bins (full classification) –
what type of features, classifier?

Page  74
The solution: We classify the other unclassified
bins in the test 2D histogram
using:
 a kNN classifier
 distance metric learning
 x and y coordinates of the non-
zero bins in the train matrix and
the test 2D histogram

Page  75
Input data (for a given test MR image):
 train matrices for CSF, GM and WM tissues
 test segments for each tissue
 parts at the start and end of the diagonal of the test
2D histogram
Output data (for a given test MR image):
 classified 2D histogram of the test MR image

Page  76
For each test 2D histogram:
1. Train a kNN classifier with the bins from the
calculated train matrices, corresponding to
separate tissues of the test 2D histogram.
2. Classify using the trained kNN classifier the
other bins from the test segments of the test
2D histograms.
Use as features the x and y coordinates of the
non-zero bins of the train matrices and the test
segments of the test 2D histogram.

Page  77
3. After the kNN classification we add parts at the
start and end of the diagonal of the test 2D
histogram to the CSF and WM classes.
The start and end parts were removed from the
test 2D histogram during the computation of the
test segments.
We can apply LMNN and NCA distance metric
learning to the kNN classifier to improve the results
but it slows down the whole algorithm.

Page  78
Classify a Тest 2D Histogram
CSF test
segment
GM test
segment
test 2D
histogram
classified
2D histogram

Page  79
WM test
segment
CSF train
matrix
WM train
matrix
GM train
matrix

Page  80
Properties
Properties of the test 2D histogram
classification algorithm:
 if we truncate with а smaller percentile interval,
some unclassified areas will be left
 if we truncate with a larger percentile interval, this
might lead to unstable classification
 distance metric learning improves slightly accuracy
but increases significantly execution time

Page  81
Contents

Page  82
Segment using Back Projection
The problem: We have a completely classified
test 2D histogram. Can we
segment the test MR image?
The challenge: Classification of pixel pairs
along the borders between
the neighboring tissues in test MR
image and edges pixel pairs –
bins.

Page  83
The solution: We apply a back projection
algorithm from the classified 2D
histogram as we classify each
pixel in the test MR image
through classification of eight
pixel pairs within a window.3x3

Page  84
Properties of the back projection algorithm:
 classify the central pixel using all pixel pairs within
a window in accordance with labels of the
corresponding classified bins of tissue and edges
classes
 compute probability maps of each tissue based on
the results of the classification of all pixels
 select the class of the central pixel with a majority
vote between the probability maps
3x3

Page  85
Properties
Important points:
 classified edges bins are important for the correct
classification of edges pixels along borders
 since we do not consider edges tissues-bckgr, we
prioritize the tissue classes over bckgr class during
the majority vote
 unclassified parts of the test 2D histogram lead to
classification of some tissue pixels as bckgr class

Page  86
Input data (for a test MR image):
 a classified test 2D histogram
 a classified edges matrix
Output data (for a test MR image):
 a segmented test MR image

Page  87
For each classified test 2D histogram:
1. Classify all pixel pairs in a window in
accordance with the classified test 2D histogram
and edges bins from the edges matrix. Apply it for
all pixels in the test MR image.
2. Compute weight maps of the separate tissues
and bckgr, from all pixel pairs within a window.
Add classified edges with weight to their
constituent tissue classes.
5.0
3x3
3x3

Page  88
3. Compute probability maps as weight maps for
all 8 directions*.
4. Segment the corresponding test MR image with a
majority vote between the probability maps.
5. Evaluate the segmentation results for each
tissue after a comparison with the corresponding
ground-truth segmented masks.
* we visualise the probability maps of the edges
classes for better understanding

Page  89

Page  90
input MR
image
test 2D
histogram
class. 2D
histogram
edges
matrix

Page  91
CSF prob.
map
WM prob.
map
GM prob.
map
CSF-GM
prob. map

Page  92
CSF-WM
prob. map
WM-GM
prob. map
final segm.
MR image

Page  93
Properties
Properties of the segmentation results:
 the addition of classified edges bins improves the
correct classification of pixels across borders
between tissues
 a stable classification of the test 2D histogram is
important for the overall segmentation results
 the correspondence between pixel pairs and 2D
histogram bins is vital during back projection

Page  94
Summary
Points to remember:
 what is new – 2D histogram, 2D histogram
matching, back projection algorithms
 separate algorithms – their sequence, separate
parameters values, input and output data, etc.
 motivation for each algorithm – problem,
challenge and solution
 analysis of each algorithm – properties,
advantages and disadvantages

Page  95
Next – Part 3
1. Main idea and contributions
2. Introduction
3. Method description
4. Experimental results
5. Conclusions and future work

Page  96
Next – Part 3
What will the result be, after we apply the
developed segmentation methods to current test
research data sets of MR brain image sequences?

Tissue Segmentation Methods Using 2D Histogram Matching in a Sequence of MR Brain Images_part2 (Extd.)

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Tissue Segmentation Methods Using 2D Histogram Matching in a Sequence of MR Brain Images_part2 (Extd.)

Similar to Tissue Segmentation Methods Using 2D Histogram Matching in a Sequence of MR Brain Images_part2 (Extd.) (20)

Recently uploaded

Recently uploaded (20)

Tissue Segmentation Methods Using 2D Histogram Matching in a Sequence of MR Brain Images_part2 (Extd.)