Methodology of the suggested method for tissue segmentation in MR brain images using 2D histogram matching. Each algorithmic step is given in detail and analyzed.

1. Tissue Segmentation Methods using
2D Histogram Matching in a Sequence
of MR Brain Images (Part 2)
Vladimir Kanchev, PhD
Radiocommunications and
Videotechnologies Department
TU Sofia, Sofia, Bulgaria
July 2017

2. Page  2
This Research is Reported in:
Kanchev, Vladimir and
Roumen Kountchev.
"Tissue Segmentation Methods Using
2D Histogram Matching in a Sequence
of MR Brain Images."
New Approaches in Intelligent Image
Analysis. Springer International
Publishing, 2016. 183-222.
(Chapter 6)

3. Page  3
Summary – Part 1
Points to remember:
 MRI data – what are their characteristics,
artefacts, etc.
 Transductive learning framework – how we
compute and apply our segmentation model
 2D histogram – how we construct it
 2D histogram matching – how we perform it

4. Page  4
Contents
1. Main idea and contributions
2. Introduction
3. Method description
4. Experimental results
5. Conclusions and future work

5. Page  5
Method Description

6. Page  6
Method Description
For each algorithm we:
 highlight motivation, solution, input and output data
 give a brief math description, staying at a high level
 give a text description of the main properties
We aim to increase reproducibility and
understandability. See Chapter 6 in the book above
and full version of the presentation for more details.

7. Page  7
Method Description
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a 2D histogram
3.6 Segment using back projection

8. Page  8
Preprocess an MR Image Sequence
The problem: MR images originally are set into
different imaging planes and have
various types of tissues. We want
to set all of them under equal
conditions before applying our
segmentation method.
The challenge: How can we do it fast and
accurately?

9. Page  9
Preprocess an MR Image Sequence
The solution: We apply preprocessing
operations to all MR images from
all MR image subsequences.
Operations:
1. Remove redundant tissues using their ground-
truth masks (Brainweb).
2. Transform into the coronal plane (IBSR18 and
Brainweb) and rotate, if it is necessary. Then
resample (Brainweb).
3. Perform gamma correction (optional).

10. Page  10
Preprocess an MR Image Sequence
Input data:
 all MR images from the MR image sequences
Output data:
 preprocessed MR images – all the above MR
images from the sequences

11. Page  11
Properties
In order to produce proper results, the
preprocessing should:
 be applied to each MR image sequence of the
given data sets with the same parameters alike
 use ground truth masks (Brainweb) to remove
skulp and other unnecessary tissues accurately
 use additional resampling of Brainweb datasets
due to the inconsistency of the size of simulated
MRI data and their labeled masks

12. Page  12
Contents
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a 2D histogram
3.6 Segment using back projection

13. Page  13
Divide into MR Image Subsequences
The problem: Brain tissues (CSF, GM and WM
tissues) change gradually
their properties (area, intensity
distribution, etc.) in the separate
MR images along the MR image
sequence.
The challenge: How can we adapt our
segmentation method to it?

14. Page  14
Divide into MR Image Subsequences
The solution: We divide the MR image
sequence into a few MR image
subsequences using a similarity
distance between the
2D histograms of separate MR
images.

15. Page  15
Motivation
We also divide into subsequences since:
 consecutive MR images have greater correlation
 artefacts have local character – they appear
frequently in consecutive MR images
 2D histogram matching between similar 2D
histograms provides better results
 we can speed up the segmentation method by
parallelization

16. Page  16
Divide into MR Image Subsequences
Input data:
 MR image sequence
Output data:
 a few MR image subsequences (from the
sequence above)

17. Page  17
Divide into MR Image Subsequences
We evaluate the similarity between consecutive
MR images from the MRI sequence as follows :
 we use the corresponding normalized, non-
preprocessed 2D histograms
 we use a modification of the wave hedges
distance (Hedges, 1976) to evaluate the similarity
between the computed 2D histograms of the
consecutive MR images

18. Page  18
Divide into MR Image Subsequences

19. Page  19
Divide into MR Image Subsequences
Wave hedges distance between 2D
histograms (Hedges, 1976):
,
, – 2D histograms of two MR images
– the range of intensity levels of a 2D histogram
– indices of the current bin from the 2D histogram
We apply the wave hedges distance within a MR
image sequence, where is a 2D histogram of the
first (reference) MR image and – a 2D histogram of
the current MR image.
 
 






1
1
1
0 ,max
,
B
i
B
j ijij
ijij
c
FE
FE
FED
E F
 .,.cD
B
E
F
ji,

20. Page  20
Divide into MR Image Subsequences
When the similarity distance goes out of the interval
then , the current MR image becomes the reference
MR image and a new MR image subsequence starts.
 1.1,9.0
rc DD 

21. Page  21
Properties
After we apply the MR image division (IBSR20):
 we obtain longer MR image subsequences in the
middle and shorter at the end
 tissues in the middle have larger areas, well-
shaped and compact 2D histograms and perform
better matching
 tissues at the end have smaller areas and do not
have enough 2D histogram bins for the matching

22. Page  22
Contents
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a 2D histogram
3.6 Segment using back projection

23. Page  23
Compute a 2D Histogram
The problem: We want a single type of 2D
histogram to describe existing
tissues and edges in a MR image
and the MR image itself.
The challenge: How can we construct
the 2D histogram in such a
way to build in a segmentation
model?

24. Page  24
Compute a 2D Histogram
The solution: A 2D histogram is produced after
a summation of eight gray-level
co-occurrence matrices (GLCMs)
(see subsection 2.4 in the first
presentation)
We use 2D histograms to describe:
 tissues – model 2D histograms
 edges – edges 2D histograms (an edges matrix)
 whole test MR image – test 2D histogram

25. Page  25
Compute a 2D Histogram
The summation of eight GLCMs of a given MR
image can be shown with the following formula:
𝑝 𝑘 𝑖, 𝑗 =
𝐶𝑖𝑗
𝑘
𝑁 𝑥∙𝑁 𝑦∙𝑘
,
𝐶𝑖𝑗
𝑘
– number of intensity transitions of and intensities
– number of directions for pixel pairs computation (8)
, – and resolution of the MR image
– provides the normalization of a 2D histogram
i j
xN yN
k
kNN yx ..
x y

26. Page  26
Properties
Properties of a 2D histogram (before
preprocessing):
 pixel pairs of separate tissues CSF, GM and WM
are situated on the main diagonal
 pixel pairs of inter-tissue edges CSF-GM, CSF-
WM and GM-WM stay far from the diagonal
 pixel pairs of edges between separate tissues and
background (Bckgr) stay on the first column and
row
 most of the Bckgr pixel pairs stay on bin in the
2D histogram
)1,1(

27. Page  27
2D Histogram
(a) (b)
A (non-normalized) 2D histogram before (a) and
after (b) the preprocessing

28. Page  28
Compute a 2D Histogram
preprocessed
2D histogram
benchmark
2D histogram

29. Page  29
Compute a 2D Histogram
We compute model 2D histograms as:
 we sum GLCMs of neighboring pixel pairs of CSF,
GM and WM tissues from the model MR images
 remove edges pixel pairs tissue-bckgr
We compute edges 2D histograms as:
 we sum GLCMs of neighboring pixel pairs of edges
classes CSF-GM, CSF-WM and GM-WM of the
model MR images

30. Page  30
Compute a 2D Histogram
We compute test 2D histogram as:
 we sum GLCMs of neighboring pixel pairs of the
test MR image
 remove edges pixel pairs tissues-bckgr from the
test 2D histogram
 remove computed edges pixel pairs of edges
classes CSF-GM, CSF-WM and GM-WM from the
test 2D histogram

31. Page  31
Model and Edges 2D Histograms

32. Page  32
Compute a 2D Histogram
Input data (for each MR image subsequence):
 first and last (model) MR image
 segmented ground-truth masks for CSF, GM and
WM tissues (for the first and the last MR image)
 other (test) MR images in the subsequence (w/o
ground truth masks)

33. Page  33
Compute a 2D Histogram
Output data (for each MR image subsequence):
 (preprocessed) model 2D histograms of CSF, GM
and WM tissues (of the first and the last MR
image)
 edges matrix of CSF-GM, CSF-WM and GM-WM
edges classes (of the first and the last MR image)
 (preprocessed) test 2D histograms

34. Page  34
Model and Edges 2D Histograms
CSF
model 2D
histogram
GM
model 2D
histogram
WM
model 2D
histogram
edges
matrix

35. Page  35
Test 2D Histogram
test 2D histogram

36. Page  36
Properties
Properties of the output 2D histograms:
 a non-preprocessed and normalized 2D
histogram gives the stastistics of appearance of
pixel pairs in a MR image
 the number of bins in model (non-preprocessed)
2D histograms is proportional to the size of each
tissue
 the shape and distribution of 2D histograms
depend on the type of MRI data

37. Page  37
Contents
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a 2D histogram
3.6 Segment using back projection

38. Page  38
Match a 2D Histogram
The problem: We have overlapping bin
distribution of separate tissues in
a test 2D histogram. How can we
label train and test bins from the
test 2D histogram using model 2D
histograms?
The challenge: Can we use a matching operation
to label the train set of bins?

39. Page  39
Match a 2D Histogram
The solution: We perform 2D histogram
matching using a vector
(histogram) specification
between a separate model and
a given test 2D histogram.

40. Page  40
Match a 2D Histogram
Basic operations of the 2D histogram matching:
1. Compute and preprocess model 2D histograms
of CSF, GM and WM tissues from the MRI
subsequence. Extract their model vectors.
2. Compute and preprocess a test 2D histogram,
extract segments and test vectors for each
tissue.
3. Specify the corresponding model and test
vectors.
4. Compute a train matrix for each tissue/segment
of the test 2D histogram.

41. Page  41
Motivation
We use a vector specification, since we have:
 a well-known theory of histogram specification
 less memory consumption and shorter execution
time
 available zig-zag ordering algorithms for
conversion into a vector

42. Page  42
Match a 2D Histogram
We perform a vector specification after a
truncation within a percentile interval:
 that should be the same value for model and test
vectors
 a shorter percentile interval would reduce the
influence of outliers but might leave unclassified
areas in the test 2D histogram
 a longer percentile interval produces overlapping
test segments and train matrices

43. Page  43
Match a 2D Histogram
We cut segments of the test 2D histogram to get
corresponding model and test 2D histograms with:
 non-zero bins of similar positions
 similar number of non-zero bins

44. Page  44
Match a 2D Histogram
Input data (for a given test MR image):
 a model 2D histogram of each tissue – CSF, GM
and WM
 a test 2D histogram
Output data (for a given test MR image):
 train matrices for all three tissues
 test segments for all three tissue
 parts at the start and end of the diagonal of the test
2D histogram

45. Page  45
Match a 2D Histogram

46. Page  46
Vector Specification

47. Page  47
Compute a Train Matrix
CSF
WM
GM

48. Page  48
Properties
The properties of the output train matrices:
 most of the non-zero bins are concentrated in the
central areas of each tissue segment, only few in
the periphery
 the number of non-zero bins of each tissue is
proportional to its area in the MR image
 the number and position of non-zero bins depend
also on the used percentile interval for truncation

49. Page  49
Contents
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a test 2D histogram
3.6 Segment using back projection

50. Page  50
Classify a Test 2D Histogram
The problem: We have already labeled train
bins in the test 2D histogram
(partial classification). What
about other bins?
The challenge: How can we classify robustly the
other bins (full classification) –
what type of features, classifier?

51. Page  51
Classify a Test 2D Histogram
The solution: We classify the other unclassified
bins in the test 2D histogram
using:
 a kNN classifier
 distance metric learning
 x and y coordinates of the non-
zero bins in the train matrix and
the test 2D histogram

52. Page  52
Classify a Test 2D Histogram
Input data (for a given test MR image):
 train matrices for CSF, GM and WM tissues
 test segments for each tissue
 parts at the start and end of the diagonal of the test
2D histogram
Output data (for a given test MR image):
 classified 2D histogram of the test MR image

53. Page  53
Classify a Тest 2D Histogram
CSF test
segment
GM test
segment
test 2D
histogram
classified
2D histogram

54. Page  54
Classify a Test 2D Histogram
WM test
segment
CSF train
matrix
WM train
matrix
GM train
matrix

55. Page  55
Properties
Properties of the test 2D histogram
classification algorithm:
 if we truncate with а smaller percentile interval,
some unclassified areas will be left
 if we truncate with a larger percentile interval, this
might lead to unstable classification
 distance metric learning improves slightly accuracy
but increases significantly execution time

56. Page  56
Contents
3.1 Preprocess an MR image sequence
3.2 Divide into MR image subsequences
3.3 Compute test and model 2D histograms
3.4 Match a 2D histogram
3.5 Classify a 2D histogram
3.6 Segment using back projection

57. Page  57
Segment using Back Projection
The problem: We have a completely classified
test 2D histogram. Can we
segment the test MR image?
The challenge: Classification of pixel pairs
along the borders between
the neighboring tissues in test MR
image and edges pixel pairs –
bins.

58. Page  58
Segment using Back Projection
The solution: We apply a back projection
algorithm from the classified 2D
histogram as we classify each
pixel in the test MR image
through classification of eight
pixel pairs within a window.3x3

59. Page  59
Segment using Back Projection
Properties of the back projection algorithm:
 classify the central pixel using all pixel pairs within
a window in accordance with labels of the
corresponding classified bins of tissue and edges
classes
 compute probability maps of each tissue based on
the results of the classification of all pixels
 select the class of the central pixel with a majority
vote between the probability maps
3x3

60. Page  60
Segment using Back Projection
Input data (for a test MR image):
 a classified test 2D histogram
 a classified edges matrix
Output data (for a test MR image):
 a segmented test MR image

61. Page  61
Segment using Back Projection

62. Page  62
Segment using Back Projection
input MR
image
test 2D
histogram
class. 2D
histogram
edges
matrix

63. Page  63
Segment using Back Projection
CSF prob.
map
WM prob.
map
GM prob.
map
CSF-GM
prob. map

64. Page  64
Segment using Back Projection
CSF-WM
prob. map
WM-GM
prob. map
final segm.
MR image

65. Page  65
Properties
Properties of the segmentation results:
 the addition of classified edges bins improves the
correct classification of pixels across borders
between tissues
 a stable classification of the test 2D histogram is
important for the overall segmentation results
 the correspondence between pixel pairs and 2D
histogram bins is vital during back projection

66. Page  66
Summary
Points to remember:
 what is new – 2D histogram, 2D histogram
matching, back projection algorithms
 separate algorithms – their sequence, separate
parameters values, input and output data, etc.
 motivation for each algorithm – problem,
challenge and solution
 analysis of each algorithm – properties,
advantages and disadvantages

67. Page  67
Next – Part 3
1. Main idea and contributions
2. Introduction
3. Method description
4. Experimental results
5. Conclusions and future work

68. Page  68
Next – Part 3
What will the result be, after we apply the
developed segmentation methods to current test
research data sets of MR brain image sequences?