Gaussian Multi-Scale Feature Disassociation Screening in Tuberculosise
Final report
1. Cell Tracking Program for
Cancer Research
Supervisor: Dr. Phil Birch
CandNo: 130493
MSc Engineering Business Management
University of Sussex
2. Statement of originality
At the beginning of the report, I need to announce about the statement of originality. This is the
thesis of my graduation design. The content of this report is my own work and this report has not
been submitted for any degree or other purposes.
I certify that the intellectual content of this thesis is the product of my own work and that all the
assistance received have been acknowledged. All the used reference knowledge and approaches
have been listed in references.
Tian Hao
2
3. Abstract
Quantitive measurement of dividing cancer cells is important in realizing the characteristics of
cancer cells and understanding the effects of cancer treatments. Rent advances in time-lapse
fluorescence microscopy imaging provide an effective method to study the cell cycle process. But
because of the side effect of fluorescer, the cell viability will be influenced during the process.
Compare to traditional methods of cell observation, image processing provides a more interference-
free environment to the observed cells. The growth and behaviors of observed cells will be more
close to the processes in the natural environment. [1] (Xiaowei Chen, 2006) This paper presents a
Matlab based simulation of microscopy image processing for cancer cell researching. The principle
of each image processing approach will be explained in detail in the methodologies section. As the
result of this project, a program will be presented, which contains the functions of cell detection,
tracking and quantity statistics. The way of using the approaches into the actual program will be
introduced in the result section. The performance of this program will be discussed in the discussion
section. And this section will also state the potential improvements of future development.
Conclusion section will summarize both the success and defect of the project.
3
4. Acknowledgements
First and foremost, I would like to show my deepest gratitude to my project supervisor, Dr. Phil
Birch, a respectable, responsible and resourceful scholar, who provided me with valuable guidance
in most important stages of program development. With his advices of the methods which should be
used in this project, and the correction of some mistakes I made in programming, the project went
very well. His friendliness and patience helped me fast quickly progress in Matlab studying.
Then, I shall extend my thanks to my friends Yunong Shu and Nazrine Biskut. They are both the
students of the University of Sussex. As one of the students in Dr. Phil’s image processing class,
Yunong Shu gave me significant information about image processing methods. With his help, I soon
learned what methods could be used to solve my problems. And the other one, Nazrine Biskut
helped me a lot in report writing. He gave me a lot of important suggestions about the structures of
this report and wording.
Last but not least, I’d like to thank my girlfriend and my parents, for their encouragement and
support.
4
5. Contents
1 Aim and Objectives 6
2 Introduction 7
3 Literature Reviews 9
3.1 What is cancer 9
3.2 What is image processing 10
3.3 Detecting a cell using image segmentation 11
3.4 Cell segmentation 15
3.5 Motion-based Multiple Object Tracking 19
4 Methodologies 22
5 Result 27
5.1 Flowchart 27
5.2 Program performance 30
6 Discussion 38
7 Conclusion 40
5
6. 1 Aim and Objectives
Aim:
To develop a system that tracks objects of interest within a video or image sequence by using
Matlab R2015a software.
Objectives:
• Learn how to use Matlab.
• Realize object detection and quantity statistics in single image.
• Track cancer cell in video or image sequence through image processing methods.
• Set up a warning system based on quantity monitoring.
Schedules:
• 1st - 15th June 2015: Project background study.
• 16th - 30th June 2015: Matlab basically practice.
• 1st - 31st July 2015: Software programming.
• 1st - 31st Aug 2015: Program test, evaluation and report writing.
Risk statements:
• Because of the lack of the programming basics, it may require more time on Matlab language
study that may cause the schedule delay.
• The development environment is based on Mac OS, so there might be some problems of
incompatibility in development and test.
• Bad habits of data saving may lead to loss of progress.
• Syntax differences between different versions of Matlab may affect the efficiency of research.
• The lack of the sample resources may lower the compatibility of the final program.
• Shortage in report writing may cause the time constrains of program development.
6
7. 2 Introduction
According to the early history of cancer, cancer is credited to the Greek physician Hippocrates (460-
370BC). Hippocrates, who is called the “Father of Medicine”, used the terms cartoons and
carcinoma to describe non-ulcer forming and ulcer forming tumors. Now, this word is defined as the
diseases which start because of out-of-control growth of abnormal cells. Compare to normal cells,
the cancer cells keep growing and forming new, abnormal cells. They can also invade other tissues.
The changes of DNA make a cell becomes in to a cancer cell. People can inherit abnormal or faulty
DNA from their parents, but most DNA damage is caused by the mistakes in cell reproducing or by
something in the environment, such as cigarette smoking or sun exposure. It is difficult to find out
the reason why any one person get his cancer. [2] (American Cancer Society)
Prevention and treatment of cancer is always one of the most important medical research topics. Till
now, 200 different types of cancer have been found. About half people in the UK will get cancer in
their lifetime. [3] (CANCER RESEARCH UK) There are three common types of cancer treatment,
surgery, chemotherapy and radiation. But none of them can ensure that the patients will recover.
Even if the patient survived, his body would still get damaged because of the side effects of cancer
treatments. [4] (American Cancer Society) For solving this unsolved problem as soon as possible,
the main aim of this project is decided to develop a efficient tool which can help researchers know
and study the features and characteristics of cancer cells.
Up to now, there are two typical ways of cell tracking. They are cell tracking with fluorescent
proteins and cell morphology and image segmentation. [5] (Life Technology) Compare to cell
tracking with fluorescent proteins, Cell morphology and image segmentation doesn’t require wash
steps and it can keep the original cell activity which can be decreased by fluorescent proteins in
wash steps. Matlab, the selected development platform of this project, is a high-level language and
interactive environment used be millions of engineers and scientists worldwide. It provides various
toolboxes with different functions, including signal and image processing, communications, control
systems, and computational finance. In this project, the research will be mainly about the processing
of the microscopic image sequences. The image processing toolbox of Matlab will be the essential
tools of program development. As a software development project, a program for cell tracking will
be finally provided as the result. Matlab and image processing related resources from the internet
are the basic requirements of research. The programming method is learned from the official
examples and tutorials on MathWorks.com. According to the features of cancer cells, the most
7
8. obvious behavior of cancer cells is the never-ending dividing. So in the expectation, the program
should contain the functions of cells detection, tracking and quantity statistic.
The following section of literature reviews will introduce the related background knowledge of
cancer and image processing. The ideas, the methodologies of programming and the corresponding
principles of the code will be explained step by step in in section 4. The flowchart and the
performance of the final program will be showed in section 5. The evaluation of the program
including both of the advantages and defects will be mentioned in discussion (section 6). Section 7
is the conclusion, it will summarize the progress of this project and present ideas about the research
of cancers.
8
9. 3 Literature Reviews
3.1 What is cancer
Cancer is the name of a collection of related diseases. It starts when body’s cells uncontrollable
divide and spread into surrounding tissues. That can happens almost anywhere in the human body.
When cancer develops, old or damaged cells survive although they should die, new cells divide
without stopping although the extra cells are not needed.
Some cancers form solid tumors, which are masses of tissue. And some cancers of the blood
generally do not form solid tumors. [6] (NATIONAL CANCER INSTITUTE)
All Cancers start with some changes in one cell or a small group of cells of more than a hundred
million million cells in a human body. The normal cells have some important features. They can:
• Reproduce themselves only when and where they are needed.
• Stick together in the right place in the body.
• Self destruct if they are damaged or too old.
• Become specialized (mature).
The cancer cells have different features, such as they:
• Don’t stop growing and dividing.
• Ignore signals from other cells.
• Don’t stick together.
• Don’t specialize.
• Don’t repair themselves or die.
• Look different.
[7] (CANCER RESEARCH UK)
9
10. 3.2 What is image processing
There are two typical types of images, vector graphics and bitmaps. Vector graphics are not made
by pixels. Instead, they are comprised of paths, which are defined by a start and end point, along
with other points, curves and angles along the way. A path can be a line, a square, a triangle, or a
curvy shape. Vector-based images can be scaled to a larger size that will not reduce the quality of
the image. The edges of the objects in the image will be still smooth and clean. This type of image
is always used as the brand logos. [8] (Tech Terms) Bitmaps graphics are made up of pixels of
different colors. So it is possible to edit each individual pixel. Since the computer need to save the
information of each pixel in the image, the file size of bitmap graphics is always quite large. And if
the bitmap graphic is resized, the image quality will be reduced. [9] (BBC)
Bitmaps image is made by pixels. It can be seen as an array, or a matrix of square pixels arranged in
columns and rows. The greyscale image is what people usually call a black and white image. An 8-
bit greyscale image includes many shades of grey with a value range from 0 (black) to 255 (white).
A true-color image is assembled from tree greyscale images, red, green and blue. So it has 24 bit
color depth = 8*8*8 bits. With the different combinations of the three greyscale values, an image
may contain up to 16 million different colors. 256*256*256 colors = ~ 16 million colors. Some
greyscale images have more greyscale, such as 16 bit.
Some of the most common file formats of images are:
• GIF — an 8-bit (256 color), non-destructively compressed bitmap format. Mostly used for web.
Has several sub-standards one of which is the animated GIF .
• JPEG — a very efficient (i.e. much information per byte) destructively compressed 24 bit (16
million colors) bitmap format. Widely used, especially for web and Internet (bandwidth-limited).
• TIFF — the standard 24 bit publication bitmap format. Compresses non- destructively with, for
instance, Lempel-Ziv-Welch (LZW) compression.
• PS — Postscript, a standard vector format. Has numerous sub-standards and can be difficult to
transport across platforms and operating systems.
• PSD – a dedicated Photoshop format that keeps all the information in an image including all the
layers.
[10] (HUBBLE SPACE TELESCOPE)
10
11. 3.3 Detecting a cell using image segmentation
It is an example of Matlab R2015a which is available on MathWorks.com. This example explains
how to process a single microscopic image for cell detection by image segmentation method. Figure
3.3.1 is the original image which this example uses.
Figure 3.3.1 Original image - I [11]
In the first step of this example, the image is converted into a binary image with the edge function.
I = imread(‘cell.tif');
[~, threshold] = edge(I, 'sobel');
fudgeFactor = .5;
BWs = edge(I,'sobel', threshold * fudgeFactor);
In the above code, the function edge is used to find edges in intensity image. With the selected
method sobel which is most suitable for microscopic image processing, it finds edges at the points
where the gradient of I is maximum. [12] In the example, a microscopic image is processed as
figure 3.3.2 shows.
Figure 3.3.2 Edge detected image - BWs [11]
11
12. Then it dilates the edges in the images to make them look like an integration more.
se90 = strel('line', 3, 90);
se0 = strel('line', 3, 0);
BWsdil = imdilate(BWs, [se90 se0]);
Strel is a function to create morphological structuring element. ‘SE = strel(‘line’,LEN,DEG)’
creates a flat linear structuring element that is symmetric with respect to the neighborhood center
with the specified angle DEG and the distance between the center and the end of the line which is
specified by LEN. [13] The example uses imdilate function to laterally and longitudinally dilate
each pixel of the edges for 3 units. After this process, the cancer cells are more obvious as figure
3.3.3 shows.
Figure 3.3.3 Edges dilated image - BWsdil [11]
Next step is to fill the holes in the target areas.
BWdfill = imfill(BWsdil, ‘holes');
Above command fills holes in the binary image. In this syntax, a hole is a set of pixels that cannot
be reached by filling in the background from the edge of the image. [14] Figure 3.3.4 shows the
result of holes filling.
Figure 3.3.4 Hole filled image - BWdfill [11]
12
13. In the example, the program has a step to remove connected objects on border.
BWnobord = imclearborder(BWdfill, 4);
Function imclearborder suppresses light structures connected to image border. The function has
been set to remove diagonal connections with 4-connectivity. [15] Not only the cells, but also some
impurities are always on the border of the sample microscopic images. Figure 3.3.5 shows the result
of border clearing.
Figure 3.3.5 Border cleared image - BWnobord [11]
After previous process, there are still some objects around the cell. They can be impurities or some
parts of the cell. The small objects need to be removed, or they will influence the accuracy of image
segmentation.
seD = strel('diamond',1);
BWfinal = imerode(BWnobord,seD);
BWfinal = imerode(BWfinal,seD);
The example program uses function imerode to smoothen the object. Figure 3.3.6 shows the result
of eroding.
Figure 3.3.6 Smoothened image - BWfinal [11]
13
14. The final step of the example program is to draw up the outline of smoothened target on the original
image.
Segout = I;
Segout(BWoutline) = 255;
As figure 3.3.7 presents, above code achieve the function of outlining the target in the image. But
the color of the outline must be a shade of grey with a color value from 0 (black) to 255 (white).
Figure 3.3.7 Outlined original image - Segout [11]
[11] (MathWorks)
14
15. 3.4 Cell segmentation
This is a tutorial of cell segmentation which was posted on MATLAB CENTRAL by Steve Eddins.
The tutorial has a similar process but it uses some different functions. Figure 3.4.1 shows the
original image. Unlike the image in the official example of Matlab, this one has a more complex
background and lower image quality.
The first step is still reading the image but there is another process here to crop the picture which is
not contained in previous program.
I = imread(‘http://blogs.matworks.com/Images/steve/60/nuclei.png');
I_cropped = I(400:900, 465:965);
Figure 3.4.1 Original image - I [16]
Then, there is a process of contrast adjustment which is realized by function adapthisteq. It is not
necessary for most segmentation tasks. But in can improve the processing effect of the low-contrast
images.
I_eq = adapthisteq(I_cropped);
After this process, the contrast of image has been enhanced by transforming the values using
contrast-limited adoptive histogram equalization (CLAHE) as figure 3.4.2 displays. CLAHE
operates on small regions in the image which are named tiles. The contrast of the tiles has been
enhanced, so that the histogram of the output region approximately matches the default ‘uniform’
flat histogram. The neighboring tiles are combined using bilinear interpolation to eliminate
artificially induced boundaries. The contrast, especially in homogeneous ares, cane limited to avoid
amplifying and noise that might be present in the image. [17] (MathWorks)
15
16. Figure 3.4.2 Contrast adjusted image - I_eq [16]
Next is the most common process of image processing.
bw = im2bw(I_eq, graythresh(I_eq);
Similarly with edge function in section 3.3, im2bw is also able to convert image to binary image
based on threshold. [18] The graythresh function uses Otsu’s method, which chooses the threshold
to minimize the intraclass variance of the black and white pixels. [19] (MathWorks) Figure 3.4.3
shows the result of conversion.
Figure 3.4.3 Binarized image - bw [16]
After binary conversion, it is the same process of holes filling. And then, there are three functions
which are not used in the Matlab cell detection example.
bw2 = imfill(bw, ‘holes’);
bw3 = imopen(bw2, ones(5,5));
bw4 = bwareopen(bw3,40);
bw4_perim = bwperim(bw4);
overlay1 = imoverlay(I_eq, bw4_perim, (.3 1 .3]);
Function imopen is used to morphologically open image. The morphological open operation is an
erosion followed by a dilation. [20]
16
17. Function bwareaopen can remove small objects from binary image. The syntax ‘bwareaopen(BW,
P)’ means the connected components which have fewer than P pixels will be removed from the
image. [21] In the tutorial, there are few impurities in the background so 40 is enough for this
process.
As the same, the last step of cell segmentation is to outline the image with bwperim function and
draw up the outline on the original image. But in this tutorial, the author uses another function for
outlining. Function imoverlay is able to change the color of the outline which is made by bwperim
function as figure 3.4.4 shows.
Figure 3.4.4 Outlined image - overlay1 [16]
As the additional development, this tutorial introduces how to detect the cell nucleuses. It uses the
approach which is called market-based watershed segmentation. [16] (Steve Eddins)
mask_em = imextendedmax(I_eq, 30);
Above command returns the extended-maxima transform for I_eq, which is the regional maxima of
the 30-maxima transform. Regional maxima are connected components of pixels with a constant
intensity value, and whose external boundary pixels all have a lower value. [22] (MathWorks)
Figure 3.4.5 shows the result of nucleuses detection.
Figure 3.4.5 Nucleuses detected image - mask_em [16]
17
18. Then is the similar process of image optimizing and outlining.
mask_em = imclose(mask_em, ones(5, 5));
mask_em = imfill(mask_em, ‘holes);
mask_em = bwareaopen(mask_em, 40);
overlay2 = imoverlay(I_eq, bw4_perim | mask_em, [.3 1 .3]);
Function imclose performs morphological closing on the grayscale or binary image. It can be
understood as the inverse processing of imopen. Figure 3.4.6 displays the cells with marked
nucleuses.
Figure 3.4.6 Nucleuses marked image [16]
Then, the tutorial introduces syntaxes to compute the watershed transform.
I_eq_c = imcomplement(I_eq);
I_mod = imimposemin(I_eq_c, ~bw4 | mask_em);
L = watershed(I_mod);
Figure 3.4.7 shows the watershed transformed image which is not necessary in this project.
Figure 3.4.7 Watershed transformed image - L [16]
[16] (Steve Eddins)
18
19. 3.5 Motion-based Multiple Object Tracking
This is another Matlab example which shows how to perform automatic detection and motion-based
tracking of moving objects in a video from a stationary camera. It is also the suggested approach by
project supervisor Dr. Phil Birch. Figure 3.5.1 shows the performance of this program.
Figure 3.5.1 Motion-based multiple object tracking program [23]
In the tutorial, the problem of motion-based tracking has been divided into two parts:
• Detecting moving objects in each frame.
• Associating the detections corresponding to the same object over time.
To figure out the first problem, the tutorial creates an array of tracks first in a sub-function.
function tracks = initializeTracks()
tracks = struct(...
'id', {}, ...
'bbox', {}, ...
'kalmanFilter', {}, ...
'age', {}, ...
'totalVisibleCount', {}, ...
'consecutiveInvisibleCount', {});
end
The structure contains the following fields:
• id : the integer ID of the track
• bbox : the current bounding box of the object; used for display
• kalmanFilter : a Kalman filter object used for motion-based tracking
19
20. • age : the number of frames since the track was first detected
• totalVisibleCount : the total number of frames in which the track was detected (visible)
• consecutiveInvisibleCount : the number of consecutive frames for which the track was not
detected (invisible).
Following is the code of another sub-function which realizes the function of object detection:
function [centroids, bboxes, mask] = detectObjects(frame)
mask = obj.detector.step(frame);
mask = imopen(mask, strel('rectangle', [3,3]));
mask = imclose(mask, strel('rectangle', [15, 15]));
mask = imfill(mask, 'holes');
[~, centroids, bboxes] = obj.blobAnalyser.step(mask);
end
As the above code shows, the detection process includes imopen, imclose and imfill. After image
processing, to associate the detections corresponding to the same object over time, it uses following
functions.
function predictNewLocationsOfTracks()
for i = 1:length(tracks)
bbox = tracks(i).bbox;
predictedCentroid = predict(tracks(i).kalmanFilter);
predictedCentroid = int32(predictedCentroid) - bbox(3:4) / 2;
tracks(i).bbox = [predictedCentroid, bbox(3:4)];
end
end
function [assignments, unassignedTracks, unassignedDetections] = ...
detectionToTrackAssignment()
nTracks = length(tracks);
nDetections = size(centroids, 1);
cost = zeros(nTracks, nDetections);
for i = 1:nTracks
cost(i, :) = distance(tracks(i).kalmanFilter, centroids);
end
costOfNonAssignment = 20;
[assignments, unassignedTracks, unassignedDetections] = ...
20
21. assignDetectionsToTracks(cost, costOfNonAssignment);
end
function updateAssignedTracks()
numAssignedTracks = size(assignments, 1);
for i = 1:numAssignedTracks
trackIdx = assignments(i, 1);
detectionIdx = assignments(i, 2);
centroid = centroids(detectionIdx, :);
bbox = bboxes(detectionIdx, :);
correct(tracks(trackIdx).kalmanFilter, centroid);
tracks(trackIdx).bbox = bbox;
tracks(trackIdx).age = tracks(trackIdx).age + 1;
tracks(trackIdx).totalVisibleCount = ...
tracks(trackIdx).totalVisibleCount + 1;
tracks(trackIdx).consecutiveInvisibleCount = 0;
end
end
[23] (MathWorks)
According to the code of above functions, the process of associating includes location prediction,
tracks assignment and information updating.
21
22. 4 Methodologies
In the expectation, the actual program should be like a perfect combination of all the contained
knowledge from section 3, which is able to detect the cancer cells, continually track them in video
or image sequence, correctly mark number on each cells.
Section 3.1 introduced a method to detect cells by binary conversion and other processes. It had a
good performance in the example. But in the actual test, edge function cannot provide a good result
of conversion if the image quality is not high enough or there are too many obvious impurities in
the image. Figure 4.1 is result of edge processing on the sample image. The function cannot identify
the correct edges in the image because of the low quality and the low contrast.
Figure 4.1 Edge processes sample image
Function imclearborder only appeared in section 3.1, but it is very suitable to process the sample
image in this project. The impurities on the top edge in the sample image influence the performance
of image segmentation. As figure 4.2 shows, the mass of unknown objects on the top will be
identified as cancer cell without border clearing. So this function has been added into the actual
program as a necessary step of cell detection.
Figure 4.2 The image after processing without border clearing process
22
23. The process of eroding is also not necessary because there are few small objects around the cancer
cells in sample image. For keeping the most original outline of the cells, this process will be
removed from the final program.
Section 3.2 introduced some other approaches to process binary image. The cropping process of
sample image is not necessary but this step may be useful for more situations. So this function has
been kept but used non-functional sentence ‘I_cropped = imageArray’ in stead of the original
syntax. The sample images is very similar with the original image in section 3.2. They are both very
dark and they have a same black background. Function adapthisteq can effectively improve the
performance of image processing on this type of images.
As the most common processes of cell detections, function im2bw, imfill, imopen and bwareaopen
have also been contained in the final program. The detailed explanation of the parameter ones(5,5)
in syntax ‘bw3 = imopen(bw2, ones(5,5))’ has still not been found after research. So in the actual
test, this parameter has been changed into strel(‘disk’,5). The result didn't show any big difference
after this change.
In section 3.2, the parameter of bwareopen used 40. That is already enough because there are just
few very small impurities in the background. But in the real case of this project, the size of the
image is different and there are much more individual objects in the background. Because of the
special shape of these cancer cells, some parts of cells are disconnected with the main body after
processed. They have to be removed, or they will be identified as cells by image segmentation.
Figure 4.3 Result if use parameter 40 in bwareaopen
23
24. There are two methods of target outlining in section 3. The first tutorial used syntax
‘Segout(BWoutline) = 255’ and the other one used imoverlay function which is able to change the
color of the outline. Black or white outline is not obvious enough in a grayscale image so imoverlay
will be used in the actual program for the convenience of cancer cell observation.
Next step is to count the cancer cells and point out the correct target in the images. Function
bwlabel is used to label connected components in 2-D binary image. Syntax ‘[L, num] = bwlabel()’
returns the number of connected objects found in the image. [24] (MathWorks) After that, build a
loop with the times based on the number returned by bwlabel function. Then, combine the existing
result with the method posted by iLoveMATLAB BBSUser tdcq127, which uses syntax ‘[row, col]
= find()’ to returns the row and column subscripts of each nonzero element in the image [25].
(MathWorks) To ensure that the counted objects are all cancer cells, an area-based screening
function should be added in this loop. That can be realized by logical statement
‘if(bwarea(BW)>n)’, where parameter n means the area of the processing object. Then add a
variable into this judgement sentence. If the target achieve the standard area of a cancer cell in
microscopic image, it will be marked and the variable will increase one to calculate the overall
number of the cancer cells. Marking process is realized according to the subscripts by plot function.
[26] (tdcq127) Till this step, it has achieved the objective of single image processing and quantity
statistics.
For processing the video input, the method in section 3.5 may be a very effective way to fulfill this
task. But because of the lacks of video codec on Mac and some missing functions used in section
3.5, which are not included in the image processing toolbox, it failed in the practice. So the next
stage of program development will be focus on image sequences processing.
Although the function of video input processing can not be realized, a simple version of this process
has still been added into the program. According to the official tutorial ‘Convert Between Image
Sequences and Video’, following code is able to read an AVI video and convert it into an image
sequence.
shuttleVideo = VideoReader(‘shuttle.avi');
ii = 1;
while hasFrame(shuttleVideo)
img = readFrame(shuttleVideo);
filename = [sprintf('%03d',ii) '.jpg'];
fullname = fullfile(workingDir,'images',filename);
24
25. imwrite(img,fullname) % Write out to a JPEG file (img1.jpg, img2.jpg, etc.)
ii = ii+1;
end
[27] (MathWorks)
The path of output storing can be changed by changing the parameter workingDir in the syntax
‘fullname = fullfile(workingDir,’images’,filename);’.
For continuously processing the microscopic images, a for loop has been added into the program.
Firstly, the path and input files need to be defined by following code.
myFolder = '/Users/Hao/Documents/MATLAB/Samples';
if ~isdir(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:n%s', myFolder);
uiwait(warndlg(errorMessage));
return;
end
filePattern = fullfile(myFolder, '*.jpg');
jpegFiles = dir(filePattern);
In the above code, myFolder defines the location where the input images are being saved.
FilePattern defines the full path of the files including the format. After that, use a for loop to process
the files which have been searched out from this path as following code shows.
for i = 1:length(jpegFiles)
baseFileName = jpegFiles(i).name;
fullFileName = fullfile(myFolder, baseFileName);
fprintf(1, 'Now reading %sn', fullFileName);
imageArray = imread(fullFileName);
end
The image processing processes should be written under imread function. By now, the objective of
video or image sequence processing has been completed. For convenience, a judge sentence of
input format has been added at the beginning of the program. It was realized by a multiple choices
dialog box with case logical syntax.
choice = menu('Sample format','Video','Images')
switch choice
case 1 %False because of lack of memory in Mac OS
Video = VideoReader('/Users/Hao/Documents/MATLAB/Samples','Sample.avi');
25
26. ii = 1;
while hasFrame(Video)
img = readFrame(Video);
filename = [sprintf('%03d',ii) '.jpg'];
fullname = fullfile('/Users/Hao/Documents/MATLAB/Samples',filename);
imwrite(img,fullname) % Write out to a JPEG file (img1.jpg, img2.jpg, etc.)
ii = ii+1;
end
msgbox('Images have been saved','Message');
case 2
end
The above code creates a multiple choice dialog box with Video and Images options by menu
function. If the user choose video, the program will run case 1 to convert the sample video into
images and save the image sequence into a specified folder. The program will process these images
later. If the input is already an image sequence, user can just click Images button, the case will end
directly and it will skip to the step of image processing.
At the end of the program, dialog boxes display the quantity of cancer cells, the quantity increase
and a warning message if the cancer cells increased too fast. It has been realized by dialog function.
Quantity dialog box shows the quantity of cells in the last image which has been named as ‘last’.
Increase dialog box shows the difference value between ‘last’ and ‘first’, which presents the
quantity of cells in the first image in the sequence. As the last step of the program, if last>1.5*first,
a message box will appear and display the message ‘Warning! Quantity increased too fast!’
26
30. 5.2 Program performance
At the beginning of the program, there is a function of multiple-choice dialog box which is used to
select the format of input files as figure 5.2.1 shows. The input files should be a video in .avi format
or a microscopic image sequence of cancer cells which have been moved into the specific location.
There is a small flaw that the title of this dialog box has Chinese character because the Matlab has
been used is a Chinese version. It should be named as ‘Menu’ instead of ‘菜单’.
Figure 5.2.1 Multiple-choice dialog box
If user choose the Video option, the program will convert the sample video into .jpg image
sequence. A message box which displays “Image have been saved” will appear after the conversion
completed. Because there is no video codec suitable on Mac, this function cannot be tested. Figure
5.2.2 displays the error information when user selects Video:
Figure 5.2.2 Error information of video conversion
If the sample is a image sequence, the case loop of Image will end directly. Then the program will
step into image processing section without a video conversion process. Before process the images,
the program checks if the path of the input is correct or not. If the folder is not existed, it will
display ‘The following folder does not exist’.
30
31. Figure 5.2.3 shows the the first image in the sample image sequence. All the samples use the images
like this one. It is so dark that it is difficult to easily distinguish the cancer cells from the black
background.
Figure 5.2.3 Original image
After the process of the function adapthisteq, the contrast of the original image increases as figure
5.2.4 shows. The image has become slightly brighter and the color difference between objects and
background has become more obvious.
Figure 5.2.4 Contrast enhanced image
31
32. Function im2bw converts the image to a binary image. As figure 5.2.5 displays, the color value of
each pixel in the image has been polarized into black and white.
Figure 5.2.5 Binarized image
Figure 5.2.6 shows the result of holes filling. Function imfill fills the empty areas inside the cancer
cells. But it also fills some big objects in the image which may be impurities in the background.
And it makes some groups of small impurities become into larger combinations.
Figure 5.2.6 Hole filled image
32
33. As the hole filled image shows, there are many impurities on the edge of the image, especially on
the top edge. That will influence result of quantity statistics a lot. Function imclearboarder removes
the objects which are connect with the border. As figure 5.2.7 shows, the impurities on border have
been removed but a cell on the top left corner has been also cleared.
Figure 5.2.7 Border impurities removed image
Function imopen morphologically optimizes the image. It clears all the small objects which are
probably not the right targets. But as figure 5.2.8 displays, some parts of the cancer cells have not
been identified as the part of the whole. And some small impurities are still survived. The image
needs to be further optimized.
Figure 5.2.8 Morphologically optimized image
33
34. After the process of function bwareopen, all the small objects have been completely removed as
figure 5.2.9 presents. Although more parts of the cancer cells are lost, the retained main bodies still
can be used to calculate the quantity.
Figure 5.2.9 Small objects removed image
Then the function bwperim draws up the outline of cancer cells as figure 5.2.10 shows.
Figure 5.2.10 Outlines of cancer cells
34
35. Figure 5.2.11 shows that function imoverlay combines the outlined image with the original image.
The green lines have been added around the cancer cells in the microscopic image.
Figure 5.2.11 Outlined microscopic image
Function bwlabel labels connected components in the image. After size filtering process with the
function bwarea, the identified cells have been marked with red points as figure 5.2.12 shows. The
quantity statistics process is also finished in this step.
Figure 5.2.12 Cancer cell marked image
35
36. After image processing sections, the program displays the quantity of the cancer cells in the last
image in sequence as figure 5.2.13 shows.
Figure 5.2.13 Dialog box of quantity
To achieve the function of monitoring the growth of cancer cells, counting the quantity of cancer
cells in each image is necessary. But for simplifying the process and reducing the computer load,
this program only counts the quantities of the first and last images in the sequence. Figure 5.2.14
shows the second dialog box which displays the increase of cancer cells. It is the calculation of the
difference value between the quantity of cancer cells in first image and the quantity in last image.
Figure 5.2.14 Dialog box of cancer cell increase
As the last step of this program, if the quantity of cancer cells increases too much, a warning box
which is displayed in figure 5.2.15 will appear. It only happens when the quantity increase 1.5 times
higher than before. Following is just a display which appears because of a condition change.
Figure 5.2.15 Dialog for warning when cancer cells increase too much
36
37. The program spends about 13 seconds processing a sequence of 47 images. The pictures are fluent
while the program is running. All the cancer cells which are not connected to border have been
identified but some of them are recognized as one cell when they are too close to each other. The
green outlines of cancer cells are always slightly smaller than the real outline the cells should have.
37
38. 6 Discussion
The final program achieves most functions of cells tracking. It is able to detect the cancer cells in
the images and continuously track them in sequence and calculate the number of cells in specified
images.
As the performance showed in section 5.2, the methods of image processing this program uses is
able to segment the image. But after the small objects removing process, the outline it created is
always slightly smaller than the actual outline in the original image. Some narrow parts of the cells
with special shapes are usually missing from the outlined result in the test. It will not influence the
result of quantity statistics but it is inconvenient for researchers to observe the cancer cells. This
problem cannot be fixed or improved because of the limitation of morphological analysis.
In most situation, this program can accurately complete the quantity statistics. But in some special
cases, the connected cells are identified as one target by morphological image analysis.
Figure 6.1 Connected cancer cells
As figure 6.1 shows, there are two cancer cells connected with each other in the centre of the image.
If the program calculate the number of quantity from this frame, the result value will be one less
than it should be. This is the most influential problem of single frame processing in this program.
The error value of quantity statistics depends on the actual action of the cancer cells in the image
sequence. So it is hard to calculate an accurate error rate caused by connected cells.
In consideration of other reasons which may also cause targets missing, for instance the border
clearing process, the accuracy of cancer cells quantity statistics approximately equals to 80%. The
problem of targets missing might be figured out by the method which were introduced in section
3.5. But it requires video processing which is unavailable to test on Mac and the computer vision
system toolbox of Matlab it requires is not one of the selected development tools in this project.
38
39. Because of the same reason, the function of numbering each cancer cell in the video or image
sequence has also not been realized. So the objective of tracking cancer cells in a video or an image
sequence can only be described as basically completed.
As section 5.1 introduced, the logical flow of this program is to firstly select the input format,
process the first image after input file conversion, get the quantity of cancer cells in the first image,
process the sequence, get the quantity in the last image, show result. In the section of input
selection, whether the input is a video or an image sequence, it has to be saved into the specified
folder in advance. To improve the user experience, it should be optimized as the user can select the
input file through a common path selecting user interface.
Till the end of the program development, the warning system is still based on the comparison of the
two quantity values. Ideally, the logic should be improved that to calculate the quantity in each
frame in the sequence, the system should continuously monitor the change of quantity. If the
increase happens in the middle of the sample video or the image sequence, the program should also
give the alarm. This improvement requires more complex logic grammar and a large number of
complex samples.
As section 5.2 shows, the detecting function of this program is based on image processing,
morphological analysis and area-based filtering. In theory, any object which is as big as a cell will
be detected as a target. From another perspective, this program has not been able to identify the
varieties of the detected cells. Solving this problem can be considered as a scheme of future
development.
At last, a good user interface is very important to any program. In this project, user interface
building is not one of the objectives need to achieve. The program needs to run based on Matlab
with its source code. The operation method is inconvenient to common user who doesn't know
about programming. The user has to understand the code and know how to debug the parameters if
he want to slightly change the functions. A fully functional user interface can fix this problem and
greatly improve user experience.
39
40. 7 Conclusion
The original aim of this project is to develop a system which is able to track objects of interest
within a video or image sequence by Matlab. According to the supervisor Dr. Phil Birch’s
suggestion, the research focuses on cell tracking through image processing methods. In
consideration of the available resources for this project, cancer cell has been selected as the research
target, Matlab R2015a and the corresponding image processing toolbox have been chosen as the
development tools. After the three months of research, I realized some advantages and
disadvantages of cancer cell research by image processing methods.
First, image processing method brings an interference-free research environment, which brings
possibility to observe more natural behaviors of the cancer cells comparing to the method of
tracking the cells with fluorescent proteins.
Second, using program to observe the cells doesn't require much experience of microscopy
observation. The accuracy of the result will not be influenced by the lack of observing experience.
Third, program simplifies the process. Most of the the operations can be achieved by mouse clicks.
There are also some disadvantage of image processing. For example, the outline made by the
program is not completely same with the actual outline of the cell. The border connected cells
shouldn't be detected in the program, or the result will be inaccurate.
On the whole, I think cells tracking by image processing is a very potential technology in cancer
research area. One day in the future, it may help researchers find the way to prevent or treat cancers.
40
41. References
[1] Xiaowei Chen, Xiaobo Zhou, and Stephen T.C. Wong (2006). UTOMATED SEGMENTATION,
CLASSIFICATION, AND TRACKING OF CANCER CELL NUCLEI IN TIME-LAPSE
MICROSCOPY. Boston: IEEE Trans on Biomedical Engineering. p1.
[2] American Cancer Society. (2014). Early history of cancer. Available: http://www.cancer.org/
cancer/cancerbasics/thehistoryofcancer/the-history-of-cancer-what-is-cancer. Last accessed 4th Aug
2015.
[3] CANCER CELL UK. What is cancer?. Available: http://www.cancerresearchuk.org/about-
cancer/what-is-cancer. Last accessed 1st Jun 2015.
[4] American Cancer Society. (2015). What are the different types of cancer treatment?. Available:
http://www.cancer.org/treatment/understandingyourdiagnosis/talkingaboutcancer/
whensomeoneyouknowhascancer/when-somebody-you-know-has-cancer-cancer-treatment-
questions. Last accessed 2nd May 2015.
[5] Life technologies. Cell Tracking. Available: http://www.lifetechnologies.com/uk/en/home/life-
science/cell-analysis/cell-tracing-tracking-and-morphology/cell-tracking.html. Last accessed 2nd
Jun 2015.
[6] NATIONAL CANCER INSTITUTE. (2015). A Collection of Related Diseases. Available: http://
www.cancer.gov/about-cancer/what-is-cancer. Last accessed 9th Aug 2015.
[7] CANCER CELL UK. Cancer cells. Available: http://www.cancerresearchuk.org/about-cancer/
what-is-cancer/how-cancer-starts/cancer-cells. Last accessed 1st Jun 2015.
[8] Tech Terms. Vector Graphic. Available: http://techterms.com/definition/vectorgraphic. Last
accessed 9th Aug 2015.
[9] BBC. Bitmap and vector graphics. Available: http://www.bbc.co.uk/schools/gcsebitesize/dida/
graphics/bitmapvectorrev1.shtml. Last accessed 9th Aug 2015 .
[10] HUBBLE SPACE TELESCOPE. A short introduction to astronomical image processing.
Available: https://www.spacetelescope.org/projects/fits_liberator/improc/. Last accessed 9th Aug
2015.
[11] MathWorks. Detecting a Cell Using Image Segmentation. Available: http://uk.mathworks.com/
help/images/examples/detecting-a-cell-using-image-segmentation.html?refresh=true. Last accessed
11th Aug 2015.
[12] MathWorks. edge. Available: http://uk.mathworks.com/help/images/ref/edge.html. Last
accessed 15th Aug 2015.
[13] MathWorks. strel. Available: http://uk.mathworks.com/help/images/ref/strel.html. Last
accessed 15th Aug 2015.
42. [14] MathWorks. imfill. Available: http://uk.mathworks.com/help/images/ref/imfill.html. Last
accessed 25th Jun 2015.
[15] MathWorks. imclearborder. Available: http://uk.mathworks.com/help/images/ref/
imclearborder.html. Last accessed 25th Jun 2015.
[16] Steve Eddins. (2006). Cell segmentation. Available: http://blogs.mathworks.com/steve/
2006/06/02/cell-segmentation/. Last accessed 12th Aug 2015.
[17] MathWorks. adapthisteq. Available: http://uk.mathworks.com/help/images/ref/
adapthisteq.html?refresh=true. Last accessed 12th Aug 2015.
[18] MathWorks. im2bw. Available: http://uk.mathworks.com/help/images/ref/im2bw.html. Last
accessed 25th Jun 2015.
[19] MathWorks. graythresh. Available: http://uk.mathworks.com/help/images/ref/graythresh.html.
Last accessed 12th Aug 2015.
[20] MathWorks. imopen. Available: http://uk.mathworks.com/help/images/ref/imopen.html. Last
accessed 12th Aug 2015.
[21] MathWorks. bwareaopen. Available: http://uk.mathworks.com/help/images/ref/
bwareaopen.html. Last accessed 12th Aug 2015.
[22] MathWorks. imextendedmax. Available: http://uk.mathworks.com/help/images/ref/
imextendedmax.html. Last accessed 13th Aug 2015.
[23] MathWorks. Motion-Based Multiple Object Tracking. Available: http://uk.mathworks.com/
help/vision/examples/motion-based-multiple-object-tracking.html?refresh=true. Last accessed 14th
Aug 2015.
[24] MathWorks. bwlabel. Available: http://uk.mathworks.com/help/images/ref/bwlabel.html. Last
accessed 16th Aug 2015.
[25] MathWorks. find. Available: http://uk.mathworks.com/help/images/ref/find.html. Last accessed
16th Aug 2015.
[26] tdcq127. (2013). Count objects in a image. Available: http://www.ilovematlab.cn/
thread-272277-1-1.html. Last accessed 16th Aug 2015.
[27] MathWorks. Convert Between Image Sequences and Video. Available: http://
uk.mathworks.com/help/matlab/examples/convert-between-image-sequences-and-video.html. Last
accessed 17th Aug 2015.
43. Appendix
Source code
clc;
clf;
choice = menu('Sample format','Video','Images')
switch choice
case 1 %False because of lack of memory in Mac OS
Video = VideoReader('/Users/Hao/Documents/MATLAB/Samples','Sample.avi');
ii = 1;
while hasFrame(Video)
img = readFrame(Video);
filename = [sprintf('%03d',ii) '.jpg'];
fullname = fullfile('/Users/Hao/Documents/MATLAB/Samples',filename);
imwrite(img,fullname) % Write out to a JPEG file (img1.jpg, img2.jpg, etc.)
ii = ii+1;
end
msgbox('Images have been saved','Message');
case 2
end
myFolder = '/Users/Hao/Documents/MATLAB/Samples';
if ~isdir(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:n%s', myFolder);
uiwait(warndlg(errorMessage));
return;
end
filePattern = fullfile(myFolder, '*.jpg');
jpegFiles = dir(filePattern);
first_baseFileName = jpegFiles(1).name;
first_fullFileName = fullfile(myFolder, first_baseFileName);
first_image = imread(first_fullFileName);
