This document summarizes Dylan Derwael's internship project at the University of Valladolid developing a computer vision application to detect blood and medical bandages for a robotic tool during laparoscopic surgery. The project utilizes parallel programming with Intel Thread Building Blocks and OpenCV for computer vision tasks like color detection using RGB and HSV color spaces and texture detection. Detection methods tested include using color, texture features like standard deviation and GLCM, and edge detection with Canny. The application was built in C++ with a Qt user interface for potential future expansion.

1. Department
Industrial Science & Technology
Parallelized computer vision application, detecting blood
and medical bandages for a robotic tool in laparoscopic
operations.
Internship report of
Dylan Derwael
Candidate for the degree of Professional Bachelor in
the education Bachelor in Electronics-ICT
major ICT
Promoters:
Eusebio De La Fuente López
Gerard Schreurs
Academic year: 2014-2015
Reference: E15_S_ELO-ICT_ICT_02

3. Department
Industrial Science & Technology
Parallelized computer vision application, detecting blood
and medical bandages for a robotic tool in laparoscopic
operations.
Internship script admitted by
Dylan Derwael
Candidate
Bachelor in Electronics-ICT
Major ICT
Internship mentor(s):
Eusebio De La Fuente López
Gerard Schreurs
Academic year r: 2014-2015
Reference: E15_S_ELO-ICT_ICT_02

5. Table of Contents
Introduction..........................................................................................................................................5
About the University of Valladolid.......................................................................................................5
Project description................................................................................................................................6
Computer vision...................................................................................................................................6
Parallel programming...........................................................................................................................7
Parallel vs serial...............................................................................................................................7
Qt-Framework......................................................................................................................................9
Linux.....................................................................................................................................................9
Ubuntu................................................................................................................................................10
Detecting with OpenCV.....................................................................................................................11
RGB color space............................................................................................................................11
HSV color space............................................................................................................................12
RGB vs HSV..................................................................................................................................13
Texture detection............................................................................................................................13
Texture detection with GLCM.......................................................................................................15
Grey Level Co-occurrence Matrix and Haralick texture features.............................................15
Gray-Level Co-occurrence Matrix (GLCM).............................................................................15
Most important texture features proposed.................................................................................16
Contrast (Sum of square variance).......................................................................................16
Entropy.................................................................................................................................16
Variance................................................................................................................................17
Why GLCM was not a valid solution.......................................................................................17
Texture detection with standard deviation.....................................................................................18
Texture detection with canny detector...........................................................................................19
Blood detection..............................................................................................................................21
Medical bandage detection.................................................................................................................24
Bandage detection by color.......................................................................................................24
Bandage detection by texture....................................................................................................24
Combining color and texture.....................................................................................................24
Intel Thread Building Blocks (TBB)..................................................................................................27
Threading Building Blocks: tasks instead of threads.....................................................................27
Thread Building Blocks: Flow Graph............................................................................................27
Thread Building Blocks: Flow graph components........................................................................28
Simple TBB Flow Graph example.................................................................................................30
TBB parallel_for............................................................................................................................32
Application structure..........................................................................................................................33
User Interface.....................................................................................................................................34
Recommendations for the future........................................................................................................36
Accelerometer................................................................................................................................36
More and real data.........................................................................................................................37
Ideas for detection efficiently........................................................................................................38
Conclusion..........................................................................................................................................39
Bibliography.......................................................................................................................................40

6. Introduction
During the senior year of a professional bachelor at the University College - Leuven Limburg.
We are required to complete a internship successfully in a field related to our study area. When I
was offered the choice to complete my internship abroad, I didn't hesitate and applied to become an
Erasmus exchange student for Spain. My internship took place at the University of Valladolid in the
heart of Spain.
When I arrived I met Prof. Francisco Javier Garcia Ruiz who was assigned to help me around the
school. He introduced me to Prof. Eusebio De La Fuente López of the faculty of engineering. He
gave me a project to build a visual detection application for a robotic tool during a Laparoscopic
surgery.
The first priority of the visual detection is to detect internal bleeding and as a secondary to detect
medical bandages. The project fascinated me on many different areas because it gave me the
opportunity to learn a lot of new and very interesting topics from a programming language as C++
to visual detection to parallel programming.
I want to thank Prof. De La Fuente López for giving me the opportunity to do this project and the
support I got during the last four months. His direction and council always made sure that this
project was heading in the right direction.
About the University of Valladolid
The University of Valladolid is a public university in the city of Valladolid,
province of Valladolid, in the autonomous region of Castile-Leon, Spain.
Established in the 13th
century, it is one of the oldest universities in the world.
The university has 32 000 undergraduate students and more than 2 000
teachers. It is responsible for teaching higher education in seven campuses
distributed through four cities of Castile and Leon: Valladolid, Palencia, Soria
and Segovia.
The internship takes place at the faculty of engineering for the department
of engineers of systems and automation. My project includes multiple study
fields within this department. The parallelization aspect connects to Computer Science, the visual
processing aspect connects to the study of control and the whole project is build for a robotic tool
that connects to the study of robotics.
5
Figure 1: Logo
university

7. Project description
The purpose of the application is to create a visual processing element for a robotic tool. This
robotic tool will be utilized to assist a surgeon during an operation. During this project we will use a
laparoscopic surgery as an example for the image processing. During the operation the robot will be
required to be aware when internal bleeding occurs within the abdominal cavity so that it can
provide a medical bandage to the surgeon at the location of the bleeding as soon as possible, to
apply compression to the wound. Later on the application will be expanded to also detect the
medical bandages.
The application will not be limited to these situations but with having an eye to the future. The
project could be expanded to detect the organs within the abdominal cavity and construct a virtual
scenario of the abdominal cavity. This virtual mapping will aid both the robot and the surgeon in
completing the tasks at hand more accurately.
To build this application to be capable of handling all of the visual processing we will engineer it
upon the concept of parallel programming. This will help divide the tasks between the different
hardware cores of the CPU. Letting the application use all of the hardware resources will increase
the efficiency, scalability and load-balance capabilities of the application and thus making it future
proof.
Computer vision
This application will use the OpenCV library to handle all the aspects of the visual processing.
This library is written in C and C++, it is designed for computational efficiency with a strong focus
on real-time applications. OpenCV runs on Linux, Android, Mac OSX, IOS and Windows.
The library has more than 2500 optimized algorithms, which includes a comprehensive set of
both classic and state-of-the-art computer vision and machine learning algorithms. These algorithms
can be used to detect and recognize faces, identify objects, classify human actions in videos, track
camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds
from stereo cameras, stitch images together to produce a high resolution image of an entire scene,
find similar images from an image database, remove red eyes from images taken using flash, follow
eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc.
Written in optimized C/C++, the library can take advantage of multi-core processing. Enabled
with OpenCL, it can take advantage of the hardware acceleration of the underlying heterogeneous
compute platform. Adopted all around the world and a usage ranging from interactive art, to mines
inspection, stitching maps on the web or through advanced robotics.
6

8. Parallel programming
This application will also be build according to a parallel programming structure. Most
programming languages (like C++) work in a sequential manner or serial. This means that each step
will run after the other. This might not be capable of using all the hardware resources efficiently.
The program will be as fast as the total amount of time required to process each step.
A parallel structure is capable of running multiple steps next to each other on different hardware
threads. This way the time required to run the same application will decrease depending on how
much you are capable of running next to each other. In an ideal situation where you can parallelize
all of the steps and work on a high concurrency the application will go as fast as the slowest step.
There are a couple of ways to develop a parallel structure in a sequential programming language by
manually programming threads, OpenMP and Intel Thread Building Blocks.
When the threads are programmed manually, the highest level of efficiency can be reached
because the entire process is under the control of the developer but the downside of doing
everything manually is that it requires a lot of time and a certain degree of expertise to make it work
properly.
OpenMP and Intel Thread Building Blocks are both libraries like OpenCV but then for parallel
programming. Both have lots of predefined functions and structures to make it easier to develop the
software and to maintain. Since both are similar in usage the choice of which one to use came down
between the differences. After some research it became clear that for this project it would be better
to use the Intel Thread Building Blocks library. Because it's written in C++ where OpenMP is
written in C, and OpenCV already makes use of Intel Thread Building Blocks under the hood.
Parallel vs serial
When we look at parallel programming there are two ways to look at it. One is to look for a more
efficient use of time and another is to process more work in the same amount of time. Both ways
will improve the algorithm but with a different point of view. By example we will have a serial
algorithm (Figure 2) in which each task takes up 100 milliseconds.
7Figure 2: Serial process

9. Then we can use parallelism to speed up the process in time. By splitting up some tasks and
letting them run concurrently (Figure 3). The speed gain will decrease when we divide the workload
among more tasks. Theoretical with an infinite amount of cores this algorithm will run as fast as
300 milliseconds. Giving a speed gain of 1.6 times.
The time won will not always seem that impressive in this example. There is a way to keep the
time as a constant and add weight to the the algorithm. See in the example below (Figure 8). This
means that we can add the task to spend the same amount of time to do more work. On the right you
can see that the algorithm would have done the work in 500 milliseconds where the serial would
have taken 1100 milliseconds. Theoretically we can say that this will be able to increase to the
workload equals 2 * nCores * 100 + 300 in 500 milliseconds.
8
Figure 3: Parallel time improvement
Figure 4: Parallel Increase in workload

10. Qt-Framework
Qt is a cross-platform application and UI framework
used for device creation and application development
supporting deployment to over a dozen leading
platforms. Qt uses standard C++ with extensions
including signals and slots that simplifies handling of
events, and this helps in development of both GUI and
server applications which receive their own set of event
information and should process them accordingly.
Qt supports many compilers, including the GCC C++ compiler and the Visual Studio suite. Qt
also provides Qt Quick, that includes a declarative scripting language called QML that allows using
JavaScript to provide the logic. With Qt Quick, rapid application development for mobile devices
became possible, although logic can be written with native code as well to achieve the best possible
performance.
Qt can be used in several other programming languages by language bindings. It runs on the
major desktop platforms and some of the mobile platforms. It has extensive internationalization
support. Non-GUI features include SQL database access, XML parsing, JSON parsing, thread
management and network support.
Linux
The application will be run on a computer with a Linux operating system. We have chosen this
for many reasons over the windows operating system. For this project we have chosen for the
Ubuntu flavor of Linux. Because it's a good mixture between up to date available software and
stability. There are many other options to chose from but this is one of the biggest distributions
ensuring that it will have support for years to come.
One of the most obvious reasons that we chose Linux over windows, is that you are assured that
the operating system will not restart in the middle of an operation for updates or any other reason.
As a proof for this you only have to look at the servers that host websites and a multitude of online
services. They can run for months, even years, without having to restart.
A second reason would be that a Linux OS is less prone to slow down over time. There are
multiple reasons that causes this. To list up a couple no actual viruses/virus scanners to slow down
the system, it uses a lot less resources in general (this also gives more resources to the actual
application), a more efficient file management system that defragments itself and so on.
Overall it's the best operating system especially when you consider that this will provide the
images to a surgeon during an operation. One could say it would be even critical not to use
windows in a situation like that.
9
Figure 5: Logo Qt

11. Ubuntu
It is arguably the most user-friendly version of Linux out there. Ubuntu's whole philosophy is
based around making it easier for the user, hiding any unnecessary complexity behind the scenes.
This makes it perfect for users not accustomed to Linux.
In terms of our project the people operating this application will have a low learning curve to get
around the operating system, while it still allows developers to have the full access of the Linux
power house hidden behind the scenes. It releases a stable update of it's operating system every 6
months and a long term support edition every two years. The long term support will provide
security updates, bug fixes and platform stability updates for 5 years. Canonical the company
behind Ubuntu is also investing in different markets as the mobile phone. Giving the reason that this
operating system will stand for a long time. The stability and prospect of a long continuation of the
operating system makes it an excellent choice as the platform operating system.
10
Figure 6: Ubuntu User Interface

12. Detecting with OpenCV
Most computation visual detections will either be found by the color or the texture item within
an image. For color detection the two most color models used for detection. Are RGB and HSV. We
will quickly overview both of them and explain why we chose the one above the other.
The texture detection is very important to find the bandage so we will overview the different
methods for texture detection that we tried to use to find the bandages within the images.
RGB color space
The RGB color space is defined by three primary colors Red, Green and Blue. With these three
colors you can create any possible color. To make a specific color you have to set each color at a
certain intensity. This color model can be represented as a cube
For example, shining three lights together onto a
white wall: one red light, one green light, and one
blue light, each with dimmer switches. If only the red
light is on, the wall will look red. If only the green
light is on, the wall will look green.
If the red and green lights are on together, the wall
will look yellow. Dim the red light and the wall will
become more of a yellow-green. Dim the green light
instead, and the wall will become more orange.
Bringing up the blue light a bit will cause the orange
to become less saturated and more whitish.
In all, each setting of the three dimmer switches
will produce a different result, either in color or in
brightness or both. The set of all possible results is
the gamut defined by those particular color lamps.
Swap the red lamp for one of a different brand that is
slightly more orange, and there will be a slightly different gamut, since the set of all colors that can
be produced with the three lights will be changed.
11
Figure 7: RGB color cube

13. HSV color space
The HSV(hue-saturation-value) color space is cylindrical, but usually represented as a cone or a
hexagonal pyramid (Figure 9), as displayed in this picture, because this is the subset of the space
with valid RGB values.
The V (value) is the vertical axis pointing up in the picture. The top of the hex-cone corresponds
to V = 1, or the maximum intensity colors (Bright colors). The point at the base of the hex-cone is
black and here V = 0. Figure 8 shows this a "slice" through the HSV hex-cone. As the value of V
changes, the hex-cone will grow and shrink in size.
The value of S(saturation) is a ratio, ranging from 0 on the center line vertical axis (V) to 1 on
the sides of the hex-cone. Any value of S between 0 and 1 may be associated with the point V = 0.
The point S = 0, V = 1 is white. Intermediate values of V for S = 0 are the grays. Note that when S =
0, the value of H is irrelevant.
H (hue) is the angle around the V-axis (center line of the hex-cone). When S=100%, all primary
colors(RGB) will be 120° opposite to one another and as measured by H, in an angle around the
vertical axis (V), with red at 0°. When S=0% H will give back a shade of gray. In he table below
you will see a display of the colors values.
Range Black White Red Yellow Green Cyan Blue magenta
H 0-360° - - 0° 60° 120° 180° 240° 300°
S 0-1 0 0 1 1 1 1 1 1
V 0-1 0 1 1 1 1 1 1 1
Table 1: HSV color values
12
Figure 8: HSV slice V=1
Figure 9: HSV hexagonal pyramid

14. RGB vs HSV
Between these two most common used color spaces we had to weigh in, both there positive and
negative features.
HSV is the least sensitive to illumination changes. But it's not a linear model making it difficult
to set accurate boundaries to the range that we perceive as blood. Also red is at 0° degrees in the
model making it difficult to accurately differentiate between two tints of red. One tint of can be at
1° while another is at 359°, this is a huge difference.
With the RGB model we have to define a range on a smaller cubical object within the RGB cube.
This object will contain those parts that we perceive as blood. But due to illumination changes this
might not always be the most accurate, as it can alter the values of all three components.
We choose to use RGB for being less complex and giving the ability to also use a ratio between
the different components.
Texture detection
To humans, an image isn't just a random collection of pixels. It's a meaningful arrangement of
regions and objects. Despite the large variations possible in images, humans have no problem
interpreting them.
We interpret the images through a difference in color, texture and brightness. When it's harder for
us to find an object through it's color, ex. When the bandage is smudged with blood we will find it
in an image because of it's texture. Therefor we will also use the texture to detect an image with
computer vision.
Color information is very valuable but it does not provide enough indications to accurately
segment the areas we are looking for in the image. More features need to be included in the
detection algorithm. Because we are trying to identify areas that are well characterized by their
texture (the lack of texture in the case of blood and a significant texture in the medical bandages)
the analysis of this feature can be helpful.
Image texture analysis will provide information about the spatial arrangement of color or
intensities in a region of an image. The perceived texture of an image will be quantified computing
a set of metrics. The intuitive qualities described by terms such as rough, smooth, silky, or bumpy
can be measured as a function of the spatial variation in pixel (Figure 10) Texture is a remarkable
feature in medical bandages.
13
Figure 10: Image bandage

15. Intensities and regions can be characterized by their texture content. In this sense, the roughness
or bumpiness refers to variations in the intensity values. Texture analysis has been used in many
applications, including remote sensing, automated inspection, and medical image processing. We
will use the texture analysis to find the texture boundaries, what is called texture segmentation.
In order to segment the image according to the texture, the image can be filtered using standard
statistical measures. These statistics can characterize the texture because they provide information
about the local variability of the gray level values of pixels. For example, calculating the variance
of pixels, gray level in a neighborhood can indicate the degree of variability of pixel values in that
region. In areas with smooth texture, the variance of gray level values in the neighborhood around a
pixel will be a small value; in areas of rough texture, the variance will be large. Similarly, others
statistical measures can be computed such as contrast, entropy, etc.
Table 2: Texture examples
14
Figure 11: Original Figure 12: Contrast
Figure 13: Variance Figure 14: Entropy

16. Texture detection with GLCM
1. Grey Level Co-occurrence Matrix and Haralick texture features
In 1973, Robert Haralick [6] introduced the co-occurrence matrix and it's texture features which
are the most popular second order statistical features today. Haralick proposed two steps for texture
feature extraction: the first is computing the co-occurrence matrix and the second step is calculating
texture feature base on the co-occurrence matrix. This technique is useful in wide range of image
analysis applications from biomedical to remote sensing techniques.
2. Gray-Level Co-occurrence Matrix (GLCM)
One of the defining qualities of texture is the spatial distribution of gray values. The use of
statistical features is therefore one of the early methods proposed in the image processing literature.
Haralick suggested the use of co-occurrence matrix or gray level co-occurrence matrix. It
considers the relationship between two neighboring pixels, the first pixel is known as a reference
and the second is known as a neighbor pixel. In the following, we will use {I (x, y) , 0 ≤ x ≤ N x −
1, 0 ≤ y ≤ N y − 1} to denote an image with G gray levels. The G × G gray level co-occurrence
matrix P d θ for a displacement vector d = (dx, dy) and direction θ is defined as follows. The
element (i, j) of P d θ is the number of occurrences of the pair of gray levels i and j which is the
distance between I and j following direction θ is d.
P d θ (i, j) = # {((r, s) , (t, v)) : I (r, s) = i, I (t, v) = j}
Where (r, s) , (t, v) N x × N y ; (t, v) = (r + dx, s + dy).∈
15
Figure 15: GLCM example

17. Figure 15 shows the co-occurrence matrix P d θ with distance d = 1 and the direction is
horizontal (θ = 0). This relationship (d = 1, θ = 0) is nearest horizontal neighbor. There will be (N x
− 1) neighboring resolution cell pairs for each row and there are N y rows, providing R = (N x − 1)
× N y nearest horizontal pairs. The co-occurrence matrix can be normalized by dividing each of its
entry by R.
In addition, there are also co-occurrence matrices for vertical direction (θ = 90) and both
diagonal directions (θ = 45, 135). If the direction from bottom to top and from left to right is
considered, there will be eight directions (0, 45, 90, 135, 180, 225, 270, 315). From the co-
occurrence matrix, Haralick proposed a number of useful texture features.
3. Most important texture features proposed
● Contrast (Sum of square variance)
Contrast is a measure of intensity or gray-level variations between the reference pixel and its
neighbor. In the visual perception of the real world, contrast is determined by the difference in the
color and brightness of the object and other objects within the same field of view.
When i and j are equal, the cell is on the diagonal and i − j = 0. These values represent pixels
entirely similar to their neighbor, so they are given a weight of 0. If i and j differ by 1, there is a
small contrast, and the weight is 1. If i and j differ by 2, the contrast is increasing and the weight is
4. The weights continue to increase exponentially as (i − j) increases.
● Entropy
The concept comes from thermodynamics, it refers to the quantity of energy that is permanently lost
to heat every time a reaction or a physical transformation occurs. Entropy cannot be recovered to do
useful work. Because of this, the term can be understood as amount of irremediable chaos or
disorder. The equation of entropy is:
16

18. ● Variance
Variance is a measure of the dispersion of the values around the mean of combinations of
reference and neighbor pixels. It is similar to entropy, answers the question ‘What is the dispersion
of the difference between the reference and the neighbor pixels in this window?’
GLCM Variance in texture measures and performs the same task as does the common descriptive
statistic called variance. It relies on the mean, and the dispersion around the mean, of cell values
within the GLCM. However, GLCM variance uses the GLCM, therefore it deals specifically with
the dispersion around the mean of combinations of reference and neighbor pixels, so it is not the
same as the simple variance of grey levels in the original image.
Variance calculated using i or j gives the same result, since the GLCM is symmetrical.
There is no particular advantage to using Standard Deviation over Variance, other than a different
range of values.
Why GLCM was not a valid solution
Once you have calculated the GLCM you can get out a lot of information about textures within
an image. And you can compare this to a GLCM you have stored in a database, to find the element
that you are looking for. In most cases this will work perfectly but in the case of the bandage it's not
ideal to use.
First the bandage will be deformed in any possible way the chaos theory can be applied to this.
Either you will only detect the parts that are well aligned with the camera or you need to have a lot
of different samples in the database to compare with.
Second if the bandage is fully blooded or partially covered with blood or not at all. Once we
convert the image to a gray scale image. The parts that are covered with blood will be displayed as
darker than the parts that are not covered with blood. By default the darker parts will be placed on a
different part of the GLCM than the lighter parts.
Also for this you have to have samples in the database of examples where the bandage is fully or
partially covered with blood. Multiply that with the different deformed bandage examples and there
will be a requirement to have a lot of samples to compare to for which there might not be sufficient
time to process it all. Since this is a real-time application, that will display a video (30 frames per
second), this will set the bar of maximum amount of time to process one frame in 0.03 seconds.
Calculating the GLCM of a high resolution image in 4 or 8 directions will take up some time to
process. This can be optimized by parallelization through the TBB library. But then it has to be
compared with a high amount of possibilities. Taking up both time and resources this would be a
less than ideal option to use.
17

19. Texture detection with standard deviation
The standard deviation can detect texture boundaries (where two textures meet). This follows
logically. If only one texture was in the local neighborhood, the standard deviation only has to
encapsulate the intra-class variation. When a second texture is present, the standard deviation must
represent the inter-class variation in addition to the intra-class variation. This means that, standard
deviation will peak wherever two or more textures meet, as long as there is sufficient inter-class
variation.
Every texture boundary detector also needs a way to suppress the intra-class variations within a
texture from appearing as boundaries. Standard deviation can achieve this as well, because in
general, for different areas within the same texture, standard deviation tends to be approximately
equal. This allows for a properly designed and tuned algorithm to detect different areas of the same
texture.
The standard deviation can be calculated fast. Benefits make standard deviation an excellent
choice for a real-time algorithm. It must be noted that not all forms of texture can be distinguished
solely by standard deviation, but this is acceptable when the primary goal is a real-time algorithm.
The formula (below) produces an image, that is a measure, that is used to quantify the amount of
variation or dispersion of a set of data values. It can be computed by adding the squared differences
of each value and the mean.
18

20. Texture detection with canny detector
The Canny edge detection algorithm is known to many as the optimal edge detector. Canny's
intentions were to enhance the many edge detectors already out at the time he started his work. He
was very successful in achieving his goal and his ideas and methods can be found in his paper, "A
Computational Approach to Edge Detection" [11] . In his paper, he followed a list of criteria to
improve current methods of edge detection. The first and most obvious is low error rate. It is
important that edges occurring in images should not be missed and that there be NO responses to
non-edges. The second criterion is that the edge points be well localized. In other words, the
distance between the edge pixels as found by the detector and the actual edge is to be at a minimum.
A third criterion is to have only one response to a single edge. This was implemented because the
first 2 were not substantial enough to completely eliminate the possibility of multiple responses to
an edge.
Based on these criteria, the canny edge detector first smooths the image to eliminate and noise. It
then finds the image gradient to highlight regions with high spatial derivatives. The algorithm then
tracks along these regions and suppresses any pixel that is not at the maximum (non-maximum
suppression). The gradient array is now further reduced by hysteresis. Hysteresis is used to track
along the remaining pixels that have not been suppressed. Hysteresis uses two thresholds and if the
magnitude is below the first threshold, it is set to zero (made a non-edge). If the magnitude is above
the high threshold, it is made an edge. And if the magnitude is between the 2 thresholds, then it is
set to zero unless there is a path from this pixel to a pixel with a gradient above T2.
19

21. Table 3: Results canny edge detector
Top row : original image32
Middle row : Canny Detector
Bottom row : Standard deviation
When we compare the results of the canny detector. We can observe that with the canny detector
we can point out more and bigger parts of the bandage then with the standard deviation. But it also
detects more noise and allows the noise to form bigger objects after some morphological operations
are applied to the binary image. The results of the canny detector displays only the borders of all the
textures. First we have to use the morphological dilate operation on the image. Making the small
lines bigger and letting them connect. Then we use the close operation to close small gaps in
between and then erode to make the remaining figures more rounded. This will give a number of
big objects in the right image. Whereas for the standard deviation we can start with a close
operation.
The example of the right image with the standard deviation will not let the structure in the
middle from top to bottom and become one even with the other operations that are added later. With
canny this will be the case making this the biggest object in the picture. The only thing it needs is to
reflect the light and we have a false positive.
We do not have any image available to display this false positive. But while testing the bandage
color we have encountered organs that were reflecting enough light to meet the requirements (ex.
Table 6 middle picture).
20

22. Blood detection
To detect the process of bleeding we first need to be capable of detecting blood as you can see in
the pictures presented above most of the pictures is dominated with the color red. This makes it
difficult to easily differentiate between what is blood and what is human tissue.
To detect blood we are using a couple of conditions as a filter to find the bloody parts within a
frame. The first set of conditions is to set a range on the RGB values since blood will always be red.
Everything else like fat, bandages, operation tools, etc. will have to be excluded from the frame. For
this we give a wide range to the red color channel, a very limited range for the blue en green color
channels.
The second part of conditions is to work with a ratio between two different color channels. This
is based on a project with a capsule endoscopy [5]. This is to first divide the green channel by the
red one and do the same for the blue channel and divide it by the red one. After that you combine
the results and only take in account the pixels where both values are below 0.6. For that project it
was a good number but in the abdominal cavity there is a lot more red present then in the intestine
we lowered the values and looked for the best results.
If the following conditions are met we consider it to be blood
0.1 < (B/R) < 0.4
0.1 < (G/R) < 0.4
40 < R < 255
B < 20
G < 20
Table 4: Results blood detection success
21

23. Table 5: Detection fails and adjustment is required
22

24. In table 5 you can see when the the conditions fail. The brightness level is higher then normal .
But if we alter the conditions according we can detect the blood again. To solve this the values need
to alter depending on the brightness. With a couple of preset conditions to go with different levels of
brightness. It's possible that the illumination changes can be calculated and added into the
conditions.
Another possible solution to handle this is to find the area's with a high illumination make a
segment of them and process them differently from the darker parts in the image.
Settings for the brighter images, as you can see the ratio between blue and red, green and red
have been set to a low standard. While the red color minimum value has increased together with the
blue and red maximum values. All of these are directly bound to the brightness increase.
If the following conditions are met we consider it to be blood when the brightness is high
(B / R) < 0.2
(G / R) < 0.2
150 < R < 255
B < 60
G < 60
23

25. Medical bandage detection
1. Bandage detection by color
The first condition to detect a bandage is by filtering the image on the color this works in a
similar fashion as the blood detection. The main difference is that we are looking for white and any
color where the blue and green values are very close to each other . With red we don't take into
account if it's to be close to the green and blue values as the bandage can be smudged with blood.
Making it more red to completely red. Then we set a minimal value for the blue, green and red
values. Anything that's below those values will be too dark to be the bandage. There is a point
where the bandage is to badly smudged with blood that it would be impossible to distinguish it from
actual blood. For this situation there needs to be more visual marital to analyze if there is other
solution the detect it properly.
The parts that comply with these conditions will be marked as white in a binary image. Below
you can see the conditions for the bandage color detection.
If the following conditions are met we consider it to be blood when the brightness is high
-15 < (B - G) < 15
R > 150
B > 60
G > 60
(R – B) < 100
2. Bandage detection by texture
The second condition to detect the bandage is to get out all the parts that have some sort of
texture. We do this by taking the standard deviation and of the entire frame.
First we convert the frame to a gray scale. secondly we take the standard deviation of the frame.
At last we convert the image to a binary image where the with parts are those with any kind of
texture. This will not always succeed. In some of the available images due to that the white parts are
over saturated when the light of the camera hits the white bandage. Removing the texture from the
bandage. It's possible that when the actual HD footage will not have this problem due to that there
are more pixels that the texture is divided among.
3. Combining color and texture
The third condition is to combine both of the conditions of color and texture to determine if there
is a bandage present in the image. We do this by taking both of the binary images and applying a
bitwise and operation on both of them.
Once we have the part where both the color and texture conditions are met we look for the
biggest object in the result and cut out the smaller objects which can include reflections of the light
on an organ that has a lot of textures through the veins.
24

26. Original
Color detection
Texture detection
Color and texture
Biggest object
Table 6: Results combination color and texture detection
25

27. As you can see in table 6 the bandage gets pointed out it is not always the entire bandage. This is
due to the light that over saturates the image losing some of the texture information of the bandage.
This wouldn't be detected by the standard deviation. But still we can very accurately point out
where the bandage is in the frame.
Where if falls short is that when the image doesn't contain a bandage. This algorithm will still
point out the biggest object where all conditions are met. Most of the times these are the surgery
tools or part of the tools.
But this could be overcome by adding a fourth condition, we can segment the part that is
considered to be the bandage out of the image and process it with the GLCM method. Than all the
reasons why we didn't implemented it isn't a factor anymore. If we segment only the part marked in
blue we will have a small area to calculate the GLCM off. Then the conditions of both color and
texture will ensure that the GLCM will have less combinations, than if we where to calculate the
entire frame for the bandage with the GLCM method.
26

28. Intel Thread Building Blocks (TBB)
The goal of a programmer in a modern computing environment is scalability: to take advantage
of both cores on a dual-core processor, all four cores on a quad-core processor, and so on.
Threading Building Blocks makes writing scalable applications much easier than it is with
traditional threading packages. The advantage of Threading Building Blocks is that it works at a
higher level than raw threads, yet does not require exotic languages or compilers.
Threading Building Blocks: tasks instead of threads.
Most threading packages require you to create, join, and manage threads. Programming directly
in terms of threads can be tedious and can lead to inefficient programs because threads are low-
level, heavy constructs that are close to the hardware. Direct programming with threads forces you
to do the work to efficiently map logical tasks onto threads. In contrast, the Threading Building
Blocks runtime library automatically schedules tasks onto threads in a way that makes efficient use
of processor resources. The runtime is very effective at load-balancing the many tasks you will be
specifying.
By avoiding programming in a raw native thread model, you can expect better portability, easier
programming, more understandable source code, and better performance and scalability in general.
Thread Building Blocks: Flow Graph
The flow graph feature from the TBB library is designed to let developers easily express parallel,
reactive and streaming applications. Unlike the other elements from the TBB library the flow graph
can be any kind of structure. This gives us the freedom to tailer it to what the application needs to
perform its best and most efficient (Figure 16). The base of the application will be made in a flow
graph manner. This gives us the ability to let a lot of different detections to run simultaneously,
using all of the hardware would not only save us in time that is required to process all of the data
but also gives us room to process more without a performance loss visible for the end user (robot).
27
Figure 16: A simple
dependency graph

29. Thread Building Blocks: Flow graph components
Within the project we only used a couple of the components of the flow graph, in this chapter we
are going to explain them in some detail and giving some example code on how they work.
1. Graph
A graph object contains a root task that is the parent of all tasks created on behalf of the
flow graph and its nodes. It provides methods that can be used to access the root task, to
wait for the children of the root task to complete, to explicitly increment or decrement the
root task's reference count, and to run a task as a child of the root task.
2. Function Node
A function_node receives messages of an input type at a single input port and generates a
single output message of output type that is broadcast to all successors. Rejection of
messages by successors is handled using the protocol described in Message Passing
Protocol.
3. Multi-function node
This node works in a similar fashion as the regular function node it takes a single input
port. But it can generate one or more messages that are broadcast to successors.
28
Figure 17: Node types

30. 4. Join Node
A join_node is a graph_node and a sender< flow::tuple< T0, T1, ... >>. It contains a tuple
of input ports, each of which is a receiver<Ti> for each of the T0 .. TN in OutputTuple. It
supports multiple input receivers with distinct types and broadcasts a tuple of received
messages to all of its successors. All input ports of a join_node must use the same buffering
policy. The behavior of a join_node based on its buffering policy which can be found in the
documentation of TBB flow Graph.
5. Broadcast node
A broadcast_node is a graph_node, receiver<T> and sender<T> that broadcasts incoming
messages of type T to all of its successors. There is no buffering in the node, so all messages
are forwarded immediately to all successors.
6. Buffering node
A buffer_node is a graph_node, receiver<T> and sender<T> that forwards messages in
arbitrary order to a single successor in its successor set. Successors are tried in the order that
they were registered with the node. If a successor rejects the message, it is removed from the
successor list according to the policy in the Message Passing Protocol, and the next
successor in the set is tried. This continues until a successor accepts the message, or all
successors have been attempted. Items that are successfully transferred to a successor are
removed from the buffer.
A buffer_node is reservable and supports a single reservation at a time. While an item is
reserved, other items may still be forwarded to successors and try_get calls will return other
non-reserved items if available. While an item is reserved, try_put will still return true and
add items to the buffer.
An allocator of type A is used to allocate internal memory for the buffer_node. T must be
copy-constructible and assignable.
7. Split node
This node receives a tuple at its single input port and generates a message from each
element of the tuple, passing each to the corresponding output port.
A split_node has unlimited concurrency, no buffering, and behaves as a broadcast_node
with multiple output ports.
29

31. Simple TBB Flow Graph example
Code for Figure 16:
Explanation code on next page.
#include <cstdio>
#include "tbb/flow_graph.h"
using namespace tbb::flow;
//Struct that acts as a function to print a given letter
struct body {
std::string my_name;
body( const char *name ) : my_name(name) {}
void operator()( continue_msg ) const {
printf("%sn", my_name.c_str());
}
};
int main() {
//Creation of the graph body
graph g;
//Creation of the broadcast node
broadcast_node< continue_msg > start;
//Creation of all the continue nodes, these could also be function nodes
continue_node<continue_msg> a( g, body("A"));
continue_node<continue_msg> b( g, body("B"));
continue_node<continue_msg> c( g, body("C"));
continue_node<continue_msg> d( g, body("D"));
continue_node<continue_msg> e( g, body("E"));
//Creating the connections between all the nodes
make_edge( start, a );
make_edge( start, b );
make_edge( a, c );
make_edge( b, c );
make_edge( c, d );
make_edge( a, e );
//starting and restarting the flow graph to produce the letters
for (int i = 0; i < 3; ++i ) {
start.try_put( continue_msg() );
g.wait_for_all();
}
return 0;
}
30

32. In this example, nodes A-E print out their names. All of these nodes are therefore able to use
struct body to construct their body objects.
In function main, the flow graph is set up once and then run three times. All of the nodes in this
example pass around continue_msg objects. This type is used to communicate that a node has
completed its execution.
The first line in function main instantiates a graph object, g. On the next line, a broadcast_node
named start is created. Anything passed to this node will be broadcast to all of its successors. The
node start is used in the for loop at the bottom of main to launch the execution of the rest of the flow
graph.
In the example, five continue_node objects are created, named a - e. Each node is constructed
with a reference to graph g and the function object to invoke when it runs. The successor /
predecessor relationships are set up by the make_edge calls that follow the declaration of the nodes.
After the nodes and edges are set up, the try_put in each iteration of the for loop results in a
broadcast of a continue_msg to both a and b. Both a and b are waiting for a single continue_msg,
since they both have only a single predecessor, start.
When they receive the message from start, they execute their body objects. When complete, they
each forward a continue_msg to their successors, and so on. The graph uses tasks to execute the
node bodies as well as to forward messages between the nodes, allowing computation to execute
concurrently when possible.
It is also possible to replace the struct with a lambda expression, this is not advised for a function
that will do the same in more then one node, according to the do not repeat yourself philosophy .
But for an unique function it will be a valid option to utilize a lambda expression.
Example lambda expression with a function node:
function_node<string, bool > writeName (g, concurrency, [] (const int name) {
printf( name.c_str());
return true;
} )
In the above example the input type of the function node will be a string and the output type a
bool that will be passed on to all of the successors that will be connected to this function node. The
g will point out to the graph in which the function node will need to work. It is possible to use
multiple flow graphs in the same application. The concurrency is normally a variable you can set
this to a specific amount, serial or unlimited. This will allow you to set how many tasks the flow
graph is allowed to create for that node. For simple tasks you will set it to serial or 1, for the very
intense to unlimited, for everything in between you can chose the amount or make relate it to the
amount of available threads (ex. Concurrency = 2 * nThreads).
31

33. TBB parallel_for
The parallel_for function of the TBB library is the most interesting one if we where to
implement a detection feature with a GLCM. A parallel for can be nested just as a normal iteration
in C++. The difference is that a parallel for will have a concurrency equal to the amount of threads.
To explain by example.
A normal for-loop has to process an array of a 1000 numbers it will process one number every
millisecond taking it one full second to process the entire array. When a parallel_for-loop will
process the same array on a CPU where four threads are available, it will process four numbers
every millisecond.
This will allow the same array to be finish within 250 milliseconds, four times faster. If there are
eight threads available it will be 125 milliseconds and eight times faster to complete the array and
so on. This speed gain will be useful to process and calculate the GLCM of an image and the speed
of it will increase as the hardware of the platform will increase.
There will be a point that the increase of available threads will not matter anymore as you can
see the speed increase will less and less the more threads are available. Four threads give a speed in
crease of 750 milliseconds to a single core, and eight threads give only a speed increase of 125
milliseconds over the four thread setup.
#include "tbb/tbb.h"
void ParallelApplyFoo( float a[], size_t n ) {
parallel_for(blocked_range<size_t>(0,n), ApplyFoo(a));
}
A blocked_range<T> is a template class provided by the library. It describes an one-dimensional
iteration space over type T . Class parallel_for works with other kinds of iteration spaces, too. The
library provides blocked_range2d for two-dimensional spaces as well. It's possible to define custom
spaces like this.
The iteration space here is of type size_t , and it goes from 0 to n-1 . The template function
tbb::parallel_for breaks this iteration space into chunks and runs each chunk on a separate thread.
32

34. Application structure
Explaining the components:
1. This receives the captured frame and sends it to the 4 components that can run parallel next
to each other each doing it's own task.
2. Calculates one part of the equation for the standard deviation.
3. Calculates another part of the standard deviation equation.
4. Processes the frame and provides a binary image of where all the color conditions are met
that could possible be the bandage.
5. When process 2, 3 and 4 are finished this will use the results of 2 and 3 and apply the
standard deviation on this. Then transform the results to a binary image as well pointing out
with white where the texture is present. This will be followed by comparing the binary
image of the texture with the color. In the resulting binary image of this comparison only the
pixels where both the color and the texture is present will remain. After this we will look for
the biggest object within the binary image and draw this part out on the original image in
blue.
6. This component will process the image for blood. The result of this will be a binary image
where white is what is considered to be blood and black what is not.
7. This component will only take all of the results and relay them to the user interface.
Transforming them first from the OpenCV Mat structure to the Qimage/Qpixmap format for
the Qt graphical user interface
8. This component waits until both the per-calculations (2 and 3) are finished for the standard
deviation and the bandage color detection component (4). Once all three are done it will give
the signal to continue to the bandage detection component(5).
9. This component will wait as well but this will wait for the blood detection (6) and the
bandage detection (5) to be finished before giving the go a head signal to display component
(7)
33

35. User Interface
As can be seen in the Figures 18 to 21, we have kept the user interface simple. This user
interface is only for demonstration purposes only. When the application is going to be deployed the
full screen should be occupied by the original video stream. The bandage detection and blood
detection are only needed for the robot to assist. There's no need to relay this information to the
surgeon.
In Figure 18 under source there will be two options to chose a source. This can be either the
camera or a video file. The camera is selected by default and when clicked it takes the first available
camera. This can be altered as well to take in another parameter to select the source of the camera.
When the video option is clicked a file employer will open (Figure 20) to select a video file to play.
In Figure 19 the action to start and stop is build in there is a pause option is not build in since
this application targets a camera and not a video. The video is only for the testing purposes of this
application.
In Figure 21 the application is in run time. In the future the whole screen should be taken in by
the actual video without any of the detection markers visible to the users.
34
Figure 18: Source options
Figure 19: Action options

36. 35
Figure 20: File explorer
Figure 21: During runtime

37. Recommendations for the future
1. Accelerometer
At this moment it is impossible to detect solely on the video if there is bleeding. Only if there is
blood, not the process of bleeding. To detect this it would require the video to be still at all times to
see a difference in the amount of blood over time. A best possible solution for this is to upgrade the
camera with an accelerometer. This will allow for the detection if the camera is still or moving.
Once the camera is still the application can start keeping records of the blood within the frame, if
there is an increase over time (1-2 seconds/20-40 frames), you can say that there is a bleeding in
process. Without being capable to detect if the camera is still it is certainty that it will provide false
positives every time the camera moves forward or backwards. Detecting if the camera is moving on
a pure visual basis will take a heavy toll on the processing resources or result in a loss of data
depending on the technique used.
There is an algorithm that can focus on one point of interest in the image and when that changes
it can say that the camera has moved, this would be ideal to use if there are no surgeon tools that are
moving throughout the image and will cross that point of interest multiple times, this will provide
false positives if the application thinks the camera is moving it will not keep track of the blood in
the image. If at that moment there would occur a bleeding you would also have a false negative for
the blood detection.
So the most viable solution to keep track of the camera is that there is a sensor placed in the
camera to detect this and this will be signaled to the application to have the most accurate
measurement for this. The camera does not need to be perfectly still. As long as there are no big
movements the detection should be easy to handle, by just counting the amount of white pixels in
the binary image and see if that number increases over a certain amount of frames. This will also
save on the processing resources required.
36

38. 2. More and real data
At this point we can say that the video and images available to work with were sufficient to get a
general idea of what we are dealing with. But most of the data is limited to a certain amount of
situations and often jumping from one part to another. They are meant to teach medical students on
what kind of problems can arise during a laparoscopic surgery. It's not sufficient for testing the
algorithms on these videos. For future work on this project it would be best that there is at least one
full video of a surgery. More data will always be better with the actual video resolution that will
need to be processed.
Testing the algorithms on this kind of data will provide a better insight on how efficient the
algorithms need to be written as well to give a better way to analyze the difficulties the algorithms
might encounter.
An example of this is to see how the algorithm reacts to changes in the illumination. In the
current available video material we can see that the illumination takes a big toll, but the differences
are only seen when we change between videos. It's unclear if this will also be the case if the source
camera is the same. If this is also the case a part of that video needs to be cut out analyzed and the
algorithm needs to be adjusted to take in account the illumination changes. This will require to
make the parameters for the blood detection increase and decrease as the illumination value
changes. This will be a linear process and therefore should not be hard to take in account.
Another reason to work with actual data is that the resolution will be higher then the current
available data. This will make texture detection easier since there are more pixels that the texture is
divided among. This will also inflict required processing resources. But this will also be a good then
the program can be tested with the actual load it will require to process.
37

39. 3. Ideas for detection efficiently
• Explore to combine HSV and RGB to detect blood more effective
HSV has problems detecting to red being at 0° and one tint of red can be at 1°
while another can be 359°. The color red should be turned 180° degrees to solve that
issue and from there on the combination of HSV and RGB might be interesting to
explore as well to make a better distinction between blood and what is not blood.
• Texture detection on a HD video
At this moment, the full frame is being processed to find the bandage color and the
bandage texture. The texture detection requires a lot of resources and will be too
much on a HD image. A suggestion to take of a load is to let the part for bandage
color run first and when an object of interest is found segment that part and process it
for the texture of a bandage.
• Split up blood and bandage detection
The bandage and blood detection work both on a 30 frames per second base.
Maybe it could be interesting to keep the blood at 30 frames per second and alter the
detection frames for the bandage to 10 frames per second of maybe even lower since
this is less critical.
This will mean that the TBB structure needs to be reevaluated. Because for the
blood detection alone. TBB will cause more an overhead then an efficiency profit.
The blood detection could run on a single thread while TBB would be a perfect
match for the bandage detection.
• Improved texture detection algorithm
The process of texture detection is intricate because the bandage appearance in the
images is hardly ever alike. Lighting often varies over images as well as color due to
the presence of blood. Also if there's no bandage it will point out the biggest object
that meets all of the conditions.
We have implemented a simple approach to texture detection that consists in
computing the standard deviation in a sliding window. For each pixel, we calculate a
description of the texture based only in the standard deviation of gray level in its
neighborhood. We have chosen this statistic because of its discriminating capability
in our images and the efficient way it can be computed.
However the object detected is not certain to be a bandage, for this it's best to
segment the detected object out of the frame and apply the GLCM on it to classify it.
Others statistics should be explored such as energy, entropy, contrast or correlation. A
combination of some of them may improve the detection of the bandage in the
segment.
38

40. Conclusion
The developed application accomplishes the detection tasks initially considered. The algorithm
runs in real time detecting fairly well for both blood and medical bandages. There is room for
improvement but this can be carried out without compromising it's real time execution. The
implemented TBB framework doesn't need to be manually reprogrammed to add execution tasks
when additional processing resources are introduced. If the computing demands exceed the
available resources, the same program will optimize it's parallel execution as best as possible. The
TBB flow graph leaves room for expansion in the code to add other texture detection methods for
other objects or improvement of already implemented algorithms.
Personally this internship in Valladolid has been a very rewarding experience. I gained a lot of
knowledge and skills. From C++ to the little bit of the Spanish that I learned during my stay. But
most importantly the concepts of computer vision and parallel programming that I acquired.
Looking back on the project I would say that this project would have been better suited for a
master student in computer science. The two concepts that I had to learn for this project are in there
courses. The time that I needed to spend on learning C++, OpenCV and TBB could have been used
to improve the project further. Due to a lack in background knowledge, I created a lack of time for
this project.
When I had the choice to take this project, I was well aware of the challenge it would give me as
a programmer. It is also the reason that I picked it, to make something that would be equally
challenging as interesting. This project gave me both, the only thing I could wish for now, is more
time to see the project through. The results I have made are a step towards finishing the project, but
there is still room for improvement and an even more urgent need for more video material to
analyze, test and perfect the algorithms.
39

41. Bibliography
1. OpenCV2 Computer Vision Application Programming cookbook
- Robert Laganière
2. Learning OpenCV second edition
- Adrian Kaeher and Gary Bradshi
3. Intel Thread Building Blocks
- James Reinders
4. Video Demystified: A Handbook for the Digital Engineer.
- Keith Jack
5. Automatic Bleeding Detection in Wireless Capsule Endoscopy Based on RGB Pixel Intensity Ratio
- Tonmoy Ghosh, Shaikh Anowarul Fattah and Khan Arif Wahid
6. Textural feature for image classification
- Robert Haralick, K. Shanmugan and Its'Hak dinstein
7. Optimization of Texture Feature Extraction Algorithm
- Tuan Anh Pham
8. Statistical Texture Measures Computed from Gray Level Coocurrence Matrices
- Fritz Albregtsen
9. Texture-boundary detection in real-time
- Ray Hidayat
10. Real-time texture boundary detection from ridges in the standard deviation space
- Ray Hidayat
11. A Computational Approach to Edge Detection
- John Canny
12. The HSV and HSL Color Models and the Infamous Hexcones
- Douglas A. Kerr
13. www.cplusplus.com
14. http://doc.qt.io/qt-5/qtexamplesandtutorials.html
15. http://www.fp.ucalgary.ca/mhallbey/tutorial.htm
40