This document describes a medical hands-free system using gesture recognition to help surgeons keep track of surgical instruments and materials without direct contact. It uses a Leap Motion controller to detect hand movements and recognize customized gestures. Image moments are used to distinguish between gestures by calculating weighted averages of pixel distributions in captured images. The goal is to introduce hands-free control to medical settings where direct contact poses infection risks, helping surgeons prevent accidental retention of foreign objects in patients.

1. 1
Medical Hands-Free System
Authors
Avraham Levi, Guy Peleg
Supervisors
Ron Sivan, PhD , Yael Einav, PhD
Abstract - Contemporary Human Computer Interaction (HCI) is based on usage of multiple
devices such as keyboard, mouse or touchscreen, all of which require manual contact. This
imposes a serious limitation when it comes to deal some medical equipment and applications
where sterility is required, such as in the operating room. Statistical reports show that keyboard
and mice contain more bacteria than lavatories. We focus in our project on the application of a
hands free motion detector in surgical procedures. The purpose of the application in this setting is
to enable a surgeon to account for the placement of surgical items to prevent forgetting any inside
the patient by mistake. The application makes use of a hand motion detector from Leap Motion,
Inc. We hope it will give us a good opportunity to solve this issue, and, furthermore, introduce the
concept of hands free controller to the medical world.
Keywords - Human computer interaction (HCI), Leap motion (Leap) , Infra-red (IR) ,Gesture,
Image moments, Retained foreign object (RFO).
1. INTRODUCTION
This project addresses the issue of recognizing hand and finger movement. Using a 3D
motion controller, and utilizing computer vision method which is image moments. we
hope to solve the issue of defining the hand motion patterns. In practice, we will need to
expand the SDK the manufacturer of the device provides, to enable developers to record
and create their own custom made gestures with our SDK we will develop an application
that we hope will help surgeons in the operating room by assisting them in keeping track
of instruments and materials they use during surgery so they wouldn't forget anything
inside the patient. Our purpose is to introduce the possibility of hands-free control of a
computer to the medical world, which is usually very conservative and is slow in adopting
new technologies.
2. THEORY
2.1. Background and related work
2.1.1. Gesture
Gestures are a method of communication using only hand movement, such as sign
language used by the deaf (See Figure 1). A gesture includes movement of body parts
(hands, fingers) or various implements, such as flags, pens, etc.
Figure 1 : Hand gestures used in sign language

2. 2
2.1.2.The leap motion controller
The Leap Motion controller [1] is a computer hardware sensor designed to detect hand
and finger movement requiring no contact or touch. The main components of the device
are three IR LEDs for illumination and two monochromatic IR cameras. The device is
sensitive to hand and finger movements to a distance of about 1 meter. The LEDs
generate a 3d pattern of dots of IR light, and the cameras generate 200 frames per
second of captured data. This data is sent via USB to a host computer. The device
comes with a development kit (SDK) capable of recognizing a few simple gestures. The
main part of our project will consist of expanding the SDK for enabling users to define
their own, more complex gestures. In particular, we plan to use that environment to
create applications based on customized gestures to control medical equipment used in
sterile conditions, where forms of control which require touch are precluded.
Figure 2 : The inside components of the Leap Motion controller
2.1.3.Other products
Other products in the 3D motion controller domain are Kinect [2] and Intel Perception
Camera [3] . Compared with the Leap Motion device, the Kinect does not come with
finger recognition and its main purpose is to capture the whole body, therefore it is much
slower in capturing hand and fingers movement, too slow for gesture recognition.
Because the Kinect camera captures the soundings as well, computing time is much
longer. Moreover, the Kinect is much larger and requires more space. The Kinect,
however, has better range and can capture the surroundings from a distance.
Another comparable device is the Intel Perceptual Computing Camera. However, It is
available for purchase, although it rather more expensive than the Leap Motion ($200 for
the device and $150 for Intel SDK). The device is not designed specifically for gesture
recognition, it works in visible light (not IR), and depends on ambient lighting. It supports
voice and various other technologies that are immaterial to our application.
2.1.4. The health care industry
Research [4] [5] strongly supports that computer keyboards and other input devices are
source of bacteria and cross-contamination that can lead to hospital acquired infections.
Therefore washable keyboards, mice, TV remotes and mobile products with anti-
microbial product protection should be put in place along with proper disinfection
protocols to reduce the risk of infection and cross-contamination.
2.1.5. Surgical inventory count
Oftentimes surgical instruments may accidentally be left behind in a surgical cavity
causing, in the worst case, severe infection or even death [6]. These types of events
(called retained foreign object – RFO) are considered as "never events" , namely – they
are preventable medical errors. Unlike other medical errors, RFO errors were declared as
"touch zero" errors – namely, the goal is to reach zero events, since this type of error is
considered easy to prevent [7].

3. 3
Over the history of surgical procedures a strict surgical inventory count protocol was
developed and is now obligatory in all surgical settings around the world. Two nurses are
responsible for counting all surgical sponges, needles and instruments in most surgical
settings. However, there are some surgical procedures in which the count protocol is not
performed on a regular basis. These are "small", "simple" procedures in which there is no
nurse continuously aiding the surgeon (e.g., episiotomy). Naturally, where there is no
formal count protocol performed or no count at all, the chances for retained surgical
sponge rises dramatically [6] . In these procedures the surgeon has to rely on his or her
memory to recall how many items were used or how many items were in the set that was
opened (for example 10 pads, 2 needles and 3 gauzes). To account for all items the
surgeon actually needs to compare the number in his memory (e.g., "there were 5 pads in
the set and then I opened a package of 5 more, and there were 3 needles in the set.") To
the number of items found at the end of the procedure. However, to keep these numbers
in mind for the whole procedure is much of a burden and since short term memory is so
vulnerable, there is a good chance that the surgeon may make a mistake.
In the past 10-15 years there were few technological developed to support the counting
protocol: "SURGICOUNT Medical" and "SmartSponge System" are two examples. These
systems require hand contact to operate and are controlled mostly by the nurses
responsible for the count. Our solution is designed to help surgeons in cases where there
is no nurse available to write down (or type) items' count. Using the Leap Motion device
for this purpose will provide the surgeon means to document the usage of items without
the need to write down or type anything. This solution is the only solution (until now) that
surgeons have to document anything themselves during surgery. It is expected to
dramatically reduce memory load and promote a safer care for patients.
Figure 3 : Intra-operative radiograph performed because of an incorrect sponge count in a 54 year
old woman undergoing urethral suspension. The radio-opaque marker (arrow) of a 4 x 4 inch
surgical sponge is visible in the pelvis. The sponge was identified.
2.2. Detailed description
2.2.1. Gesture recognition
Gesture recognition refers to interpretation of human gestures via mathematical
algorithms. The IR cameras in the Leap Motion device reads the movements of the
human hand and communicates the data to a computer that uses the gesture input to
control devices or applications. Using mathematical algorithms the computer can analyze
the captured data and categorize it to one of several predefined patterns.

4. 4
2.2.2. Preprocessing
For the moment we confine our attention to planar gestures – gestures in which the index
finger traces a path in a plane. We assume it will be easier for humans to reproduce and
hope it will also be simpler to recognize, while not restricting the repertoire of possible
gestures too severely.
It is obvious that any free hand motion cannot be constrained to be completely planar:
planarity will be only approximated. We therefore find, as a first step in interpreting
gesture data, the plane whose distance from the captured gesture trace is minimal, using
Singular Vector Decomposition (see 2.2.3).
2.2.3. Singular vector decomposition
Singular Vector Decomposition (SVD) is a factorization method with many useful
applications in signal processing and statistics.
One useful application that can be used for our purpose is fitting planes and lines by
orthogonal distance regression [8] .Say we want to find the plane that are as close as
possible to set of 𝑛 3-D points (𝑝1, … , 𝑝 𝑛) which we captured by the device.
Let the matrix A of size n x 3 holds the 𝑝𝑖 = (𝑥𝑖, 𝑦𝑖, 𝑧𝑖) in each row :
𝐴 = [
𝑥1 𝑦1 𝑧1
: : :
𝑥 𝑛 𝑦𝑛 𝑧 𝑛
]
Using the Transpose Matrix operation we create the 3 x n AT
:
𝐴 𝑇
= [
𝑥1 … 𝑥 𝑛
𝑦1 … 𝑦𝑛
𝑧1 … 𝑧 𝑛
]
Building matrix B by Matrix Multiplication between A and AT
yielding a 3 x 3 matrix :
𝐵 = 𝐴 ∙ 𝐴 𝑇
Solving the eigenvalue equation for matrix B:
det(𝐵 − 𝜆 ∙ 𝐼 ) = 0
This equation is a polynomial of degree 3, and hence has 3 solutions. Under the
conditions of the problem at hand, they are expected to be real, positive and distinct.
λ1, λ2, λ3 ∈ ℝ+
For each of the λ values we compute the corresponding eigenvector:
(𝐵 − 𝜆𝑖 ∙ 𝐼 )𝑢̅ 𝑖 = 0
Using the eigenvalues we have calculated, we need to create the U matrix that built from
each of the eigenvectors.
U = [
𝑢1𝑥
𝑢1𝑦
𝑢1𝑧
𝑢2𝑥
𝑢2𝑦
𝑢2𝑧
𝑢3𝑥
𝑢3𝑦
𝑢3𝑧
]
⏟
u1 u2 u3

5. 5
The eigenvector that belongs to the minimal value 𝜆𝑖 is the normal for our working
plane.
2.2.4. Projecting point on the new plane
After we have found the closest plane to all the given points, we project each point onto
the plane to be given set of N coplanar. We use the canonical form of the plane equation
to compute the distance of each point 𝑃𝑖(𝑥𝑖, 𝑦𝑖, 𝑧𝑖)from the plane:
𝐴𝑥𝑖 + 𝐵𝑦𝑖 + 𝐶𝑧𝑖 + 𝐷
√𝐴2 + 𝐵2 + 𝐶2
= 𝑑𝑖
𝑃𝑖
⃗⃗ − 𝑑𝑖 𝑛⃗ = 𝑃𝑖′⃗⃗⃗⃗
2.2.5. Reducing dimensions
Now that we have moved all points into one plane, we want to reduce the number of
coordinates of each point from 3 to 2. Let {𝑃𝑖(𝑥𝑖, 𝑦𝑖, 𝑧𝑖)} be that set of points, and let
𝑀(𝐴𝑥 + 𝐵𝑦 + 𝐶𝑧 + 𝐷 = 0) be that plane. In general, the plane 𝑀 need not be 𝑀0(𝑧 = 0),
the XY plane, and therefore the 𝑧 component of the points 𝑃𝑖 need not vanish. We
therefore construct a Cartesian system on plane 𝑀 by choosing two perpendicular lines
on 𝑀, namely LX Let and LY. 𝐿 𝑋(𝐴𝑥 + 𝐵𝑦 + 𝐷 = 0) is the intersection line between 𝑀 and
𝑀0. As the origin we choose the point 𝑂 (0, −
𝐷
𝐵
, 0) on 𝐿 𝑋. 𝐿 𝑌 will then be the line lying in
𝑀0 that passes through the origin 𝑂 and is perpendicular to 𝐿 𝑋. The distances 𝑥′𝑖 and
𝑦′𝑖 of every point 𝑃𝑖 from 𝐿 𝑋 and 𝐿 𝑌 correspondingly will act as the coordinates of the
points for further analysis.
Considering Figure 4, we find the distance d on 𝑀0 between the projection on 𝑃𝑖 on 𝑀0
and 𝐿 𝑋, which by definition also lies on 𝑀0. This distance d and the z coordinate of point
𝑃𝑖 form a right-angle triangle whose hypotenous is the distance of 𝑃𝑖from LX, hence is y'i.
Defining point Q as the intersection of y'i and 𝐿 𝑋, the distance from Q to the origin O is the
distance on 𝑀 from 𝑃𝑖 to 𝐿 𝑋, and hence is x'i.
Figure 4 : Reducing the number of coordinates
Developing the math we get:
22
2222
'
))(())((
BA
ABx
B
D
yAxB
B
D
yAB
x
iiii
i



2
22
2
' )(
i
ii
i z
BA
DByAx
y 




6. 6
2.2.6. Building the image
Once a planar shape is obtained, we find a bounding rectangle for the points inside.
Forming a matrix M with the dimensions of that rectangle, we initialize the matrix
according to this formula:
𝑀(𝑥, 𝑦) = {
1 𝑥, 𝑦 𝑖𝑠 𝑎 𝑑𝑎𝑡𝑎 𝑝𝑜𝑖𝑛𝑡
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Yielding a matrix representing the image.
2.2.7.Image moments
In order to distinguish between patterns we compute Image Moments [9] [10]. Image
moments, each being a real number, are various weighted averages of the image pixels
intensities, representing increasing detail of pixel distribution, such as centroid, area, and
information about orientation. Moments are invariant to translation, and some are
invariant to rotation as well, and we limit our attention to those only.
We have used the following mapping function:
𝑓(𝑥, 𝑦) = {
0 , 𝑖𝑓 𝑡ℎ𝑒 𝑝𝑖𝑥𝑒𝑙 𝑖𝑠 𝑤ℎ𝑖𝑡𝑒
1 , 𝑖𝑓 𝑡ℎ𝑒 𝑝𝑖𝑥𝑒𝑙 𝑖𝑠 𝑏𝑙𝑎𝑐𝑘
We first calculate the raw moments
𝑀 𝑝𝑞 = ∑ ∑ 𝑥 𝑝
𝑦 𝑞
𝑓( 𝑥, 𝑦)
𝑦𝑥
𝜇 𝑝𝑞 = ∑ ∑(𝑥 − 𝑥̅)
𝑝
(𝑦 − 𝑦̅) 𝑞
𝑓(𝑥, 𝑦)
𝑦𝑥
𝑤ℎ𝑒𝑟𝑒 𝑥̅ =
𝑀10
𝑀00
𝑎𝑛𝑑 𝑦̅ =
𝑀01
𝑀00
We have chosen to use 9 moments for now, but this number may increase if it turns out
good recognition needs more. Now that we have calculated the moments we need make
sure that they are scale invariants so we use the following formula:
𝜂𝑖𝑗 =
𝜇𝑖𝑗
𝜇00
1+
𝑖+𝑗
2
Now we want our moments to be invariant under rotation, most known set of moments
are Hu's [11] set of invariant moments, also known as the 7 moments of Hu:
𝝓 𝟏 = 𝜂20 + 𝜂02
𝝓 𝟐 = (𝜙1)2
+ 4𝜂11
2
𝝓 𝟑 = (𝜂30 − 3𝜂12)2
+ (3𝜂12 − 𝜂30)2
𝝓 𝟒 = (𝜂30 + 𝜂12)2
+ (𝜂30 + 𝜂21)2
𝝓 𝟓 = (𝜂30 − 3𝜂12)(𝜂30 + 3𝜂12)[(𝜂30 + 𝜂12)2
− 3(𝜂30 + 3𝜂12)2]
+ (3𝜂21 − 𝜂03)(𝜂03 + 3𝜂21)[3(𝜂30 + 3𝜂12)2 − (𝜂21 + 𝜂03)2]
𝝓 𝟔 = (𝜂20 − 𝜂02)[𝜂30 + 𝜂12)2
− (𝜂21 + 𝜂03)2
] + 4𝜂_11 (𝜂30 + 𝜂12)(𝜂21 + 𝜂03)
𝝓 𝟕 = (3𝜂21 − 𝜂03)(𝜂30 + 𝜂12)[(𝜂30 + 𝜂12)2
− 3(𝜂21 + 𝜂03)2] − (𝜂30 − 3𝜂12)(𝜂21 + 𝜂03)[(3(𝜂30 + 𝜂12)2
− (𝜂21 + 𝜂03)2]

7. 7
The 9 moments of a gesture are taken as coordinates of the gesture in some abstract 9-
dimensional space. It is assumed that the representation of similar gestures will
congregate into "clouds" whose extent, in Euclidean metric, will be small compared to the
distance between "cloud" centroids. Those centroid will be calculated for each "cloud"
and will be saved, and with each insertion of new moments the centroid will be updated,
the usage of this data is represented in the next section.
2.2.8. Minimum distance algorithm
Minimum Distance algorithm is basic method to compare distance between our ongoing
gesture and the "clouds" of calculated method, the main idea of the usage of the algorithm
in this context is to calculate the distance between the ongoing gesture centroid to the other
centroids, so in high probability the "cloud" with the minimum distance to the ongoing
gesture is the same gesture,
We have considered two distance function:
𝑙𝑒𝑡 𝑋, 𝑌 𝑏𝑒 𝑉𝑒𝑐𝑡𝑜𝑟 𝑖𝑛 𝑘 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑎𝑙 𝑠𝑝𝑎𝑐𝑒
𝐸𝑢𝑐𝑙𝑖𝑑𝑒𝑎𝑛 ∶ √∑(𝑋𝑖 − 𝑌𝑖)2
𝑘
𝑖=1
𝑀𝑎𝑛ℎ𝑎𝑡𝑡𝑎𝑛 ∶ ∑|𝑋𝑖 − 𝑌𝑖|
𝑘
𝑖=1
We assume that the number of gesture recorded is finite and considerably low. So in that
case we can allow us to calculate each distance. Since the number of rows of data is finite
the complexity of this process the complexity is low. Which is 𝑂(𝑆).
The Algorithm:
Let 𝑆 = 𝑐1, … , 𝑐 𝑛 be the set of centroids of gesture {𝑔𝑖} in feature space.
Let p be the point in representing a new gesture to be recognized in feature space
 Compute the distance d(p, ci)
 Find the minimum distance.
 Identify the new gesture as the gesture i with the minimal distance
 Return the id
2.3. Expected results
The final product will be implemented in each device that require hand contact
especially for counting procedures in operating room. This would reduce the number
of retain sponge cases.
Our SDK will give the opportunity for developers around the world to implement
application which based on custom-made gesture.

8. 8
3. PRELIMINARY SOFTWARE ENGINEERING DOCUMENTATION
3.1. Requirements (Use Cases)
3.2. GUI
In this section we would like to introduce our UI prototype for the project. This prototype
may demonstrate the usability of the SDK and the Medical application.
3.2.1. SDK expansion
We have divided the toolkit to 3 major steps which give the user all the functionally he
needs to create a custom made gesture. Each step screen has its own purpose, we gave
the user a wizard that give him information about the current step.
Step 1 (Figure 5): In this step the user records the custom made gesture he or she
would like to create.
Step 2 (Figure 6): In this step the user trains the computer to recognize the new gesture
by recording additional examples preferably by different people.
Step 3 (Figure 7): In this step the user tests the system by presenting the gesture to see
if it is recognized.
Step 4 : In this step the user gets the information about the new gesture he has just
created.
Each screen has the following components (See figures below):
A) Wizard- A UI component which presents the user a sequence of screens that lead him
through a series of well-defined steps.
B) Gesture grid panel – A real time panel displayer which tracks the movement of the
user finger and display this movement in this panel.
C) Coordinates table – A table which is filled with the point coordinates of each finger
position that was detected in the current frame.
D) Operation buttons – the four control buttons which can be clicked upon to perform an
action.
E) Status bar – A status line that gives an information about the Leap controller
connection , and about the action that is performed.
F) Help button – a control button which stands for giving help for the user.

9. 9
G) Match rate bar –a bar that presents the likelihood rate of the gesture that the user is
doing with the custom made gesture that was saved in the record step. (refers to steps
2+3)
G
A
B
C
D
E
F
Figure 5 : Record Step
Figure 6 : Train Step
Figure 7 : Exercise Step

10. 10
3.2.2. Items counting application
We design our medical application to be simple and intuitive to the surgeons that will
use it eventually. We focused on the Episiotomy procedure kit since it perfectly
demonstrate the usage of our system.
As we can see in Figure 8 below, there are 4 components:
A) Item panel – this panel includes elements : gesture icon , item image and counter .
B) Undo panel – this panel show the gesture icon that the surgeons needs to do in
order to undo his last gesture action.
C) More panel - this panel show the gesture icon that the surgeons needs to do in
order to add different item to the surgery.
D) Status bar – A status line that gives an information about the Leap controller
connection , and about the action that is performed.
Using these components the surgeon makes the gesture according the item that was
entered to the surgery environment and the counter that belongs to this gesture is
incremented, and a feedback will be shown on the screen.
3.3. Program structure – Architecture, Design (UML diagrams)
3.3.1. Software architecture
The SDK itself will divide into two main parts, the main part the core of the program
with all the logic is going to be C++ based application. When the GUI is going to be
done with QT or WPF/WINFORMS and will communicate with the C++ program that
will transfer event for each gesture that have been made, as for the medical application
the whole application is going to be written with WPF and communicate with the SDK.
The following API is an initial prototype of the interface of our SDK:
Init() – this function initialize the setting of the device.
SetupListener() – this function connect to the infrastructure of the system.
CallBackFunc() – this interface stands for the user in order to send it as a delegate
function to the device.
OnClose() – operation to be done when the application is close.
BA
C
D
Figure 8 : Instruments counting application

11. 11
SDK Class Diagram
Medical Counter Class Diagram

12. 12
3.4. Testing plan
3.4.1. Testing plan for the SDK
3.4.2. Testing plan for the medical application
REFERENCES
[1] L. M. Controller, "Leap Motion Specs," Leap Motion Inc, 2013. [Online]. Available:
https://www.leapmotion.com/product.
[2] "Kinect WikiPedia," [Online]. Available: http://en.wikipedia.org/wiki/Kinect.
[3] I. Corp, "Intel Dev Guide for perceptual-computing," Intel Corp, [Online]. Available: http://software.intel.com/en-
us/vcsource/tools/perceptual-computing-sdk.
[4] A. K. Al-Ghamdi, S. M. A. Abdelmalek, A. M. Ashshi, H. Faidah, H. Shukri and A. A. Jiman-Fatan, "Bacterial
contamination of computer keyboards and mice ,elevtors and shoping carts," African Journal of Microbiology
Research, no. 5(23), 2011.
[5] D. C. -. A. N. M. Unit, "Abc News - Your Keyboard: Dirtier Than a Toilet," 5 May 2008. [Online]. Available:
http://abcnews.go.com/Health/Germs/story?id=4774746.
[6] M. M. D. M. S. L. S. M. a. M. J. Z. M. Atul A. Gawande, "Risk Factors for Retained Instruments," The new england
journal of medicine, 2003.
[7] M. t. S. F. B. M. K. P. S. R. J. T. S. B. M. a. H. A. K. B. C. William Kaiser, "The Retained Surgical Sponge," ANNALS
OF SURGERY, vol. 224.
[8] W. Comunity, "SVD - WIKI," [Online]. Available: http://en.wikipedia.org/wiki/Singular_value_decomposition.
[9] J. Flusser, "On the independence of rotation moment invariants," Pattern Recognition, no. 33, 1999.
[10] J. F. a. T. Suk, "Rotation Moment Invariants for Recognition of Symmetric Objects," IEEE TRANSACTIONS ON
IMAGE PROCESSING, vol. 15, 2006.
[11] J. L. Zhihu Huang, "Analysis of Hu's Moment Invariants on Image Scaling and Rotation," ECU Publications Pre,
2011.
Test name Scenario Expected result
Record input data We record input from the
device
A file with the recorded data will
appear in the DB.
Call SVD function over input data file SVD function gets the raw data
from the file
Function return the normal of
closest plane.
Call image moments function Image moment will be
extracted from the bit map
matrix.
All moment will be recorded to
the DB.
Record And test 3 Different Gestures
repeat this with different people
We record 3 different gesture
and test them
There system will notice
between the 3 different
gestures
Test name Scenario Expected result
Open application User starts the application Application opens in main
screen with no errors
User imitates gesture shown on the
screen
The user follow the gesture
shape.
A counter of the instrument in
being increased , and the
shape is being highlighted .
The user chose to undo the action User imitates gesture of undo. The last counter is being
decreased.
Open surgery log The user press log button. A log of his actions is being
shown
"you made X , pad 4X4 was
incremented by 1".
Open surgery inventory report The user click inventory report
button.
A report of all the required
equipment is being shown
"two pair of scalpels" etc.

13. 13