Gesture recognition & manipulation in phone applications for projected surfaces (GRAS) as an (Graduation Project)

1. Gesture recognition & manipulation in phone applications
for projected surfaces (GRAS)
Alaa F.Drabseh , Esraa A.Alsyis ,Wafaa M.Towaiq ,and
Mostafa Z.Ali
Computer Information Department , Jordan University of Science & Technology
Irbid ,Jordan
E-mails : { afdarabseh08, eaalsayis08 , wmtwaiq08}@cit.just.edu.jo
mzali@just.edu.jo
Abstract--- This project focuses on facilitating human GRAS allows humans to interact naturally without any
interaction with the digital word by giving a direct link mechanical device through pointing fingers on the projected
between the digital world and our physical interaction screen and then the cursors move accordingly, where we used
with the real world. Gesture recognition & manipulation four fingers and simulate four cursors for each one .
in phone applications for projected surfaces (GRAS) is a Experiments in the use of cross-correlation as means of
wearable gestural interface that arguments the physical tracking pointing devices for a digital desk, that describing the
world around us with digital information and lets us use potential for applying real time computer vision to users
natural hand gestures to interact with that information. interaction.
The system is been modified and geared towards
applications like: capturing & manipulating thermal
images at hospitals in a portable manner where they get
immediately stored on the phone memory. Another
application is customizing architectural CAD designs via
hand gesture recognition (This is intended for enhancing
the way students perceive instructions on the basics in
their introductory design courses). The latter application
will enhance the way through which students perceive the
model and interactively generate and amend sequence of
modules for a project in an immersive environment.
Keywords— GRAS , Gesture , fiducials , CAD ,Thermal
images
1. INTRODUCTION
People evolved our millions of years to sense the world
around them, through which they perceive the information
around them using their five natural senses. Although their Figure 1: Show the concept of GRAS
mineralization of the computing devices allows us to carry
computing devices in our project, keeping us connected to our
digital world, there is no link between our devices and our Vision-based hand tracking is an important problem in the
interactions with the physical world. This involves a planning field of human-phones interaction, since hand motions and
phase, design phase and a phase of formation of all the gestures could potentially be used to interact with phones in
necessary elements and then connects it to each other. The more natural ways . A number of solutions have been
final product development process in GRAS is normally proposed in our project, but the problem is still far from being
structured as a set of applications grouped in one device in solved since the hand exhibits.
which each application works independently from other
applications.

2. The project prototype is comprised of pocket projector (laser multi- touch and multi-user interaction inherited from phone
or L.E.D projector), a mirror, colored markers , a mobile features, but on a more flexible projected surface.
computing device, a camera with fine resolution and a
Thermal Cam for Thermal images. The camera, mirror and
projector are connected wirelessly to Bluetooth Smartphone
2. METHODOLOGY
device that can easily fit into the user's pocket. The Phone
application then processes the data that is collected by the This project is focuses on facilitating humans interaction
capturing device and does the analysis. The projector projects with the digital word by gives a direct link between the digital
visual information enabling surfaces, walls and physical world and our physical interaction with the real world.
objects around us to be used as interfaces; while the camera Gesture recognition & manipulation in phone applications for
recognizes and tracks user's hand gestures and physical projected surfaces (GRAS) is a wearable gestural interface
objects using computer-vision based techniques. that arguments the physical world around us with digital
information and lets us use natural hand gestures to interact
Application equipments contain from colored markers on with that information.
the user's fingers are recognized with colored tapes on them
by the camera. The camera captures the user in view and
tracks his hand gestures, then sends the data to the smart
phone which processes the video data, to identify the object.
Where there will be a tiny projector (with a mirror) that
displays data sent from the smart phone on any surface in
view like wall, or person.
Figure 3: How we Can using your Fingers as Cursors
2.1 PROJECT IDENTIFICATION & SELECTION
A. Mission statement
• To build a wearable gesture interface that allows users to
improve the manipulate of objects around them from GRAS in
an efficient way ,as a preparation step for controlling the
CAD objects to streaming design processes through the
simulate the gesture recognition of their fingers &
manipulating thermal images at hospitals in a portable
Figure 2: Fingers Movement Examples manner .
The software processes the video stream data captured by B. Programming Language and Software used
the camera and tracks the locations of the colored markers at
the tip of the user’s fingers using simple computer-vision  For Implementation purposes we will use the
techniques. The movements and arrangements of these following software :
fiducials are interpreted into gestures that act as interaction  Visual Studio .Net 2010
instructions for the projected application interfaces. The
maximum number of tracked fingers is only constrained by  Direct X SDK
the number of unique fiducials, thus GRAS also supports  C++/C# as a programming Language

3. 2.2 INITIATION AND PLANNING  Intended for every person who interested is in
A. Statement of work (SOW) knowing the product information before purchasing
it.
 The goal of our project is to develop an applications
for Smart phones that helps use and control of
objects around us in an easy way , as a manipulate B. The project will introduce a mobile technology:
architecture models in a CAD-like environment , by including the following
import the “object” to move it by our fingers in any
surface through gesture fingers movements , and to  Smart Phone
develop an application to hospitals sector that allows  Windows platform written in C++ and C#
them to manipulate thermal images by use the  Projector
fingers as a calibrations to zoom in-out the borders to  Camera and Thermal Camera
take a photo and then process it as an thermal image ,  Flat surface
the last app is Barcode reader by read any supported  Markers ( 4-differenet Colors )
product barcode format and get information regards .
• The first part of our project enables the user to have 2.4 BENEFITS
the look and feel which is the process of the
walkthrough the building to see constriction  The phone user for project apps will do his/her
internally and to know the characteristics of the activates in a highly speed and fixable manner.
building and check its feasibility by themselves any  Provide Portable, scalable and wearable device.
time anywhere with an available surface. when we  Use these Apps in one small wearable device we will
could see and move the object by our self before it is reduce the cost and errors occur.
constricted it will increase the confidence of such  Convert any surface into an interactive touch screen.
designs through easiness multiple scenarios that  The users of the CAD-App can draw on any surface
could not be easy or possible on paper . zoom-in, zoom-out elements by pointing via fingers.
 Speed to obtain thermal images in cases
 The second Part allows medical doctors to quickly
of emergency to thus speed in the provision
see any internal problems and give a suitable cure
of medical care.
even if they outside the hospital .when we could see
 Interact with real time data and information regard to
these images quickly after the accident we will
Products.
reduce the advance symptoms.
 The third part enhances the purchasing process for
any product to eliminate the mediators (self-
catering ) , by reading the information regarding any
product and making the decision to buy it or not . 2.5 COST
A. The major cost is linked with following
equipments and devises :
2.3 SCOPE
A. Target Group Subject  Smartphone (Mobile computing device )
 Projector
 Intended to designers who need a friendly  Camera (phone Cam)
application to design the building with CAD shapes  Thermal Camera
objects even if they are not familiar with CAD  Mirror
software .
 Intended for Hospital sectors who want to gain a
competitive advantage through Speed in the
provision of medical care for their patients.

4. 2.6 RISK Risk Risk Likel Mitigation
Level ihoo Strategy
A .Risk that might be faced in this project with L/M/H d of
suggested strategies of avoiding this risk is Even
shown in table 1 t
Physical M: Team Likel Use of Intranet
Location of is y project
Team dispersed website,
prevents among comprehensive
Risk Risk Likel Mitigation
effective several Communicatio
Level ihoo Strategy
L/M/H d of managemen sites ns Plan
Even t
t Table 1: A sample line graph using colors which contrast
Project well both on screen and on a black-and-white hardcopy
Size
Estimated H: Over 7 Cert Created 2.7 FEASIBILITY STUDY:
Project monthes ainty comprehensiv A. Operational Feasibility:
Schedule e project
timeline with  Our proposed system serves a medical,
frequent architectural and market purposes. It can be
baseline
applied on any smart-phone such Android.
reviews
Project B. Schedule Feasibility:
Definition
Narrow M: Likel Assigned  As an estimation of the time needed to
Knowledge Knowledge y Project complete all features of the system, it needs
Level of able of Manager(s) to
around 6 months with all equipments needed
Users user area assess global
are available through the work time.
only implications
Cost L: likel
C. Economic Feasibility:
Estimates Thoroughl y
Unrealistic y
 Tangible benefits:
predicted
by team  Increase flexibility, and speed of activity.
members.  Opening new markets and increasing sales
Project opportunity.
Staffing
 Using natural fingers gestures to work with the
Project M: Som Continuous
applications.
Team Distributed ewh review of
Availability team at project
makes likel momentum by  Intangible benefits:
availability y all levels.  Convert any surface into an interactive touch
questionab Consultant to screen.
le identify any
impacts
 Users can interact with real time data and
caused by information.
unavailability.  Support multi-user interaction at the same
If necessary, time.
increase
committmment
 Tangible costs:
by participants
 Hardware cost like LED mini-projector and
to full time
status thermal Camera.

5.  Intangible costs: 4- ANALYSIS:
 Loss of customer goodwill. The process of gathering and analysing the system
requirements was done by interested Archie people and
understanding the major problems face them and how they do
 One-Time Cost: their work to understand the basic aspects, and we met health
 New Hardware and software purchase. care staff to recognize the importance of thermal images in
their job , and how our application will help them to make
 System development. their tasks be more flexible and easer . Also, we made a
survey about how much they are interested to have barcode
scan application in their phones.
D. Legal Feasibility:
 Development the project does not conflict with all A. System Requirement:
applicable federal statuses.
 The system should build the knowledge of
3- SCHEDULE PLANNING:
transaction in our project.
1. Tasks
 The system should provide the users with more
flexibility in terms of install, uninstall, update and
use it easy.
 The system should be able to be installed on different
smart phones.
B. User Requirement:
 User should be able to know how to use our
application phone.
 User shall be familiar with the building features.
 Users should be able to install, update and uninstall
softwares .
C. Function Requirement:
 Simulating dynamic operations.
 Coordinating detailed design
 Scheduling constriction
 Demonstrating Technical compliance
Table 2: Project Tasks with estmate time to complet it
 Design Review
2.Gantt Chart D. Non Function Requirement
 The system shall be clear enough to be understood by
the user.
 Usability .
 The system should be compatible with different
platforms used.
Figure 4:Show Order of Tasks to get the project

6. 5 -DESIGN visual models of 3-D objects and take advantage of hardware
 5.1 Object Oriented Diagram acceleration .
D. System.Drawing:
The System.Drawing namespace provides access to GDI+
basic graphics functionality.
E. AutoCAD .NET API:
The AutoCAD .NET API is made up of different DLL files
that contain a wide range of classes, structures, methods,
and events that provide access to objects in a drawing file
or the AutoCAD application. Each DLL file defines
different namespaces which are used to organize the
components of the libraries based on functionality.
6.2 APPLICATIONS
Our project contains a set of buttons and we have a simple
Figure 5: Show high level OO Diagram description to its work.
representation the System
A.CAD Design
6-IMPEMENTATION:
Is an application to manipulate architecture models in a CAD-
6.1 LIBRARIES like environment , by import the “object” to move it by our
fingers in any surface through gesture fingers movements.
Our system uses some libraries that implements different
functionalities we needed to make our code flexible and
behave in such desired manner.
The main ones used are:
A. TouchlessLib :
Touchless is an SDK that allows users to create and
experience multi-touch applications. Touchless started as
Mike Wasserman’s college project at Columbia University.
The main idea: to offer users a new and cheap way of
experiencing multi-touch capabilities, without the need of
expensive hardware or software. All the user needs is a
camera, which will track colored markers defined by the user.
B. DirectShowNET Library :
The purpose of this library is to allow access to Microsoft's
DirectShow functionality from within .NET applications. This
library supports both Visual Basic .NET and C#, and
theoretically, should work with any .NET language. Figure 6:Snapshot show CAD-Design Application works
C. Microsoft.DirectX.Direct3D:
Microsoft Direct3D is a low-level graphics application
programming interface (API) that enables you to manipulate

7. B. Barcode Reader:
Is a readable application that represent the data of product,
each character represent by a pattern narrow and wide , when
person take picture using markers or upload it from pc then
convert image to readable data that appears through message.
Figure 9:Thermal image example ( Pain in the jaw.
Thermogram findings helped confirm diagnosis of TMJ
(Temporo-mandibular Joint Disorder) and referral to the
appropriate specialist for treatment.)
7. PREVIOUS WORK:
Gesture Recognition Applications was become one of the
Figure 7:Simple run to show scanning sample barcode most application commonly used by humans and they adapted
it as daily interaction for its availability and usability , one of
C. Thermal Images
these Application was the “ Sixth-sense” Technology
Is an Application to capturing & manipulating thermal images
at hospitals in a portable manner where they get immediately
stored on the phone memory .
Figure 8:Show how Doctors could use it
Figure 10:Show Sixth-Sense Technology idea

8. that was developed by “Pranav Mistry”, a PhD student in the http://news.cnet.com/8301-17938_105-10159601-
Fluid Interfaces Group at the MIT Media Lab which include 1.html
seven graduate students were thinking about how a person [7] New York Times - At TED, Virtual Worlds Collide
could be more integrated and access information without With Reality , Available
having to do something like take out a phone. They applied it http://pogue.blogs.nytimes.com/2009/02/11/at-
through set of applications ( draw , map , photo , Gallery , ted-virtual-worlds-collide-with-reality/
Globe , Mail ,weather , effects, news paper , book , clock , [8] TED Talks - Pranav Mistry: The thrilling potential
menu and gesture ) using “Sixth Sense Technology” that of SixthSense technology , Available
combine from camera , micro- projector , mirror and smart http://www.ted.com/talks/pranav_mistry_the_thrilli
phone , The camera recognizes objects around a person ng_potential_of_sixthsense_technology.html
instantly, with the micro-projector overlaying the information [9] sixth-sense-technology , Available
on any surface, including the object itself or hand. Also can http://www.blendernation.com/2009/11/29/sixth-
access or manipulate the information using fingers. sense-technology/
In our project we build in the exits features through some of [10] Theviewspaper.net , Available
application like barcode scanning, Thermal Image and http://theviewspaper.net/sixth-sense-technology-
architecture. will-revolutionize-the-world/
[11] Gizmodo.com , Available
8. REFERENCES http://gizmodo.com/5167790/sixth-sense-
technology-
.
[1] "Telepointer: Hands-Free Completely Self
Contained Wearable Visual Augmented Reality
without Headwear and without any Infrastructural
Reliance", IEEE International Symposium on
Wearable Computing (ISWC00), pp. 177, 2000, Los
Alamitos, CA, USA
[2] "Cyborg: Digital Destiny and Human Possibility in
the Age of the Wearable Computer", Steve Mann with
Hal Niedzviecki, ISBN 0385658257 (Hardcover),
Random House Inc, 304 pages, 2001.
[3] An Anatomy of the New Bionic Senses
[Hardcover], by James Geary, 2002, 214pp
[4] Intelligent Image Processing, John Wiley and Sons,
384pp, 02001NOV02, Available
http://en.wikipedia.org/wiki/Special:BookSources/0
471406376
[5] sixthsense . Pranav Mistry, Available
http://www.pranavmistry.com/projects/sixthsense/
[6] CNet News: MIT's 6th Sense device could trump
Apple's multitouch , Available

9. Appendix Section
Figure 11 : Show us the barcode reader demo
Figure 12:Show us the Take Photo Demo – take thermal or original picture

10. Figure 13 : Show Gesture Class code
Figure 14: Show the System Interface