Published on

Gesture recognition & manipulation in phone applications for projected surfaces (GRAS) as an (Graduation Project)

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Gesture recognition & manipulation in phone applications for projected surfaces (GRAS) Alaa F.Drabseh , Esraa A.Alsyis ,Wafaa M.Towaiq ,and Mostafa Z.Ali Computer Information Department , Jordan University of Science & Technology Irbid ,Jordan E-mails : { afdarabseh08, eaalsayis08 , wmtwaiq08} Abstract--- This project focuses on facilitating human GRAS allows humans to interact naturally without anyinteraction with the digital word by giving a direct link mechanical device through pointing fingers on the projectedbetween the digital world and our physical interaction screen and then the cursors move accordingly, where we usedwith the real world. Gesture recognition & manipulation four fingers and simulate four cursors for each one .in phone applications for projected surfaces (GRAS) is a Experiments in the use of cross-correlation as means ofwearable gestural interface that arguments the physical tracking pointing devices for a digital desk, that describing theworld around us with digital information and lets us use potential for applying real time computer vision to usersnatural hand gestures to interact with that information. interaction.The system is been modified and geared towardsapplications like: capturing & manipulating thermalimages at hospitals in a portable manner where they getimmediately stored on the phone memory. Anotherapplication is customizing architectural CAD designs viahand gesture recognition (This is intended for enhancingthe way students perceive instructions on the basics intheir introductory design courses). The latter applicationwill enhance the way through which students perceive themodel and interactively generate and amend sequence ofmodules for a project in an immersive environment.Keywords— GRAS , Gesture , fiducials , CAD ,Thermalimages1. INTRODUCTION People evolved our millions of years to sense the worldaround them, through which they perceive the informationaround them using their five natural senses. Although their Figure 1: Show the concept of GRASmineralization of the computing devices allows us to carrycomputing devices in our project, keeping us connected to ourdigital world, there is no link between our devices and our Vision-based hand tracking is an important problem in theinteractions with the physical world. This involves a planning field of human-phones interaction, since hand motions andphase, design phase and a phase of formation of all the gestures could potentially be used to interact with phones innecessary elements and then connects it to each other. The more natural ways . A number of solutions have beenfinal product development process in GRAS is normally proposed in our project, but the problem is still far from beingstructured as a set of applications grouped in one device in solved since the hand exhibits.which each application works independently from otherapplications.
  2. 2. The project prototype is comprised of pocket projector (laser multi- touch and multi-user interaction inherited from phoneor L.E.D projector), a mirror, colored markers , a mobile features, but on a more flexible projected surface.computing device, a camera with fine resolution and aThermal Cam for Thermal images. The camera, mirror andprojector are connected wirelessly to Bluetooth Smartphone 2. METHODOLOGYdevice that can easily fit into the users pocket. The Phoneapplication then processes the data that is collected by the This project is focuses on facilitating humans interactioncapturing device and does the analysis. The projector projects with the digital word by gives a direct link between the digitalvisual information enabling surfaces, walls and physical world and our physical interaction with the real world.objects around us to be used as interfaces; while the camera Gesture recognition & manipulation in phone applications forrecognizes and tracks users hand gestures and physical projected surfaces (GRAS) is a wearable gestural interfaceobjects using computer-vision based techniques. that arguments the physical world around us with digital information and lets us use natural hand gestures to interact Application equipments contain from colored markers on with that information.the users fingers are recognized with colored tapes on themby the camera. The camera captures the user in view andtracks his hand gestures, then sends the data to the smartphone which processes the video data, to identify the object.Where there will be a tiny projector (with a mirror) thatdisplays data sent from the smart phone on any surface inview like wall, or person. Figure 3: How we Can using your Fingers as Cursors 2.1 PROJECT IDENTIFICATION & SELECTION A. Mission statement • To build a wearable gesture interface that allows users to improve the manipulate of objects around them from GRAS in an efficient way ,as a preparation step for controlling the CAD objects to streaming design processes through the simulate the gesture recognition of their fingers & manipulating thermal images at hospitals in a portable Figure 2: Fingers Movement Examples manner . The software processes the video stream data captured by B. Programming Language and Software usedthe camera and tracks the locations of the colored markers atthe tip of the user’s fingers using simple computer-vision  For Implementation purposes we will use thetechniques. The movements and arrangements of these following software :fiducials are interpreted into gestures that act as interaction  Visual Studio .Net 2010instructions for the projected application interfaces. Themaximum number of tracked fingers is only constrained by  Direct X SDKthe number of unique fiducials, thus GRAS also supports  C++/C# as a programming Language
  3. 3. 2.2 INITIATION AND PLANNING  Intended for every person who interested is inA. Statement of work (SOW) knowing the product information before purchasing it.  The goal of our project is to develop an applications for Smart phones that helps use and control of objects around us in an easy way , as a manipulate B. The project will introduce a mobile technology: architecture models in a CAD-like environment , by including the following import the “object” to move it by our fingers in any surface through gesture fingers movements , and to  Smart Phone develop an application to hospitals sector that allows  Windows platform written in C++ and C# them to manipulate thermal images by use the  Projector fingers as a calibrations to zoom in-out the borders to  Camera and Thermal Camera take a photo and then process it as an thermal image ,  Flat surface the last app is Barcode reader by read any supported  Markers ( 4-differenet Colors ) product barcode format and get information regards . • The first part of our project enables the user to have 2.4 BENEFITS the look and feel which is the process of the walkthrough the building to see constriction  The phone user for project apps will do his/her internally and to know the characteristics of the activates in a highly speed and fixable manner. building and check its feasibility by themselves any  Provide Portable, scalable and wearable device. time anywhere with an available surface. when we  Use these Apps in one small wearable device we will could see and move the object by our self before it is reduce the cost and errors occur. constricted it will increase the confidence of such  Convert any surface into an interactive touch screen. designs through easiness multiple scenarios that  The users of the CAD-App can draw on any surface could not be easy or possible on paper . zoom-in, zoom-out elements by pointing via fingers.  Speed to obtain thermal images in cases  The second Part allows medical doctors to quickly of emergency to thus speed in the provision see any internal problems and give a suitable cure of medical care. even if they outside the hospital .when we could see  Interact with real time data and information regard to these images quickly after the accident we will Products. reduce the advance symptoms.  The third part enhances the purchasing process for any product to eliminate the mediators (self- catering ) , by reading the information regarding any product and making the decision to buy it or not . 2.5 COST A. The major cost is linked with following equipments and devises :2.3 SCOPEA. Target Group Subject  Smartphone (Mobile computing device )  Projector  Intended to designers who need a friendly  Camera (phone Cam) application to design the building with CAD shapes  Thermal Camera objects even if they are not familiar with CAD  Mirror software .  Intended for Hospital sectors who want to gain a competitive advantage through Speed in the provision of medical care for their patients.
  4. 4. 2.6 RISK Risk Risk Likel Mitigation Level ihoo Strategy A .Risk that might be faced in this project with L/M/H d of suggested strategies of avoiding this risk is Even shown in table 1 t Physical M: Team Likel Use of Intranet Location of is y project Team dispersed website, prevents among comprehensiveRisk Risk Likel Mitigation effective several Communicatio Level ihoo Strategy L/M/H d of managemen sites ns Plan Even t t Table 1: A sample line graph using colors which contrastProject well both on screen and on a black-and-white hardcopySize Estimated H: Over 7 Cert Created 2.7 FEASIBILITY STUDY: Project monthes ainty comprehensiv A. Operational Feasibility: Schedule e project timeline with  Our proposed system serves a medical, frequent architectural and market purposes. It can be baseline applied on any smart-phone such Android. reviewsProject B. Schedule Feasibility:Definition Narrow M: Likel Assigned  As an estimation of the time needed to Knowledge Knowledge y Project complete all features of the system, it needs Level of able of Manager(s) to around 6 months with all equipments needed Users user area assess global are available through the work time. only implications Cost L: likel C. Economic Feasibility: Estimates Thoroughl y Unrealistic y  Tangible benefits: predicted by team  Increase flexibility, and speed of activity. members.  Opening new markets and increasing salesProject opportunity.Staffing  Using natural fingers gestures to work with the Project M: Som Continuous applications. Team Distributed ewh review of Availability team at project makes likel momentum by  Intangible benefits: availability y all levels.  Convert any surface into an interactive touch questionab Consultant to screen. le identify any impacts  Users can interact with real time data and caused by information. unavailability.  Support multi-user interaction at the same If necessary, time. increase committmment  Tangible costs: by participants  Hardware cost like LED mini-projector and to full time status thermal Camera.
  5. 5.  Intangible costs: 4- ANALYSIS:  Loss of customer goodwill. The process of gathering and analysing the system requirements was done by interested Archie people and understanding the major problems face them and how they do  One-Time Cost: their work to understand the basic aspects, and we met health  New Hardware and software purchase. care staff to recognize the importance of thermal images in their job , and how our application will help them to make  System development. their tasks be more flexible and easer . Also, we made a survey about how much they are interested to have barcode scan application in their phones.D. Legal Feasibility:  Development the project does not conflict with all A. System Requirement: applicable federal statuses.  The system should build the knowledge of3- SCHEDULE PLANNING: transaction in our project.1. Tasks  The system should provide the users with more flexibility in terms of install, uninstall, update and use it easy.  The system should be able to be installed on different smart phones. B. User Requirement:  User should be able to know how to use our application phone.  User shall be familiar with the building features.  Users should be able to install, update and uninstall softwares . C. Function Requirement:  Simulating dynamic operations.  Coordinating detailed design  Scheduling constriction  Demonstrating Technical compliance Table 2: Project Tasks with estmate time to complet it  Design Review2.Gantt Chart D. Non Function Requirement  The system shall be clear enough to be understood by the user.  Usability .  The system should be compatible with different platforms used. Figure 4:Show Order of Tasks to get the project
  6. 6. 5 -DESIGN visual models of 3-D objects and take advantage of hardware  5.1 Object Oriented Diagram acceleration . D. System.Drawing: The System.Drawing namespace provides access to GDI+ basic graphics functionality. E. AutoCAD .NET API: The AutoCAD .NET API is made up of different DLL files that contain a wide range of classes, structures, methods, and events that provide access to objects in a drawing file or the AutoCAD application. Each DLL file defines different namespaces which are used to organize the components of the libraries based on functionality. 6.2 APPLICATIONS Our project contains a set of buttons and we have a simple Figure 5: Show high level OO Diagram description to its work. representation the System A.CAD Design6-IMPEMENTATION: Is an application to manipulate architecture models in a CAD-6.1 LIBRARIES like environment , by import the “object” to move it by our fingers in any surface through gesture fingers movements.Our system uses some libraries that implements differentfunctionalities we needed to make our code flexible andbehave in such desired manner.The main ones used are:A. TouchlessLib :Touchless is an SDK that allows users to create andexperience multi-touch applications. Touchless started asMike Wasserman’s college project at Columbia University.The main idea: to offer users a new and cheap way ofexperiencing multi-touch capabilities, without the need ofexpensive hardware or software. All the user needs is acamera, which will track colored markers defined by the user.B. DirectShowNET Library :The purpose of this library is to allow access to MicrosoftsDirectShow functionality from within .NET applications. Thislibrary supports both Visual Basic .NET and C#, andtheoretically, should work with any .NET language. Figure 6:Snapshot show CAD-Design Application works C. Microsoft.DirectX.Direct3D:Microsoft Direct3D is a low-level graphics applicationprogramming interface (API) that enables you to manipulate
  7. 7. B. Barcode Reader: Is a readable application that represent the data of product,each character represent by a pattern narrow and wide , whenperson take picture using markers or upload it from pc thenconvert image to readable data that appears through message. Figure 9:Thermal image example ( Pain in the jaw. Thermogram findings helped confirm diagnosis of TMJ (Temporo-mandibular Joint Disorder) and referral to the appropriate specialist for treatment.) 7. PREVIOUS WORK: Gesture Recognition Applications was become one of the Figure 7:Simple run to show scanning sample barcode most application commonly used by humans and they adapted it as daily interaction for its availability and usability , one ofC. Thermal Images these Application was the “ Sixth-sense” TechnologyIs an Application to capturing & manipulating thermal imagesat hospitals in a portable manner where they get immediatelystored on the phone memory . Figure 8:Show how Doctors could use it Figure 10:Show Sixth-Sense Technology idea
  8. 8. that was developed by “Pranav Mistry”, a PhD student in the Interfaces Group at the MIT Media Lab which include 1.htmlseven graduate students were thinking about how a person [7] New York Times - At TED, Virtual Worlds Collidecould be more integrated and access information without With Reality , Availablehaving to do something like take out a phone. They applied it set of applications ( draw , map , photo , Gallery , ted-virtual-worlds-collide-with-reality/Globe , Mail ,weather , effects, news paper , book , clock , [8] TED Talks - Pranav Mistry: The thrilling potentialmenu and gesture ) using “Sixth Sense Technology” that of SixthSense technology , Availablecombine from camera , micro- projector , mirror and smart , The camera recognizes objects around a person ng_potential_of_sixthsense_technology.htmlinstantly, with the micro-projector overlaying the information [9] sixth-sense-technology , Availableon any surface, including the object itself or hand. Also can or manipulate the information using fingers. sense-technology/In our project we build in the exits features through some of [10] , Availableapplication like barcode scanning, Thermal Image and will-revolutionize-the-world/ [11] , Available8. REFERENCES technology- . [1] "Telepointer: Hands-Free Completely Self Contained Wearable Visual Augmented Reality without Headwear and without any Infrastructural Reliance", IEEE International Symposium on Wearable Computing (ISWC00), pp. 177, 2000, Los Alamitos, CA, USA [2] "Cyborg: Digital Destiny and Human Possibility in the Age of the Wearable Computer", Steve Mann with Hal Niedzviecki, ISBN 0385658257 (Hardcover), Random House Inc, 304 pages, 2001. [3] An Anatomy of the New Bionic Senses [Hardcover], by James Geary, 2002, 214pp [4] Intelligent Image Processing, John Wiley and Sons, 384pp, 02001NOV02, Available 471406376 [5] sixthsense . Pranav Mistry, Available [6] CNet News: MITs 6th Sense device could trump Apples multitouch , Available
  9. 9. Appendix Section Figure 11 : Show us the barcode reader demoFigure 12:Show us the Take Photo Demo – take thermal or original picture
  10. 10. Figure 13 : Show Gesture Class codeFigure 14: Show the System Interface