<Capstone Involving Multi-Touch Technology>Senior Capstone ProjectDaniel A. TaylorApril 2013
Table of Contents1. INTRODUCTION1.1. THE TOUCH SCREEN1.1.1. TEST THEORY1.1.2. OTHER TECHNOLOGIES1.1.3. BACKGROUND2. MULTI TOUCH2.1. FTIR3. DESIGN3.1. PROJECT STATEMENT3.2. HARDWARE3.3. SOFTWARE3.4. WORKING3.5.ADVANTAGES3.6.APPLICATIONS4. CONCLUSION5. APPENDICES5.5.APPENDIX I6. 6. REFERENCES
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 1 -ABSTRACTMulti-touch technology is an advanced human-computer interaction technique that recognizesmultiple touch points and also includes the hardware devices that implement it, which allowusers to compute without conventional input devices. Multi-touch consists of a touch screen(screen, table, wall, etc.) or touchpad, as well as software that recognize multiplesimultaneous touch points, as opposed to the standard touchscreen which recognizes only onetouch point at a time. My Multi touch table using Frustrated Total Internal Reflection is asimple, inexpensive, and scalable technique for enabling high-resolution multi- touch sensingon rear-projected interactive surface. Different applications for multi-touch interfaces bothexist and are being developed.The use of multi-touch technology is expected to rapidlybecome common place. Throughout the developmental process, I have exploring both thehardware and software side of Multi-touch computing. On the hardware end, variousmaterials (camera, projection screen, etc.) were needed to construct the table. On the softwareend, both low level finger tracking or hand jester and high level software developmentwereused in the Open-Source domain as a part of the Multi-touch device technology.
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 2 -1. INTRODUCTION AN BACKGROUND1.1 THE TOUCH SCREENA touch screen (Figure 1) is a display which can detect the presence and location of a touch withinthe display area. The term generally refers to touch or contact to the display of the device by afinger or hand. Touch screens can also sense other passive objects, such as a stylus. However, ifthe object sensed is active, as with a light pen, the term touch screen is generally not applicable.The thumb rule is: if you can interact with the display using your finger, it is likely a touch screen- even if you are using a stylus or some other object. Up until recently, most touch screens couldonly sense one point of contact at a time, and few have had the capability to sense how hard one istouching. This is starting to change with the emergence of multi-touch technology - a technologythat was first seen in the early 1980s, but which is now appearing in commercially availablesystems. The touch screen has two main attributes. First, it enables you to interact with what isdisplayed directly on the screen, where it is displayed, rather than indirectly with a mouse or atouchpad. Secondly, it lets one do so without requiring any intermediate device, again, such as astylus that needs to be held in the hand. Such displays can be attached to computers or, asterminals, to networks. They also play a prominent role in the design of digital appliances such asthe personal digital assistant, satellite navigation devices and mobile phones.Figure 11.1.1 TEST THEORYThere are several different ways to make a multi-touch surface, but I will focus only on onemethod: the FTIR screen. An FTIR (short for Frustrated Total Internal Reflection) setup involvesthree vital components: a sheet of transparent acrylic, a chain of infrared LEDs, and a camera withan IR filter. The LEDs are arranged around the outside of the sheet of acrylic so that they shinedirectly into the thin side surfaces.
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 3 -Once the IR light is inside the acrylic, it strikes the top and bottom surfaces of the acrylic at anear-parallel angle, and is subject to the effect known as total internal reflection. This causes it tobe wholly maintained in the acrylic. This is a little tough to describe in words, so I made a simplediagram (Figure 2):Figure 2Infrared ExplainedAn IR touchscreen panel employs one of two very different methods. One method uses thermalinduced changes of the surface resistance. This method is sometimes slow and requires warmhands. Another method is an array of vertical and horizontal IR sensors that detect the interruptionof a modulated light beam near the surface of the screen.Optical imagingA relatively-modern development in touchscreen technology, two or more image sensors areplaced around the edges (mostly the corners) of the screen. Infrared backlights are placed in thecameras field of view on the other sides of the screen. A touch shows up as a shadow and eachpair of cameras can then be triangulated to locate the touch. This technology is growing inpopularity, due to its scalability, versatility, and affordability, especially for larger units.1.1.2 OTHER TECHNOLOGIESResistiveA resistive touch screen panel is composed of several layers. The most important are two thinmetallic electrically conductive and resistive layers separated by thin space. When some object
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 4 -touches this kind of touch panel, the layers are connected at certain point; the panel thenelectrically acts similar to two voltage dividers with connected outputs. This causes a change inthe electrical current which is registered as a touch event and sent to the controller for processing.Surface acoustic wave SAW technology uses ultrasonic waves that pass over the touchscreenpanel. When the panel is touched, a portion of the wave is absorbed. This change in the ultrasonicwaves registers the position of the touch event and sends this information to the controller forprocessing. Surface wave touch screen panels can be damaged by outside elements. Contaminantson the surface can also interfere with the functionality of the touch screen.CapacitiveA capacitive touch screen panel is coated with a material, typically indium tin oxide that conductsa continuous electrical current across the sensor. The sensor therefore exhibits a preciselycontrolled field of stored electrons in both the horizontal and vertical axes - it achievescapacitance. The human body is also an electrical device which has stored electrons and thereforealso exhibits capacitance. When the sensors normal capacitance field (its reference state) isaltered by another capacitance field, i.e., someones finger, electronic circuits located at eachcorner of the panel measure the resultant distortion in the sine wave characteristics of thereference field and send the information about the event to the controller for mathematicalprocessing. Capacitive sensors can either be touched with a bare finger or with a conductivedevice being held by a bare hand. Capacitive touchscreens are not affected by outside elementsand have high clarity. The Apple iPhone is an example of a product that uses capacitancetouchscreen technology.Dispersive Signal TechnologyIntroduced in 2002, this system uses sensors to detect the mechanical energy in the glass thatoccur due to a touch. Complex algorithms then interpret this information and provide the actuallocation of the touch. The technology claims to be unaffected by dust and other outside elements,including scratches. Since there is no need for additional elements on screen, it also claims toprovide excellent optical clarity. Also, since mechanical vibrations are used to detect a touchevent, any object can be used to generate these events, including fingers and stylus. A downside isthat after the initial touch the system cannot detect a motionless finger.1.1.3BACKGROUNDMulti-touch technology dates back to 1982, when the University of Toronto developed the firstfinger pressure multi-touch display. The same year, Bell Labs at Murray Hill published what isbelieved to be the first paper discussing touch-screen based interfaces.Bell LabsIn 1984 Bell Labs engineered a multi-touch screen that could manipulate images with more thanone hand. The group at the University of Toronto stopped working on hardware and moved on tosoftware and interfaces, expecting that they would have access to the Bell Labs work.A breakthrough occurred in 1991, when Pierre Wellner published a paper on his multi-touchDigital Desk, which supported multi-finger and pinching motions.
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 5 -Finger worksIn 1998, FW, a Newark-based company run by University of Delaware academics John Elias andWayne Westerman, produced a line of multi-touch products including the iGesture Pad and theTouchStream keyboard. Westerman published a dissertation in 1999 on the subject. In 2005, afteryears of maintaining a niche line of keyboards and touch pads, Finger works was acquired byApple Computer.Perceptive Pixel and the rise of FTIRPP is a company founded by New York University consulting research scientist Jefferson Y. Hanthat creates wall displays and tables that can accommodate up to 20 fingers. Han introduced theFTIR technique to multi touch screens. The displays use light emitting diodes along with infraredlight to determine the point of contact. Han envisions large collaborative spaces that will allowmultiple users to work and interact.Apple iPhone, iPod touch, MacBook Air, and MacBook Pro In 2005, Apple acquiredFingerworks. In 2007 they introduced the iPhone, marking the first time multi-touch technologywas used on a phone. The iPhone includes such components as a web browser, music player,video player, and a cell phone without the use of a hard keypad or stylus. Following the release ofthe iPhone, Apple also expanded its use of multi-touch computing with the new iPod Touch, aswell as the new MacBook Air. Multi-touch was later added to the 2008 MacBook Pro line in theform of a track pad.2. MULTI TOUCHMulti-touch is a human-computer interaction technique and the hardware devices that implementit, which allow users to compute without conventional input devices e.g., mouse, keyboard. Multi-touch consists of a touch screen like screen, table, wall or touchpad, as well as software thatrecognizes multiple simultaneous touch points, as opposed to the standard touchscreen i.e.computer touchpad, ATM, which recognizes only one touch point. This effect is achieved througha variety of means, including but not limited to: heat, finger pressure, high capture rate cameras,infrared light, optic capture, tuned electromagnetic induction and shadow capture.2.1FRUSTRATED TOTAL INTERNAL REFLECTIONTotal internal reflection is an optical phenomenon that occurs when a ray of light strikes amedium boundary at an angle larger than the critical angle with respect to the normal surface. Ifthe refractive index is lower on the other side of the boundary no light can pass through, soeffectively all of the light is reflected. The critical angle is the angle of incidence above which thetotal internal reflection occurs. When light crosses a boundary between materials with differentrefractive indices, the light beam will be partially refracted at the boundary surface, and partially
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 6 -reflected. If the ray is closer to being parallel to the boundarythen the light will stop crossing theboundary altogether and be totally reflected back internally. This can only occur where lighttravels from a medium with a higher refractive index to one with a lower refractive index. Forexample, it will occur when passing from glass to air, but not when passing from air to glass.The critical angle is the angle of incidence above which total internal reflection occurs. The angleof incidence is measured with respect to the normal at the refractive boundary. The critical angleis given by: where nis the refractive index of the less dense medium, and nis the refractive index of the denser mediumAn important side effect of total internal reflection is the propagation of an evanescent waveacross the boundary surface. Essentially, even though the entire incident wave is reflected backinto the originating medium, there is some penetration into the second medium at the boundary.Additionally, the evanescent wave appears to travel along the boundary between the twomaterials. This wave can lead to a phenomenon known as frustrated total internal reflection.Under "ordinary conditions" it is true that the creation of an evanescent wave does not affect theconservation of energy, i.e. the evanescent wave transmits zero net energy. However, if a thirdmedium with a higher refractive index than the second medium is placed within less than severalwavelengths distance from the interface between the first medium and the second medium, theevanescent wave will be different from the one under "ordinary conditions" and it will pass energyacross the second into the third medium.3. DESIGN3.1 PROJECT STATEMENTMy Multi touch is designed using the FTIR technology. FTIR describes the internal reflection oflight. It is force-sensitive, and provides unprecedented resolution and scalability. The Touchscreen had to be large enough to accommodate both hands and multiple users. This phenomenonis also used in fingerprint and robot sensors. My mission plan was to design a Multi-touchenvironment that allows an individual to work in a computer aid environment. A display tabledesigned for Multi-touch interaction was thepoint of Capstone collaboration. Users can directlymanipulate what is projected on screen and aframework within Windows 7 was programmed tomake use of the table’s Multi-Input (Touch) ability.
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 7 -3.2HARDWAREThe basic design has a hardware and software part. Hardware requires basically IR LEDs, acrylic,camera, projector and a computer. Infra-red light has a higher wavelength than that of visiblelight. Therefore it has more intensity and will be felt everywhere inside the denser medium.Acrylic is the denser medium .It is a synthetic fiber having half the density of glass. An infra-redcamera or a webcam is used to catch IR light. An IR block filter of the camera has to be removedsince it blocks IR light. It also consists of a projector and a computer. The object has to beprojected on top of the acrylic from a computer. Thus acrylic is a virtual display. The IR LEDsabout 10-20 are arranged on both sides of the acrylic along its edges.The camera below the acrylicand the projector located under the acrylic.3.3 SOFTWAREComputer applications are necessary to communicate between a multi touch display and thecomputer. These applications can be developed within several languages / programmingenvironments. For example: Processing (P5), Flash, C, C++, Java and others. Because of theusability of certain APIs and the relatively simple visualization possibilities, P5 or Flash 11, incombination with Action script 4.0 will make a great combination.3.3.1 Calibration
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 8 -I have researched multiple open source calibration programs. The central software that I havechosen to power my table is Touchlib, an open source library which takes the visual data receivedby the camera and parses it into touch events, which can be used by other programs to providemulti-touch control. Some programs implement this library directly, allowing for standalonemulti-touch apps, while others, such as those written in Action Script, require an extra softwarelayer to allow the program to receive touch input. In this section, we’ll explain how we got bothup and running.3.4 WORKINGFTIR describes the internal reflection of light, inside a certain material. In our case, it will beinfrared light, that internally reflects inside is a piece of acrylic, also known as Plexiglas. Thisway, infrared light is beamed inside the acrylic and reflects internally. In a simple way, you cansay that, IR-light bounces inside the acrylic, from one side to another. As soon as a finger touchesthe acrylic surface, the internal reflection of the IR-light, is interrupted. The infrared light scatterson the finger tips. Infrared light is invisible to the human eye, but by placing an infrared camerabehind the acrylic your fingertips will be visible on the infrared camera. The images that aregenerated by the camera contain white blobs (caused by the fingertips). These blobs will beanalyzed by software. Every blob corresponds to certain coordinates. Software can by analyzingthese coordinates perform certain tasks, for example move, resize or rotate objects. Multiplepoints are obtained on the camera .Each point is a pixel position. Either a single pixel or a groupof pixels. Each point locations are identified and all operations are performed. Suppose if I wantedto zoom a picture using two fingers and move it in or out to perform zoom in and zoom outrespectively. Two coordinates will be located on the camera.The difference is found which is putas the offset and it is either added or subtracted with the locations to zoom out and zoom inrespectively.3.5 ADVANTAGES Multi touch based on FTIR is a simple and inexpensive technique .It constructs a multitouch display with the available and less costly materials. Scalable technique that enables high-resolution graphics .It provides support to anyresolution possible as all multiple points could be generated on a camera
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 9 - It acquires true touch image information at high spatial and temporal resolutions. Theactual finger print of the touch is obtained.This could be used to determine the forcesensitivity on displays; either too hard or soft touches can be analyzed. It is scalable to large installations. Any kind of applications can be made to suit multitouch using FTIR. Allows us to create sophisticated multi-point widgets for applications Larger shared-display systems i.e. it is well suited for use with rear- projection like wallscreens, table tops .All this lead to high resolution graphics.3.6 APPLICATIONA myriad of different application for multi-touch interfaces both exists and is being developed.Some uses are individualistic e.g., iPhone, iPod touch, MacBook Pro, MacBook Air, HTCDiamond. However, multi-touch technology is mainly used to incorporate collaboration into thecomputing experience.A multi touch display can be used in Personal Computers, Laptops, Tabletops, Graphics Tablets. It supports both LCD and CRT monitors. Telephones, Watches, PDAs, Mobile phones. Advanced multi touch Gaming with high graphics support Governmental, office and business purposes An enhanced multimedia experience including audio, video and photo sharing Enhanced dining experienceApplications for a multi touch display are never ending.4. CONCLUSIONTouch screens are the interface for the 21st century. Touch screens address the conflictingdemands for smaller portable electronics with larger displays, by eliminating traditional buttonswithout sacrificing screen size. The recent release of the iPhone has created a buzz around touchscreen interfaces and its multi-touch acrobatics have caught the eye of many industry leaders.There are many ways to make a multi-touch screen. Some of the early designs measured thechange in electrical resistance or capacitance on a surface when fingers touched it. But thesedevices have limited resolution, are relatively complex, and dont easily and inexpensively scaleup to large dimensions. Multi-touch technologies have a long history. This technique using FTIRis simple and easy to implement. It provides any resolution displays supported with high graphics.The applications being both made and proposed are plenty in number. A drawback of theapproach is that, being camera-based, it requires a significant amount of space behind theinteraction surface, though we primarily expect application scenarios where rear-projection wouldhave been employed anyway (e.g. interactive walls, tables). Also, as an optical system, it remainssusceptible to harsh lighting environments.
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 10 -The use of multi-touch technology is expected to rapidly become common place. For example,touch screen telephones are expected to increase from 200,000 shipped in 2006, to 21 million in2012. Developers of the technology have suggested a variety of ways that multi-touch can be usedincluding: Enhanced dining experience Concierge service Governmental use Concept mapping Collaboration and instruction on Interactive Whiteboards
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 11 -APPENDICESAction script 4 (Flash) Adobe Flash, Wikipedia entry. http://en.wikipedia.org/wiki/Adobe_Flash “Flash for surface computing” by Manvesh Vyas http://www.adobe.com/devnet/edu/articles/manvesh-vyas.html flosc: Flash Open Sound Control http://www.benchun.net/flosc/ Migrating from ActionScript 2.0 to ActionScript 4.0: Key concepts and changes by DanCarr http://www.adobe.com/devnet/flash/articles/first_as3_ap- plication.htmlC++ C++ Programming Language. http://isocpp.org/Multi-Touch Han, Jerfferson Y. “Low Cost Multi-Touch Sensing through Frustrated To- tal InternalReflection.” Symposium on User Interface Software and Technology: Proceedings of the 18thannual ACM symposium on User interface software and technology. Seattle,WA, USA, 2005.115-118.
Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor- 12 -6. REFERENCES Low-Cost Multi-Touch Sensing through FTIR by Jefferson Y. Han Buxton, W., Hill, R., and Rowley, P. 1985. Issues and Techniques in Buxton, Bill. 2008. Multi-Touch Systems that I Have Known and Loved. http://www.billbuxton.com/multitouchOverview.html How to build a multi touch by Harry Vaan Der Opensource,MultitouchDisplay,http://www.technologyreview.com/Infotec