Gestural Interaction With In-Vehicle Audio and Climate Controls

as the right side of the driver could be shared with a front pas-   ledge or gesture stereotypes were then applied to cont...
Two problems with gestures were discovered in this ex-        last three functions, defrost, air from dashboard and air fr...
commands, having been accustomed to controlling devices             neath and from air from underneath to defrost is a cir...
design instead of the two-spoke design was adopted. This ini-       that gestural control interfaces that are already ubiq...
Upcoming SlideShare
Loading in …5

Gestural Interaction With In-vehicle Audio and Climate Control


Published on

1 Like
  • Why not? It's my pleasure.
    Are you sure you want to  Yes  No
    Your message goes here
  • Perfect, I'm a 'near-to-graduate' student at Rome (La Sapienza).
    I'm developing a special taxonomy of the gestures in IT with attention to some field IT related and a new language to help the developer. Maybe my work will be published soon.
    So can I cite your work?
    Are you sure you want to  Yes  No
    Your message goes here
  • Yes, it was published at 2010 Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting
    Are you sure you want to  Yes  No
    Your message goes here
  • Do you publish this paper?
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Gestural Interaction With In-vehicle Audio and Climate Control

  1. 1. Gestural Interaction With In-Vehicle Audio and Climate Controls Chongyoon Chung1 and Esa Rantanen Rochester Institute of Technology, Rochester, NY 1 Now at Samsung Electronics, Seoul, South Korea Among the most distractive in-vehicle interactions are audio and climate controls. If these interactions were as easy and spontaneous as natural language, driving could be much safer. Through this research it was found that drivers preferred gestural language to voice language when the control was simple and re- petitive; subsequently, gestural interactions with secondary in-vehicle tasks were investigated. Following the principle of “eyes on the road and hands on the wheel”, a steering wheel design with two touch pads on the wheel to recognize gestures was conceived. The physical design of the steering wheel incorporated good ergonomics and anthropometric data, while gesture stereotypes assigned to a number of in-vehicle controls were determined empirically by two experiments. The new steering wheel design does not have any buttons, which may contribute to driver distraction, yet it incorporates 19 functions through natural thumb-gestures. This compares favorably with most current steering wheel designs, which have more than 11 buttons and 13 functions on the average. INTRODUCTION functions on their steering wheels. These values are undenia- bly high. Modern automobiles have a myriad of manual controls To reduce driver distraction due to operation of in-vehicle for increasingly complex auxiliary systems that have the po- systems, their control interfaces should be as intuitive and tential of distracting drivers from their primary task of driving. easy to use as possible. The most natural modes of communi- Car audio, climate control, and navigation systems have stea- cation and control are language and gestures. Speech recogni- dily increased in sophistication, and CD players and mp3 tion, such as Microsoft’s Sync system in new Ford automo- player interfaces are now common even in least expensive biles, has recently been introduced to consumers. Speech rec- models. Manual control of all these systems requires drivers to ognition has limitations, however, as it is vulnerable to noises, take their eyes off the road and traffic environment, which has dialects, and individual voice differences. Moreover, current obvious safety implications. Pickering (2005) found that an technology recognizes only specific words preprogrammed in- average glance time to control a radio was 1.2 seconds; in that to the system. time a car travels over 50 ft at 30 mph. Summala, Lamble, and Laakso (1998) showed that ambient vision was not sufficient Gestural Control Interfaces for hazard detection: response times increased significantly with increasing eccentric viewing by up to 2.9 seconds, sug- Gesture control has yet to be implemented in automobiles gesting that timely hazard detection required some degree of but gesture interfaces have been very successful in many mo- focal visual resources. Driver distraction has recently received bile communications and computing devices with touch- long overdue attention as a major contributor in accidents. Ad- screen interfaces. Similar interfaces could also be designed justing radio, cassette, and CD players has been estimated to and implemented for in-vehicle systems control in automo- cause 11.4% of drivers’ distractions (Stutts, Reinfurt, Staplin, biles. Gesture recognition interaction addresses many of the & Rodgman, 2001). In addition, it is best for safe driving to problems associated with voice control and allows for reduc- keep both hands on the steering wheel in case of sudden ma- tion of both visual and cognitive load of drivers. Gestures can neuvers are needed to avoid road hazards. also be viewed as an integral part of natural language and Current automobile interfaces can be very confusing with therefore they would be easy for drivers to learn. too many functions per control. For example, the BMW 7 se- There are two main techniques for gesture recognition, ries driver-controlled systems have over 700 functions (Gil- camera recognition and touch sensor recognition. Camera- bert, 2004). Many controls are hard to find and even invisible. based gesture recognition has serious spatial limitations, how- In an attempt to make in-vehicle controls better accessible to ever. A camera requires certain distance to recognize a driver's drivers and to allow them to keep their hands on the steering gesture, but most interiors of automobiles are quite small, wheel most of the time, most recent car models have placed making it difficult to install a camera that would accurately many controls on the steering wheel spokes as push-buttons or detect and recognize gestures. Another problem is the posi- toggle switches. Unfortunately, steering wheels can become tioning of the camera. As the majority of drivers are right- very crowded with nonstandard ways of grouping and assign- handed it seems natural that gestures could be performed by ing functionality to the buttons. A survey of eight car models the right hand. The best location for a gesture recognition (ranging from luxury to compact cars) showed that modern camera would therefore be to the right side of drivers. How- automobiles have on the average 11.62 buttons and 13.86 ever, the camera could be confused by a passenger’s gestures,
  2. 2. as the right side of the driver could be shared with a front pas- ledge or gesture stereotypes were then applied to controls in senger. an automobile. To simplify this analysis, only audio and venti- Touch sensor-based gesture recognition system is a good lation controls were considered. Finally, a prototype steering solution that circumvents the spatial limitations of camera- wheel accommodating touch pads for gesture control was de- based systems. Touch-based gesture recognition requires no signed. Note that only the initial design process is described distance from drivers. There are two feasible locations for here. The steering wheel prototype is not functional and so no touch-sensitive surfaces that could recognize gestures, on the usability testing of how well this design would work in an ac- center console or on the steering wheel. If a touch pad is in- tual driving environment has yet been conducted. stalled on the center console, however, drivers will need to take their hand off the steering wheel to operate the system, PRACTICE INNOVATION defeating the “eyes on the road and hands on the wheel” de- sign principle. Therefore, the best place for touch pads is on There were two main considerations in the design. First, the steering wheel. anthropometric principles and measures had to be considered There are also two kinds of touch recognition systems, in the physical placement and size and shape of the touch pads pressure and twist recognition system, and surface touch rec- on the steering wheel. Second, the actual control gestures ognition system. For automobile applications, pressure points would have to be intuitive to the users. Anthropometric di- could be incorporated in any place along the steering wheel mensions could be found in literature but gesture stereotypes rim while a touch sensor area could be located in the hub or had to be determined empirically. spokes of the steering wheel (Figure 1). Anthropometric Considerations Recognizes Because the steering wheel-mounted touch pad would pressure necessarily be operated by thumbs, the anthropometric meas- and twist ures of hand and in particular thumbs were the take as a start- Recognizes ing point of the design. The average length of a male thumb touch on the from crotch to end is 2.3 in, with a range from 2.1 to 2.9 in, surface and the average width 0.9 in, ranging from 0.55 to 1.25 in. For female thumbs, the average length is 2.1 in, with a range of 1.7 to 2.5 in. The average width of a female thumb is 0.75 in, ranging from 0.63 to 0.87 in. The lateral movement range of the thumb is from 80 degrees abduction to 45 degrees of ad- duction (Tilley, 1993). Figure 1. Possible locations of pressure points and touch- Experiment 1 sensitive surfaces on a steering wheel. Materials. To research gesture stereotypes of people for Examples of how to use pressure and twist recognition certain functions, a mock-up steering wheel with touch pads at system include a twist out to turn climate control on while two the 10:2 positions was created (Figure 2). The touch pads twists out would turn climate control off; a twist in could be were made out of thin white fabric, through which partici- air conditioner on and two twists in could be air conditioner pants’ thumb gestures could be videotaped. The length of the off. Available mappings of gestures to pressure and twisting sides of the pads was 2.5”, or 95% women’s thumb length. motions are very limited, however. A surface touch based rec- The angle between the two sides of the touch pad was 80˚, ognition system would allow for a wider range of gestures to which is the radial abduction angle. The participants were also be used as well as have other benefits as it is incorporated into queried about their preferred gestures by a questionnaire. The the steering wheel design. Two touch pads on the wheel questions concerned which side would be better for audio con- would also overcome some of the problems with depth of the trols and asked the participants to draw a preferred gesture for menu. For example, one side would be for climate controls common control actions. and another would be for audio controls. In addition, the best Procedure. Nineteen people volunteered for the experi- hand position of drivers while driving is 10–2 position and ment, 13 males and 6 females. Everyone had a driver’s li- therefore touch pads on the steering wheel should be located cense. Most participants were young college students, with the close to these locations. exception of one person. The participants sat facing the expe- rimenter and a video camera. The experimenter asked the par- Purpose of the Research ticipants to perform a control action on the steering wheel mock-up and they complied by making a gesture they felt was This paper describes the design process for a gestural vo- the most intuitive for the required control with their thumbs on cabulary interface for selected in-vehicle tasks. How people the “touch pads”. The gestures could be videotaped through perceive and understand the meaning of certain most com- the fabric mimicking the touch pads. After this the participants monly used gestures was investigated. This gestural know- drew gestures they made on the paper.
  3. 3. Two problems with gestures were discovered in this ex- last three functions, defrost, air from dashboard and air from periment. One is that the participants had trouble imagining underneath were a tap or a tap that has spatial meaning such as gestures in front of a camera and became nervous or rushed to top, center or bottom of the touch pad. figure them out. Another problem was that they used all im- In the videotaped gestures, 10 participants gestured aginable good, easy, and spontaneous gestures to “Turn the thumb up and down for volume up and down, and 6 of them controls on and off.” Hence, they ran out of ideas for good did thumb right and left for change station to higher and lower gestures for the remaining functions. station. However, 5 answers had directional meaning from right to left for changing radio station to higher station and left to right for changing station to lower station. Twelve people performed a tap for mute, 16 did a tap for pause Mp3 player and 14 a tap for play. Eleven people gestured thumb right from left for forward to next and 13 moved their thumbs from right to left for previous song. Ten participants did thumb up and down for temperature up and down, and 4 of them did thumb right and left for fan intensity up and down; 4 other gestured thumb up and down for fan intensity up and down. Nine participants tapped for defrost and 3 participants tapped for air form dashboard; 2 others used a tap that had spatial meaning like top and bottom. For example, for air from un- derneath, 3 participants used a tap of the bottom of a touch pad, one tapped the bottom left of the touch pad; and another just tapped once. Experiment 2 Figure 2. A mock-up steering wheel with touch pads for the Since it was possible that the relatively small touch pads study of gesture stereotypes. Positions of thumbs making the used in Experiment 1 (see Figure 2) may have constrained the gestures were videotaped through the thin material in the gestures performed by the participants and limited the variety mock touch pads. of gestures, a second experiment was run with a new steering wheel mock-up that had larger “touch pads” and with a new Consequently, the order of testing was changed to have group of participants. participants first draw gestures on paper with time to imagine Materials. The length of the new mock-up touch pad was the gestures for the controls, and then perform the gestures for 2.7 in, longer than the average length of male thumb. The the camera. This arrangement helped most participants to width of the touch pad now fully accommodated thumb make more gestures that were also more variable, but many movement of 45° adduction and 80° abduction angle. still ran out of ideas for “defrost” and “air from underneath”, Because the participants in the first experiment were pre- for example. Regardless, they were asked to just do something dominantly young college students, older participants were re- for each control action. cruited for the second experiment. Fourteen people volun- Findings. The results between the drawings of gestures on teered, 5 males and 9 females. Their ages ranged from twen- paper and those performed for video were almost identical. ties (one participant) to fifties. All had driver’s licenses and Fourteen (out of 19) people chose right side as audio control. drove daily. Over half of the participants offered same gestures for each The paper-and-pencil questionnaire was also slightly function, with the exception of controls for fan intensity and modified for the second experiment: questions about prefe- airflows. In the drawing task, 11 out of 19 participants sug- rences between gestural and voice command were added and gested thumb up and down for volume up and down, 9 did the boxes for drawing gestures were eliminated not to con- thumb right and left for change station to higher and lower strain free drawing of gestures in any way. Finally, a rating station. Eleven participants offered a tap for mute, 9 drew a scale (1-5) was added to gauge how good, easy, and sponta- tap to pause Mp3 player, and 11 a tap for play. Even though neous the most common gestures were. these three functions shared the same gesture, it would work Procedure. With the exception of research material and because radio and Mp3 player are different features and play questionnaires, the procedure was identical to that in Experi- and pause are opposite functions. Twelve people gestured ment 1. One participant only answered the questionnaire and thumb right and left for forward to next and previous song. was not videotaped. One participant suggested using a long tap as turn on and off Findings. The large surfaces of the touch pads confused like on a cellular phone, which would be easy to learn and some participants who thought that a big surface had spatial perform based on this experience. meaning; they tried to push or touch a certain points as if Ten participants offered thumb up and down for tempera- pushing imaginary buttons in the pad instead of making ges- ture up and down, and 6 of them did thumb right and left for tures. This group as a whole could not carry out gestures as fan intensity up and down. The most common answers of the
  4. 4. commands, having been accustomed to controlling devices neath and from air from underneath to defrost is a circulating with buttons for most of their lives. single tap. All 14 participants chose a conventional blinker over voice command; 8 preferred a conventional wiper control and DISCUSSION: FINAL DESIGN 6 preferred voice control, suggesting a preference for gesture to voice as a control of a simple and repeated function. Eleven Based on the anthropometric data, an ergonomically de- participants selected right side for audio controls, 2 people se- signed shape of a touch pad that looks similar to a piece of pie lected left side. Most of the gestures both drawn and per- was conceived. The sides of the touch pad are is 2.3 in long, formed by the second group of participants corresponded to which corresponds to the average thumb length of males. The those discovered in Experiment 1. On average, over a half of sides form an angle of 80°, which is the abduction angle of the participants preferred the same gestures as the first group. thumb. Similar touch pads are placed on both sides of the Gestures for air flow controls were most variable, as was the steering wheel (Figure 3). case in Experiment 1. FINDINGS There was much commonality in how people imagined gestures to perform common in-vehicle control tasks. Howev- er, the number of different gestures offered by the participants was limited, and some less used controls yielded diverse ges- ture suggestions. There were also some meaningful differences between the participants in the two experiments. Participants in the first experiment consisted of young college students in their twen- ties whereas participants in the second experiment were older adults. Younger people are accustomed to digital devices, es- pecially touch screens and pads and were very good at making various gestures on touch pad. The gestures gathered from the second experiment were less variable; some of them tried push imaginary buttons in their minds. All participants seemed to Figure 3. A technical drawing of the steering wheel design. run out of imagination for gestures for airflow control. In both The dimensions are in inches and the angle between the two experiments participants sometimes used their thumbs as if sides of the touch pad in degrees. drawing a gesture on the touch pad. From these results most common gestures could nevertheless be identified and used for assigning controls to gestures on a touch pad. These ges- tures for the right side, or audio controls were: Tap and hold: Audio system on/off Up/down movement: Volume up or down Left/right movement: Radio station, previous or next A single tap, or index finger on backside: Radio to Mp3 player switch Left/right movement: Previous or next song A single tap: Pause/play For the left side, or climate controls the gestures were: Tap and hold: Climate system on/off Up/down movement: Temperature up or down Left/right movement: Fan slower or faster Figure 4. Computer rendering of the final steering wheel A single tap: Defrost design. A single tap: Air from dashboard A single tap: Air from underneath. The two touch pads on the wheel appeared to surround and wrap the wheel. This appearance was carried over as an Interaction needs to be consistent to the user. Switching overall design concept, which underwent several iterations. play and stop or pause is a single tap on right side for audio The first draft had two spokes on wheel, which started from control. Therefore commands of airflow from defrost to air the pads. To differentiate the design from others, a one-spoke from dashboard, from air from dashboard to air from under-
  5. 5. design instead of the two-spoke design was adopted. This ini- that gestural control interfaces that are already ubiquitous in tial steering wheel design was full circular. However, to pre- many mobile communication and computing devices (e.g., so- vent the two touch pads, which were protruding into the inside called smart phones, Apple’s iPhone, iPod, and iPad, and most of the wheel, from occluding the instrument cluster, the steer- laptop computers) can find wider applications in automobiles. ing wheel was stretched laterally by one inch. The final design is depicted in Figure 4. ACKOWLEDGMENTS On the steering wheel, most automobiles have average 11.6 buttons and 13.7 functions. In this design there are no This paper is based on the first author’s Master of Fine buttons. Consequently, drivers do not need to look for Arts thesis for the graduate Industrial Design program at the grouped and small buttons. Audio and climate controls cause Rochester Institute of Technology. Many thanks are due to his the most drivers’ distractions among the secondary in-vehicle chief adviser, Prof. David Morgan for his help and support tasks. If these control interactions were as easy as our daily throughout this program. Dr. Rantanen served as an associate language, it would be very easy for drivers to operate in- adviser in the thesis committee. The helpful suggestions by a vehicle tasks. “Eyes on the road and hands on the wheel” is third thesis committee member, Dr. Michelle Harris, are also the maxim for safe driving. Automobile interactions must sa- gratefully acknowledged. tisfy this for safe drive. This imposed the least visual and cog- nitive load when controlling of in-vehicle systems. The pro- REFERENCES posed gesture-based interaction satisfies all the requirements stated above, potentially allowing drivers to drive less dis- Tilley, A. R. (1993). The measure of man and woman. Human tracted and more safely. Factors in Design. New York: Henry Dreyfuss Asso- Note that only the initial design process has been de- ciates. scribed in this paper, culminating in a non-functional proto- Pickering, C. A. (2005). Interacting with the car. IEE Compu- type steering wheel. To carry the development of this product ting & Engineering, 16(1), 26. further, many engineering problems (e.g., materials of the Summala, H., Nieminen, T., & Punto, M. (1996). Maintaining touch pads, their sensitivity, and gesture-recognition algo- lane position with peripheral vision during in-vehicle rithms and software) would have to be solved. An extensive tasks. Human Factors, 38 (3), 442-451. usability study would also need to be conducted to investigate Stutts, J .C., Reinfurt, D. W., Staplin, L., & Rodgman, E. A. how well drivers could learn the gestures mapped to various (2001). The role of driver distraction in traffic crashes. control actions and perform them reliably while driving in ac- AAA Foundation for Traffic Safety. Washington, D.C. tual traffic environments. Despite these limitations, however, this study revealed many new and potentially significant as- Gilbert, R. K. (2004). BMW i-drive. INFSCI 250. Pittsburgh, pects about drivers’ interactions with ever-increasing in- PA: University of Pittsburgh vehicle technologies and functions. Our research also suggests