Multimodal Presentation of Local Danger Warnings for Drivers: A Situation-Dependent Assessment of Usability
Multimodal Presentation of Local Danger Warnings for Drivers: A Situation-Dependent Assessment of Usability Yujia Cao Mariët Theune Christian Müller Human Media Interaction Human Media Interaction German Research Center University of Twente University of Twente for Artificial Intelligence P.O. Box 217, 7500 AE, P.O. Box 217, 7500 AE, Building D32, Campus Enschede, the Netherlands Enschede, the Netherlands Saarbrücken, Germany email@example.com firstname.lastname@example.org email@example.com Abstract urgent, so drivers usually have very limited time to think This study addresses how advanced driver assistant and react. Therefore, local danger warning messagessystems (ADAS) should communicate with drivers, should be communicated in a way that allows them to befocusing on the local danger warning function. To picked up quickly (efficiency) and correctlyachieve high-quality assistance, the communication mode (effectiveness).needs to be adaptive to changes in driving situation A challenge to reach this goal lies in the fact that(driver’s state, workload and environment). In a user driving conditions are very diverse, such as in heavystudy investigating different warning communication traffic, on an empty highway, under strong sunlight, in themodes, drivers were required to assess the usability of night, in the fog, just to name a few. Variations in drivingeach communication mode in five different driving conditions alter the demand driving imposes on the driver,situations. Results revealed that driving situation as well as the requirements on appropriate communicationsignificantly affected the perceived usefulness of each modes. For example, auditory messages are appropriate inwarning communication mode, mainly due to the use of a low-visibility condition, because drivers need to keepmodality. Moreover, regardless of communication mode, their eyes on the road. However, they might be lessthe local danger warning function was considered as the effective when there are rich sounds in the car, such asmost useful in the low visibility situation and the least radio and conversations. Therefore, there might not beuseful in the highly-demanding driving situation. one fixed communication mode that is the optimum for allFindings of this study can be applied to the design of conditions. Instead, ADAS need to be adaptive to changesADAS in general. Keywords: multimodal presentation, in the needs of the user, his/her workload and thelocal danger warning, adaptability. environment that (s)he is operating in [3, 4]. In this study, we investigated the usability of various communication modes for local danger warnings. Two aspects were considered, namely the level of assistanceIntroduction (what to communicate) and the use of modality (how to Advanced Driver Assistant Systems (ADAS) are in-car communicate). Besides the influence of communicationsystems to reduce or eliminate driver’s error, and enhance modes on the effectiveness and efficiency of warnings,the efficiency of traffic . One important function of we also intended to investigate the expected usefulness ofADAS, among others, is local danger warning. Local each communication mode in different driving situations.danger warning aims to extend the driver’s horizon and To this end, drivers were required to perform situation-warn him/her of dangerous situations coming up. Recent dependent assessment for each mode, based on their real-advances in inter-vehicle communication technology have life experiences. The selection of situation took intolargely promoted the application of local danger account both environmental and cognitive factors. Thiswarnings, because relevant information can be shared subjective assessment can be considered as a very firstbetween vehicles at runtime . step in the design process of a fully-adaptive system, To achieve high-quality assistance, ADAS need to because the results provide understating about drivers’communicate with the driver in an effective and efficient needs in different situations and how they expect theway. This is especially true in case of local danger system to adapt.warnings; because first, they are usually low-frequency The remainder of the paper is organized as follows.events, so drivers might be less ready for them compared First, the experiment is described, including task, warningwith other ADAS functions. Second, they are highly design, procedure and measurements. We then present
and discuss the results. Finally, conclusions and future 2. Low visibility - when driving with a low visibilitydirections are given. (in the night, fog etc.) 3. Fatigue - when being tired and unconcentratedExperiment 4. Long drive – during a long and boring drive (e.g. a long trip on the highway) A user study was carried out using a driving simulator 5. High demand - when driving in highly demandingintegrated in a real car. Drivers drove on a highway with situations (in heavy traffic, in unfamiliar city etc.)two lanes for the same direction. At random intervals, Each rating was performed on a 6-level scale from 0they received warnings about road obstacles in a short (not useful at all) to 5 (very useful). Note that thesedistance ahead (8 to 10 seconds’ drive) but not yet visible. situations were not actually simulated in the experiment,A warning message contained three elements: obstacle thus the ratings reflect the expected usefulness based ontype, location and distance. To avoid the danger, drivers drivers’ real-life experiences. By analyzing the ratings,were required to change lane if the obstacle was on the we intended to answer the following two questions:nearside lane and to brake if the obstacle was on the In which situation(s) is the assistance of local dangeroffside lane or on the roadside. warning thought to be more useful (appreciated)? Warning communication mode was manipulated by In each driving situation, which communicationtwo factors. The level of assistance varied between modes make the warnings more useful?warning only and warning preceded by action suggestion(brake or lane change). The use of modality had fouroptions: speech warning, visual warning with beep sound Resultscue, visual warning with blinking bar cue, and combined First, a two-way repeated-measure ANOVA wasvisual and speech warning. Note that action suggestions conducted on the rating scores, using warningwere always communicated via speech. In total, 8 communication mode (simply referred as ‘mode’ in thiscommunication modes were investigated, as summarized section) and situation as two independent factors. Resultsin Table 1. showed that both factors had a significant influence on the Table 1. The 8 communication modes investigated. usefulness of warnings (mode: F (7, 25) =11.7, p<0.001; situation: F (4, 28) =24.1, p<0.001). As expected, there 1 2 3 4 5 6 7 8 was also an interaction effect between these two factors Action suggestion √ √ √ √ (F (28, 4) =5.7, p<0.05). Visual message √ √ √ √ √ √ In the ‘low visibility’ situation, the modes received the Speech message √ √ √ √ highest usefulness rating score in average, whereas in the Beep sound cue √ √ ‘high demand’ situation they received the lowest (see Blinking bar cue √ √ Figure 1). Post-hoc analysis (Bonferroni tests) further revealed significant differences in rating score between 32 drivers participated in the experiment. The ‘low visibility’ and each of the other four situations. Theprocedure started with an introduction, followed by a same was also found for the ‘high demand’ situation. Intraining session. Afterwards, each driver drove 8 tracks combination, these results suggest that the assistance ofwith warnings communicated in different modes. The local danger warnings could be generally more useful andtrack order was counter-balanced. Drivers were required more needed (appreciated) in some situations than into fill in questionnaires during the short breaks between others, regardless of how warnings are communicated.two tracks. Drivers in this study considered local danger warnings the Following the ISO usability standard (ISO 9241-11 most useful when the visibility was low and the least), we evaluated the warning communication modes in useful when driving itself was highly demanding.terms of effectiveness (the danger avoidance performance,the message recall performance etc.), efficiency (reactiontime, driving load assessment etc.) and satisfaction(situation-dependent usability assessment). Regardingeffectiveness and efficiency, a detailed description of themeasurements and the results can be found in . Here,we focus on satisfaction. Satisfaction on warning communication was measuredin terms of situation-dependent subjective assessments.Drivers were asked to evaluate how useful eachcommunication mode would be in 5 specified situations: 1. Rich sound - when driving with rich surrounding sounds (noise, radio, conversation etc.) Figure 1. Average rating score over all modes for each situation. Error bars represent standard errors.
We then moved on to investigating which mode(s) not preferred. Furthermore, an interaction effect betweenmade the warnings the most useful in each driving the modality and the level of assistance was also found.situation. The approach was to zoom into each situation This is because the benefit of AS (always in speech) wasand perform a two-way repeated-measure ANOVA on the particularly pronounced with the ‘blink + visual’ modalityrating scores, using modality and level of assistance as variant, which was purely visual without AS.independent factors. The results regarding the level of assistance showed ahigh consistency - in all 5 situations, warnings with actionsuggestions (AS) were considered significantly more (a)useful than warnings without AS (see Figure 2-6). Thissuggests that AS could be generally beneficial, regardlessof the use of modality and the driving situation. However,results from the modality factor revealed diversity, whichactually explained the interaction effect between modeand situation. Rich sound: when there are rich sounds in the drivingenvironment, visual modalities are expected to be highly (b)necessary, because the saliency of auditory modalitiesdegrades in proportion to competing surrounding soundlevel . Indeed, the two purely auditory presentationmodes (speech with and without AS) were rated as theleast useful (see Figure 2(a)). Post-hoc (Bonferroni) testsconfirmed that speech (only) received significantly lowerrating scores than each of the other three modalityvariants. The difference between the other three modalityvariants did not reach a statistically significant level, (c)indicating that as long as there are visual modalitiesinvolved, the warning communication is useful in thisdriving condition. Low visibility: in this condition the ‘speech + visual’modality variant was considered as the most useful (seeFigure 2(b)). It was shown by Bonferroni tests to besignificantly more useful than each of the other threevariants. The explanation of this finding is twofold. First,when visibility is low, it is particularly important to keep (d)eyes on the road. Therefore, it should be more appropriateto communicate warnings orally rather than visually.Second, using only speech does not offer cognitiveadvantages because speech does not allow free perception, meaning that attention has to be focused on thespeech during its presentation, in order to fully perceivethe content. Therefore, it is beneficial to also providevisual warnings as supplement to speech warnings. Fatigue: in this condition, the ‘speech + visual’ and (e)‘beep + visual’ variants were rated as significantly moreuseful than the other two modality variants, and there wasno significant difference between the two (Figure 2(c)).This finding can be explained by the fact that auditorymodalities are much more salient than visual modalities.Attention is promptly directed to an auditory signal upon Figure 2. Average usefulness rating scores in eachthe onset of its presentation . When drivers are tired driving condition. (a) rich sound, (b) low visibility, (c)and unconcentrated, they tend to be less attentive. In this fatigue, (d) long drive, (e) high demand.case, the speech warning and the beep sound cue were Long drive: The ratings in this condition closelyboth considered useful, because they are able to attract resemble the ones in the ‘fatigue’ condition (Figure 2(d)).attention timely and increase vigilance level. However, This might be due to the fact that a long drive on theusing a purely auditory modality (speech only) was still
highway often makes the drive feel tired and less placed on communication modes. In this study, thisattentive. The ‘speech + visual’ and the ‘beep + visual’ influence was mainly reflected by the use of modality,variants were judged as significantly more useful than the rather than the level of assistance. Although derived fromother two. The interaction effect was also present. a local danger warning scenario, findings of this study can High demand: When the driving task imposes a high be applied to the design of ADAS in general. Future workcognitive load on the driver, the freedom of perception is to obtain a deeper understanding of the interactionbecomes particularly important, because when available between communication mode and driving situation, bycognitive recourses are limited, drivers might need to investigating a wider range of ADAS functions,frequently switch between the driving task and the communication factors and situations. It would also bewarning perception/comprehension. The rating scores for helpful to actually simulate the investigated situations inthis situation showed that the ‘speech + visual’ modality an experiment.variant clearly stood out (see Figure 2(e)). According toBonferroni tests, it was rated significantly higher than Acknowledgementseach of the other three variants, among which nosignificant difference was found. Auditory warnings can This work was funded by the EC Artemis project onbe perceived while keeping the eyes on the traffic, while Human-Centric Design of Embedded Systems (SmarcoS,visual warnings are self-paced and allow being read in Nr. 100249) and the German Ministry of Education andsegments at multiple times. In combination, the ‘speech + Research (project Car-Oriented Multimodal Interfacevisual’ modality variant could provide the largest freedom Architecture, grant number 01IW08004).of perception among the four. In summary, visual modalities are highly necessary Referenceswhen driving with rich surrounding sounds. Auditory  K. A. Brookhuis, D. De Waard, and W. H. Janssen,modalities are recommended in a low-visibility situation "Behavioural impacts of Advanced Driver Assistancedue to their “eyes-free” nature. Due to the ability to attract Systems–an overview," European Journal onattention and enhance vigilance, auditory modalities are Transportation and Infrastructure Research, vol. 1, pp. 245-also suitable when the driver is tired or unconcentrated, or 253, 2001.the trip is long and boring. The combination of visual andspeech warnings offers freedom of perception, thus is  T. Kosch, "Local danger warning based on vehicle ad-hocparticularly suitable during a highly demanding drive. In networks: Prototype and simulation," in First Internationalfact, this combination was rated as the (or one of the) Workshop on Intelligent Transportation (WIT’04), 2004.most useful modality variant(s) in all 5 situations.  L. M. Reeves, J. Lai, J. A. Larson, S. Oviatt, T. S. Balaji, S.However, this doesn’t simply imply that local danger Buisine, P. Collings, P. Cohen, B. Kraal, and J. C. Martin,warnings should always be communicated redundantly "Guidelines for multimodal user interface design,"via both visual modalities and speech. There are factors Communications of the ACM, vol. 47, pp. 57-59, 2004.other than driving situation that might influence theselection of an appropriate communication mode, such as  N. B. Sarter, "Multimodal information presentation: Designthe type of information and the availability of system guidance and research challenges," International journal ofresources. Therefore, what is really needed for the design industrial ergonomics, vol. 36, pp. 439-445, 2006.of adaptive systems is knowledge on how driving  "ISO 9241: Ergonomic requirements for office work withsituation changes the requirements on communication visual display terminals (VDTS) - part 11: Guidance onmodes, from which the appropriateness of each usability," 1998.communication mode under each situation can beinferred.  Y. Cao, A. Mahr, S. Castronovo, M. Theune, C. Stahl, and C. Müller, "Local danger warnings for drivers: The effect ofConclusions and Future Work modality and level of assistance on driver reaction," in International Conference on Intelligent User Interfaces The situation-dependent usability assessment in this (IUI’10), pp. 239-248 , 2010.study has confirmed that it is indeed necessary for ADASto adapt the communication mode of local danger  N. O. Bernsen, "Multimodality in language and speechwarnings to changes in driving situation. First of all, the systems - from theory to design support tool," in Multimodality in language and speech systems, B.functionality of local danger warning might be more Granström, D. House, and I. Karlsson, Eds., pp. 93-148,useful (appreciated) in some situations than in others. 2001.Drivers in this study considered local danger warnings themost useful when the visibility was low and the least  C. Spence, M. E. R. Nicholls, and J. Driver, "The cost ofuseful when driving itself was highly demanding. Second, expecting events in the wrong sensory modality," Perceptionthe driving situation also influences the requirements and Psychophysics, vol. 63, pp. 330-336, 2001.