Emochat: Emotional instant messaging with the Epoc headset

ABSTRACTTitle of Document: EMOCHAT: EMOTIONAL INSTANT MESSAGING WITH THE EPOC HEADSET Franklin Pierce Wright Master of Science 2010Directed By: Asst. Professor of Human-Centered Computing Dr. Ravi Kuber Information SystemsInterpersonal communication benefits greatly from the emotional informationencoded by facial expression, body language, and tone of voice, however thisinformation is noticeably missing from typical instant message communication. Thiswork investigates how instant message communication can be made richer byincluding emotional information provided by the Epoc headset. First, a studyestablishes that the Epoc headset is capable of inferring some measures of affect withreasonable accuracy. Then, the novel EmoChat application is introduced which usesthe Epoc headset to convey facial expression and levels of basic affective statesduring instant messaging sessions. A study compares the emotionality ofcommunication between EmoChat and a traditional instant messaging environment.Results suggest that EmoChat facilitates the communication of emotional informationmore readily than a traditional instant messaging environment.

EMOCHAT: EMOTIONAL INSTANT MESSAGING WITH THE EPOC HEADSET By Franklin Pierce Wright Thesis submitted to the Faculty of the Graduate School of the University of Maryland, Baltimore County, in partial fulfillment of the requirements for the degree of Master of Science 2010

Table of ContentsAcknowledgements ................................................................................................................. iiTable of Contents ................................................................................................................... iiiList of Tables .......................................................................................................................... viList of Figures........................................................................................................................ vii1 Introduction .................................................................................................................... 1 1.1 The Importance of Emotion ..................................................................................... 1 1.2 Instant Messaging .................................................................................................... 2 1.3 The Emotional Problem with Instant Messaging .................................................... 2 1.4 Purpose of this Work ............................................................................................... 3 1.5 Structure of this Document ...................................................................................... 42 Emotion ........................................................................................................................... 6 2.1 What is Emotion? .................................................................................................... 6 2.1.1 The Jamesian Perspective ................................................................................... 6 2.1.2 The Cognitive-Appraisal Approach .................................................................... 7 2.1.3 Component-Process Theory ................................................................................ 8 2.2 Emotion and Related Affective States ...................................................................... 9 2.2.1 Primary versus Complex ..................................................................................... 9 2.3 Expressing Emotion ................................................................................................. 9 2.3.1 Sentic Modulation ............................................................................................... 9 2.4 Measuring Emotion ............................................................................................... 10 2.4.1 Self-Report Methods ......................................................................................... 10 2.4.2 Concurrent Expression Methods ....................................................................... 11 2.5 Conclusion ............................................................................................................. 123 Existing Techniques for Conveying Emotion during Instant Messaging ................ 13 3.1 Introduction ........................................................................................................... 13 3.2 Input Techniques ................................................................................................... 13 3.2.1 Textual Cues...................................................................................................... 13 3.2.2 Automated Expression Recognition .................................................................. 14 3.2.3 Physiologic Data ............................................................................................... 15 3.2.4 Manual Selection ............................................................................................... 15 3.3 Output Techniques ................................................................................................. 16 3.3.1 Emoticons.......................................................................................................... 16 3.3.2 Expressive Avatars ............................................................................................ 17 3.3.3 Haptic Devices .................................................................................................. 19 3.3.4 Kinetic Typography .......................................................................................... 21 3.4 Conclusion ............................................................................................................. 224 Study 1: Validating the Emotiv Epoc Headset ......................................................... 23 4.1 Introduction ........................................................................................................... 23 4.2 Overview of the Epoc Headset............................................................................... 23 4.2.1 Expressiv Suite .................................................................................................. 24 4.2.2 Affectiv Suite .................................................................................................... 25 4.2.3 Cognitiv Suite.................................................................................................... 25 iii

4.3 The Need for Validation ........................................................................................ 25 4.4 Experimental Design ............................................................................................. 26 4.4.1 TetrisClone System Development ..................................................................... 27 4.4.2 Measures ........................................................................................................... 28 4.5 Experimental Procedures ...................................................................................... 30 4.6 Results and Analysis .............................................................................................. 32 4.6.1 Headset-Reported versus Self-Reported Levels of Affect ................................ 32 4.6.2 Subjective Causes of Affect during Gameplay ................................................. 38 4.7 Discussion ............................................................................................................. 40 4.7.1 Consistency in the Present Study ...................................................................... 40 4.7.2 Future Direction ................................................................................................ 42 4.8 Conclusion ............................................................................................................. 425 Study 2: Emotional Instant Messaging with EmoChat ............................................ 44 5.1 Introduction ........................................................................................................... 44 5.2 EmoChat System Development .............................................................................. 45 5.2.1 Overview ........................................................................................................... 45 5.2.2 Traditional Environment ................................................................................... 47 5.2.3 Application Architecture ................................................................................... 47 5.3 Experimental Design ............................................................................................. 48 5.3.1 Measures ........................................................................................................... 49 5.4 Experimental Setup ................................................................................................ 53 5.5 Experimental Procedures ...................................................................................... 53 5.6 Results and Analysis .............................................................................................. 55 5.6.1 Emotional Transfer Accuracy ........................................................................... 55 5.6.2 Richness of Experience ..................................................................................... 59 5.6.3 Chat Transcripts ................................................................................................ 63 5.6.4 System Usability ............................................................................................... 66 5.6.5 Informal Interviews ........................................................................................... 66 5.7 Discussion ............................................................................................................. 73 5.7.1 Summary of Results .......................................................................................... 73 5.7.2 Consistency with related work .......................................................................... 75 5.7.3 Future Direction ................................................................................................ 77 5.8 Conclusion ............................................................................................................. 786 Conclusion ..................................................................................................................... 79 6.1 Limitations ............................................................................................................. 79 6.2 Summary of Contributions ..................................................................................... 82 6.3 EmoChat Design Considerations .......................................................................... 84 6.4 EmoChat Compared with Existing Methods of Emotional Instant Messaging ..... 85 6.4.1 Input Technique ................................................................................................ 85 6.4.2 Output Technique .............................................................................................. 86 6.5 Future Direction .................................................................................................... 87 6.6 Closing................................................................................................................... 88Appendix A: Validation Study Demographics Questionnaire ......................................... 90Appendix B: Validation Study Experiment Questionnaire.............................................. 91Appendix C: Validation Study Post-Experiment Questionnaire ..................................... 92Appendix D: EmoChat Demographics Questionnaire...................................................... 93Appendix E: EmoChat Emotional Transfer Questionnaire............................................. 94 iv

Appendix F: EmoChat Richness Questionnaire ............................................................... 95Bibliography .......................................................................................................................... 96 v

List of TablesTable 3.1 Common emoticons in Western and Eastern cultures ............................... 16Table 3.2 Example of hapticons................................................................................. 20Table 4.1 Facial expression features measured by the Epoc headset ........................ 24Table 4.2 Headset and self-reported levels of affect per subject, per trial................. 33Table 4.3 Spearman correlation between headset and self-reported levels of affect (N=36) ................................................................................................................. 34Table 4.4 Spearman correlation of headset and self-report data for varied time divisions (N=36) ................................................................................................. 35Table 4.5 Grand mean headset and self-reported levels of affect per difficulty level 36Table 4.6 Grand mean Spearman correlation between headset and self-reported levels of affect (N=3) .................................................................................................... 36Table 4.7 Major themes identified in subjective affect survey results ...................... 39Table 5.1 Facial movements and affective information used by EmoChat ............... 46Table 5.2 EmoChat experimental groups................................................................... 48Table 5.3 ETAQ scores for each subject-pair and both experimental conditions ..... 56Table 5.4 Spearman correlation matrix between avatar features and perceived frequency of emotional states (N=5) .................................................................. 58Table 5.5 Wilcoxons signed rank test for significant difference between score means (N=10) ................................................................................................................. 61Table 5.6 Linguistic categories with significant differences between experimental conditions ............................................................................................................ 64Table 5.7 LIWC affective processes hierarchy .......................................................... 65Table 5.8 LIWC relativity hierarchy ........................................................................... 66 vi

List of FiguresFigure 3.1 Examples of expressive avatars ................................................................ 18Figure 4.1 Initialization screen for the TetrisClone application ................................ 28Figure 4.2 TetrisClone application during trials ........................................................ 28Figure 4.3 Example output from the TetrisClone application ................................... 29Figure 4.4 Comparison of grand mean headset and self-reported levels of excitement ............................................................................................................................. 37Figure 4.5 Comparison of grand mean headset and self-reported levels of engagement ......................................................................................................... 37Figure 4.6 Comparison of grand mean headset and self-reported levels of frustration ............................................................................................................................. 37Figure 5.1 The EmoChat client application ............................................................... 45Figure 5.2 EmoChat server application ..................................................................... 47Figure 5.3 Mean scores from the richness questionnaire, questions 1-5 ................... 59Figure 5.4 Mean scores from the richness questionnaire, questions 5-10 ................. 60Figure 5.5 Comparison of mean responses to REQ between subjects with headsets versus without headsets, in the EmoChat condition, Q1-5 (N=5) ...................... 62Figure 5.6 Comparison of mean responses to REQ between subjects with headsets versus without headsets, in the EmoChat condition, Q5-10 (N=5) .................... 62 vii

1 IntroductionThis chapter introduces the importance of emotion in interpersonal communication,and presents some of the challenges with including emotion in instant messages. Thepurpose of this thesis is then stated, followed by an overview of the documentstructure.1.1 The Importance of EmotionConsider the following statement: “The Yankees won again.”Does the person who makes this remark intend to be perceived as pleased ordisappointed? Enthusiastic or resentful? The remark is purposely emotionallyambiguous to illustrate just how powerful the inclusion or absence of emotion can be.If the same remark were said with a big grin on the face, or with the sound ofexcitement in the voice, we would certainly understand that this person was quitepleased that his team was victorious.If the speaker displayed slumped shoulders and a head tilted downward we wouldassume that he was certainly less than jubilant. 1

It is clear that emotions play a very important role in interpersonal communication,and without them, communication would be significantly less efficient. A statementthat contains emotion implies context, without the necessity of explicit clarification.In some cases what is said may be equally as important as how it is said.1.2 Instant MessagingReal-time text-based communication is still on the rise. Instant messaging, in oneform of another, has infiltrated nearly all aspects of our digital lives, and shows nosign of retreat. From work, to school, to play, it’s becoming more and more difficultto shield ourselves from that popup, or that minimized window blinking in the taskbar, or that characteristic sound our phones make when somebody wants to chat withus. We are stuck with this mode of communication for the foreseeable future.1.3 The Emotional Problem with Instant MessagingAs convenient as it is, this text-based communication has inherent difficultiesconveying emotional information. It generally lacks intonation and the subtle non-verbal cues that make face-to-face communication the rich medium that it is. Facialexpression, posture, and tone of voice are among the highest bandwidth vehicles ofemotional information transfer (Pantic, Sebe, Cohn, & Huang, 2005), but arenoticeably absent from typical text-based communication. According to Kiesler andcolleagues, computer-mediated communication (CMC) in general is “observablypoor” for facilitating the exchange of affective information, and note that CMC 2

participants perceive the interaction as more impersonal, resulting in less favorableevaluations of partners (Kiesler, Zubrow, Moses, & Geller, 1985).The humble emoticon has done its best to remedy the situation by allowing textstatements to be qualified with the ASCII equivalent of a smile or frown. While thissuccessfully aids in conveying positive and negative affect (Rivera, Cooke, & Bauhs,1996), emoticons may have trouble communicating more subtle emotions. Othersolutions that have been proposed to address this problem are reviewed in chapter 3.Each solution is successful in its own right, and may be applicable in differentsituations. This work examines a novel method for conveying emotion in CMC,which is offered to the community as another potential solution.1.4 Purpose of this WorkThe main goal of this body of work is to investigate how instant messagecommunication is enriched by augmenting messages with emotional content, andwhether this can be achieved through the use of brain-computer interface (BCI)technology. The Emotiv Epoc headset is a relatively new BCI peripheral intendedfor use by consumers and is marketed as being capable of inferring levels of basicaffective states including excitement, engagement, frustration, and meditation. Astudy presented in this work attempts to validate those claims by comparing datareported by the headset with self-reported measures of affect during game play atvaried difficulty levels. The novel EmoChat application is then introduced, whichintegrates the Epoc headset into an instant messaging environment to control the 3

facial expressions of a basic animated avatar, and to report levels of basic affectivestates. A second study investigates differences between communication withEmoChat and a traditional instant messaging environment. It is posited that theEmoChat application, when integrated with the Epoc headset, facilitatescommunication that contains more emotional information, that can be described asricher, and that conveys emotional information more accurately than with traditionalIM environments.In the end, this complete work intends to provide, first, a starting point for otherresearchers interested in investigating applications that implement the Epoc headset,and second, results which may support the decision to apply the Epoc in computer-mediated communication settings.1.5 Structure of this DocumentThe remaining chapters of this work are structured as follows:Chapter 2 provides an overview of emotion, including historical perspectives, andhow emotions are related to affective computing. Chapter 3 reviews existingtechniques for conveying emotion in instant messaging environments. Chapter 4details a study to determine the accuracy of the Epoc headset. Chapter 5 introducesEmoChat, a novel instant messaging environment for exchanging emotionalinformation. A study compares EmoChat with a traditional instant messagingenvironment. Chapter 6 summarizes the contributions this work makes, and 4

compares the techniques for conveying emotion used in EmoChat with techniquesdescribed in the literature. 5

2 EmotionThis chapter describes some of the historical perspectives on emotion, and introducesits role in affective computing. It intends to provide a background helpful for thestudy of emotional instant messaging.2.1 What is Emotion?The problem of defining what constitutes human emotion has plagued psychologistsand philosophers for centuries, and there is still no generally accepted descriptionamong researchers or laypersons. A complicating factor in attempting to defineemotion is our incomplete understanding of the complexities of the human brain.Some theorists have argued that our perceptions enter the limbic system of the brainand trigger immediate action without consultation with the more developed cortex.Others argue that the cortex plays a very important role in assessing how we relate toany given emotionally relevant situation, and subsequently provides guidance abouthow to feel.2.1.1 The Jamesian PerspectiveIn 1884 psychologist William James hypothesized that any emotional experience withwhich physiological changes may be associated requires that those samephysiological changes be expressed before the experience of the emotion (James,1884). In essence, James believed that humans feel afraid because we run from abear, and not that we run from a bear because we feel afraid. James emphasized the 6

physical aspect of emotional experience causation over the cognitive aspect. Thisphysical action before the subjective experience of an emotion has subsequently beenlabeled a “Jamesian” response. For historical accuracy, note, that at about the sametime that James was developing his theory Carl Lange independently developed avery similar theory. Collectively, their school of thought is referred to as James-Lange (Picard, 1997).2.1.2 The Cognitive-Appraisal ApproachIn contrast to James’ physical theory of emotion, a string of psychologists laterdeveloped several different cognitive-based theories. Notable among them is thecognitive-appraisal theory, developed by Magda Arnold, and later extended byRichard Lazarus, which holds that emotional experience starts not with a physicalresponse, but with a cognitive interpretation (appraisal) of an emotionally-inspiringsituation (Reisenzein, 2006). In continuing with the bear example, Arnold andLazarus would have us believe that we hold in our minds certain evaluations of thebear-object (it is dangerous and bad for us), we see that the bear is running toward us(is, or will soon be present), we anticipate trouble if the bear reaches us (poor copingpotential), and so we experience fear and run away.Certainly, there are valid examples of situations that seem to trigger Jamesianresponses. Consider the fear-based startle response when we catch a large objectquickly approaching from the periphery. It is natural that we sometimes react tostartling stimuli before the experience of the fear emotion, jumping out of the way 7

before we even consciously know what is happening to us. Conversely, consider joy-based pride after a significant accomplishment. It seems as though pride could onlybe elicited after a cognitive appraisal determines that (a) the accomplishment-object ispositive, (b) it has been achieved despite numerous challenges, and (c) it will not bestripped away. If examples can be found that validate both the James-Langeapproach and the cognitive-appraisal approach, is one theory more correct than theother?2.1.3 Component-Process TheoryIt is now suggested that emotional experience may result from very complexinteraction between the limbic system and cortex of the brain, and that emotions canbe described as having both physical and cognitive aspects (Picard, 1997).Encompassing this point of view, that a comprehensive theory of emotion shouldconsider both cognitive and physical aspects, is the component-process modelsupported by Klaus Scherer (Scherer, 2005). This model describes emotion asconsisting of synchronized changes in several neurologically based subsystems,including, cognitive (appraisal), neurophysiologic (bodily symptoms), motivational(action tendencies), motor expression (facial and vocal expression), and subjectivefeeling (emotional experience) components. Note that Scherer regards “subjectivefeeling” as a single element among many in what constitutes an “emotion.” 8

2.2 Emotion and Related Affective States2.2.1 Primary versus ComplexSeveral emotion classes exist that are more basic than others. These are the emotionsthat seem the most Jamesian in nature—hard coded, almost reflex like responses that,from an evolutionary perspective, contribute the most to our survival. Fear and angerare among these basic, primary, emotions. Picard labels these types of emotions as“fast primary,” and suggests that they originate in the limbic system. This is incontrast to the “slow secondary,” or cognitive-based emotions that require time forintrospection and appraisal, and therefore require some cortical processing. Scherercalls this slow type of emotion “utilitarian,” in contrast with the fast, which he terms,“aesthetic.” An important distinction should be made between emotions and otherrelated affective states such as moods, preferences, attitudes, and sentiments. Adistinguishing factor of emotions is the comparatively short duration when consideredamong the other affective states.2.3 Expressing Emotion2.3.1 Sentic ModulationIf emotion has both physical and cognitive aspects, it seems natural that someemotions can be experienced without much, if any, outward expression. Interpersonalcommunication may benefit from those overt expressions of emotion that can beperceived by others. Picard discusses what she calls “sentic modulation,” overt orcovert changes is physiological features that, although do not constitute emotion 9

alone, act as a sort of symptom of emotional experience (Picard, 1997). The easiestof these sentic responses to recognize are arguably facial expression, tone of voice,and posture, or body language. Of these three, research suggests that facialexpression is the highest bandwidth, with regard to the ability to convey emotionalstate (Pantic, et al., 2005). There are other, more covert, symptoms of emotionalexperience, including heart rate, blood pressure, skin conductance, pupil dilation,perspiration, respiration rate, and temperature (Picard, 1997). Recent research hasalso demonstrated that some degree of emotionality may also be inferred byneurologic response as measured by electroencephalogram (Khalili & Moradi, 2009;Sloten, Verdonck, Nyssen, & Haueisen, 2008). Facial expression deserves additionalattention, being one of the most widely studied forms of sentic modulation. Ekmanand others have identified six basic emotion/facial expression combinations thatappear to be universal across cultures including, fear, anger, happiness, sadness,disgust, and surprise (Ekman & Oster, 1979). The universality of these facialexpressions, that they are so widely understood, suggests that it should be quite easyto infer emotion from them.2.4 Measuring Emotion2.4.1 Self-Report MethodsPerhaps the most widely used method for determining emotional state is through self-report. This technique asks a subject to describe an emotion he or she isexperiencing, or to select one from a pre-made list. Scherer discusses some of the 10

problems associated with relying on self-reported emotional experience. On onehand, limiting self-report of emotion to a single list of words from which the subjectmust choose the most appropriate response may lead to emotional “priming,” and/or amisrepresentation of true experience. On the other hand, allowing a subject togenerate a freeform emotional word to describe experience adds significant difficultyto any type of analysis (Scherer, 2005). Another method for self-report measurementis to have a subject identify emotional state within the space of some dimension.Emotional response is often described as falling somewhere in the two-dimensionalvalence/arousal space proposed by Lang. This dimensional model of affectdeconstructs specific emotions into some level of valence (positive feelings versusnegative feelings), and some level of arousal (high intensity versus low intensity)(Lang, 1995). As an example, joyful exuberance is categorized by high valence andhigh arousal, while feelings of sadness demonstrate low valence and low arousal.Lang posits that all emotion falls somewhere in this two-dimensional space. Problemsmay arise when emotion is represented in this space without knowing the triggeringevent, since several distinct emotions may occupy similar locations in valence-arousalspace, e.g., intense anger versus intense fear, both located in high-arousal, low-valence space (Scherer, 2005).2.4.2 Concurrent Expression MethodsAn objective way to measure emotional state is by inferring user affect by monitoringsentic modulation. This method requires the use of algorithms and sensors in order toperceive the symptoms of emotion and infer state, and is significantly more 11

challenging than simply asking a person how he feels (Tetteroo, 2008). Still, this isan active area of research within the affective computing domain because userintervention is not required to measure the emotion, which may be beneficial in somecases. Techniques include using camera or video input along with classificationalgorithms to automatically detect emotion from facial expression (Kaliouby &Robinson, 2004), monitoring galvanic skin response to estimate level of arousal(Wang, Prendinger, & Igarashi, 2004), and using AI learning techniques to inferemotional state from electroencephalograph signals (Khalili & Moradi, 2009; Sloten,et al., 2008).2.5 ConclusionPicard defines affective computing as, “computing that relates to, arises from, orinfluences emotion.” (Picard, 1997) According to Picard, some research in thisdomain focuses on developing methods of inferring emotional state from sentic usercharacteristics (facial expression, physiologic arousal level, etc.), while other researchfocuses on methods that computers could use to convey emotional information(avatars, sound, color, etc.) (Picard, 1997). A study of emotional instant messagingis necessarily a study of affective computing. In the context of instant messaging,Tetteroo separates these two research areas of affective computing into the study ofinput techniques, and output techniques (Tetteroo, 2008). The next chapter reviewshow these techniques are used during instant message communication to conveyemotional information. 12

3 Existing Techniques for Conveying Emotion during Instant Messaging3.1 IntroductionA review of the current literature on emotional communication through instantmessage applications has identified several techniques for enriching text-basedcommunication with affective content. These techniques can be broadly classified aseither input techniques, inferring or otherwise reading in the emotion of the user, oroutput techniques, displaying or otherwise conveying the emotion to the partner(Tetteroo, 2008). These categories are reviewed in turn.3.2 Input TechniquesResearch concerning how emotions can be read into an instant messaging systemgenerally implements one of several methods: inference from textual cues, inferencethrough automated facial expression recognition, inference from physiologic data, ormanual selection.3.2.1 Textual CuesInput methods that use text cues to infer the emotional content of a message generallyimplement algorithms that parse the text of a message and compare its contents with adatabase of phrases or keywords for which emotional content is known. Yeoimplements a basic dictionary of emotional terms, such as “disappointed,” or 13

“happy,” that incoming messages are checked against. When a match is found, thegeneral affective nature of the message can be inferred (Yeo, 2008). Others haveused more complicated natural language processing algorithms to account for thesubtleties of communicative language (Neviarouskaya, Prendinger, & Ishizuka,2007).Another example of using text cues to infer the emotion of a message involves simplyparsing text for occurrences of standard emoticons. The presence of the smileyemoticon could indicate a positively valenced message, while a frowning emoticoncould indicate negative valence, as implemented by Rovers & Essen (2004).3.2.2 Automated Expression RecognitionThe goal during automated expression recognition involves using classificationalgorithms to infer emotion from camera or video images of a subject’s face.Kaliouby and Robinson use an automated “facial affect analyzer” in this manner toinfer happy, surprised, agreeing, disagreeing, confused, indecisive, and neutral statesof affect (Kaliouby & Robinson, 2004). The classifier makes an evaluation aboutaffective state based on information about the shape of the mouth, the presence ofteeth, and head gestures such as nods. 14

3.2.3 Physiologic DataSome physiologic data is known to encode levels of affect, including galvanic skinresponse, skin temperature, heart beat and breathing rate, pupil dilation, and electricalactivity measured from the surface of the scalp (Picard, 1997). Wang and colleaguesused GSR data to estimate levels of arousal in an instant messaging application(Wang, et al., 2004). Specifically, spikes in the GSR data were used to infer highlevels of arousal, and the return to lower amplitudes signaled decreased level ofarousal.Output from electroencephalograph (EEG) has also been used to classify emotionalstate into distinct categories within arousal valence space by several researchers(Khalili & Moradi, 2009; Sloten, et al., 2008). These studies use AI learningtechniques to classify affective state into a small number of categories with moderatesuccess.3.2.4 Manual SelectionThe most basic method of adding emotional content to an instant message is bysimple manual selection or insertion. This method can take the form of a userselecting from a list of predefined emotions or emotional icons with a mouse click, orby explicitly inserting a marker, e.g., emoticon, directly in to the body of the messagetext. This type of input technique is widely used and is seen in research by (Fabri &Moore, 2005; Sanchez, Hernandez, Penagos, & Ostrovskaya, 2006; Wang, et al.,2004). 15

3.3 Output TechniquesOutput techniques describe methods that can be used to display emotional content toa chat participant after it has been input into the system. These techniques generallyinvolve using emoticons, expressive avatars, haptic devices, and kinetic typography.3.3.1 EmoticonsEmoticons are typically understood as small text-based or graphical representations offaces that characterize different affective states, and have been ever evolving in anattempt to remedy the lack of non-verbal cues during text chat (Lo, 2008). Examplesof commonly used emoticons are presented in the table below. Meaning Western Emoticon Eastern Emoticon Happy :-) (^_^) Sad :-( (T_T) Surprised :-o O_o Angry >:-( (>_<) Wink ;-) (~_^) Annoyed :-/ (>_>) Table 3.1 Common emoticons in Western and Eastern culturesEmoticons are perhaps the most widely used method for augmenting textualcommunication with affective information. A survey of 40,000 Yahoo Messengerusers reported that 82% of respondents used emoticons to convey emotionalinformation during chat (Yahoo, 2010). A separate study by Kayan and colleaguesexplored differences in IM behavior between Asian and North American users andreported that of 34 total respondents, 100% of Asian subjects used emoticons while 16

72% of North Americans (with an aggregate of 85% of respondents) used emoticonson a regular basis (Kayan, Fussell, & Setlock, 2006). These usage statisticsunderscore the prevalence of emoticons in IM communication.Sanchez and colleagues introduced an IM application with a unique twist on thestandard emoticon. Typical emoticons scroll with the text they are embedded in, andso lack the ability to convey anything more than brief fleeting glimpses of emotion.This novel application has a persistent area for emoticons that can be updated as oftenas the user sees fit, and does not leave the screen as messages accumulate (Sanchez,et al., 2006). Building on Russel’s model of affect (Russell, 1980), the teamdeveloped 18 different emoticons, each with three levels of intensity to represent asignificant portion of valence/arousal emotional space.3.3.2 Expressive AvatarsUsing expressive avatars to convey emotion during IM communication may beconsidered a close analog to the way emotion is encoded by facial expression duringface-to-face conversation, considering that facial expression is among the highestbandwidth channels of sentic modulation (Pantic, et al., 2005). 17

Figure 3.1 Ex xamples of expr ressive avatarsA study by Kaliouby and Robinson presents an in K p nstant messa aging applica ation calledFAIM which uses automa facial ex ated xpression re cognition to infer affect, and odisplays an ex xpressive av vatar reflectin that affec to the chat partner (Ka ng ct t aliouby &Robinson, 2004). Affecti states cuR ive urrently supp ported by FA include happy, AIMsu urprised, agr reeing, disag greeing, conf fused, indeci isive, and ne eutral.Fabri and Mo oore investig gated the use of animated avatars cap d pable of emo otional facial lex xpressions in an instant messaging environment (Fabri & M n m e Moore, 2005). Theyco ompared this with a condition in wh the avata was not an hich ar nimated and did not dch hange facial expressions except for minor rando eyebrow movement. The s, om why ypothesis wa that the co as ondition in question wou result in a higher lev of q uld vel“r richness,” co omprised of high levels of task invol o lvement, enj joyment, sen of nsepr resence, and sense of co d opresence. Participants i P interacted thr rough the IM application M ndu uring a class sical surviva exercise, in which both subjects w tasked w al n h were withco ollectively ordering a lis of survival items in ter of impor o st l rms rtance. An a avatarre epresenting each chat pa e artner could be made to d b display one o Ekman’s six universa of alfa acial express sions includi happines surprise, a ing ss, anger, fear, sadness, and disgust d 18

(Ekman & Oster, 1979), by clicking on a corresponding icon in the interface.Significant results from the study indicated higher levels of task involvement andcopresence in the expressive avatar condition, equally high levels of presence in bothconditions, and a higher level of enjoyment in the non-expressive avatar condition.The AffectIM application developed by Neviarouskaya also uses expressive avatarsto convey emotion during instant message communication (Neviarouskaya, et al.,2007). Rather than requiring a user to select an expression from a predefined set,AffectIM infers the emotional content of a message by analyzing the text of themessage itself, and automatically updates an avatar with the inferred emotion. Acomparison study identified differences between separate configurations of theAffectIM application: one in which emotions were automatically inferred, one thatrequired manual selection of a desired emotion, and one that selected an emotionautomatically in a pseudo-random fashion (Neviarouskaya, 2008). The studycompared “richness” between conditions, comprised of interactivity, involvement,sense of copresence, enjoyment, affective intelligence, and overall satisfaction.Significant differences indicated a higher sense of copresence in the automaticcondition than in the random condition, and higher levels of emotional intelligence inboth the automatic and manual conditions than in the random condition.3.3.3 Haptic DevicesHaptic instant messaging is described as instant messaging that employs waveformsof varying frequencies, amplitudes, and durations, transmitted and received by 19

pu urpose-built haptic devic (force-fe t ces eedback joys sticks, haptic touchpads, etc.), to c ,which special emotional meaning can be attached (Rovers & Essen, 2004 Roversw l m n d 4).an Essen int nd troduce their idea of “hap r pticons,” wh hich are desc cribed as “sm mallpr rogrammed force pattern that can be used to co ns b ommunicate a basic notio in a onsi imilar manne as ordinar icons are used in grap er ry u phical user in nterfaces.” T Theirpr reliminary application, HIM, parses instant mes sage text for occurrence of a H r estr raditional em moticons, e.g :), etc., an sends a pr g., nd redefined wa aveform to a connecte any edhaptic devices. For exam mple, a smiley face sends a waveform with mode y s m eratefr requency tha slowly gro in ampli at ows itude, while a frowny fac is represe ce ented byse everal abrup pulses with high frequency and am pt h mplitude. Emotic con Mean ning Hapt ticon Wavef form :-) Hap ppy :-( Sa ad Table 3.2 Example of h 2 hapticonsThe ContactIM applicatio developed by Oakley and O’ModT M on d dhrain takes a differentap pproach to in ntegrating ha aptic inform mation with a instant me an essaging env vironment. Aplugin for the Miranda IM environme was creat that mim the effec of tossing e M ent ted mics cts ga ball between partners by using a for enabled h n y rce haptic devic such as the phantom o ce e ora standard for rce-feedback joystick (O k Oakley, 2003 The appli 3). ication is des signed toal llow each us to impart a specific velocity and t ser t v trajectory to the ball dur o ring a throw.The generated momentum of the ball is persistent until the ch partner picks it up. InT d m t hatth way, the act of tossin the ball may convey s his ng m some degree of emotiona ality, e.g., a 20

lightly thrown ball as a playful flirtatious gesture, or a fast throw to indicatedisagreement or anger. Emphasis is placed on the asynchronous nature of typicalinstant message use, and the application has been designed to suit this mode ofinteraction by keeping the general characteristics of the tossed ball persistent untilinteraction by the receiver changes it.3.3.4 Kinetic TypographyKinetic typography is described as real time modification of typographiccharacteristics including animation, color, font, and size, etc., and may be used toconvey affective information (Yeo, 2008). Yeo developed an IM client that inferredaffective meaning through keyword pattern matching, and used kinetic typography toupdate the text of messages in real time (Yeo, 2008).An instant messaging client developed by Wang and colleagues represents emotion inarousal/valence space by combining kinetic typography with galvanic skin response.Manually selected text animations are meant to represent valence, while GSR that isrecorded and displayed to the chat partner represents level of arousal. Users wereasked when they felt the most involved during the online communication, andanswers typically corresponded to peaks in GSR level (Wang, et al., 2004). Thestudy participants reported that the inclusion of arousal/valence information made thecommunication feel more engaging and that it was preferred over traditional text-onlychat, although some users indicated that they would not always want their partner tobe aware of their arousal level (Wang, et al., 2004). 21

3.4 ConclusionThis chapter has separated the major components of emotional instant messaging intotwo categories: input techniques and output techniques. Among input techniques,inference from textual cues, inference through automated facial expressionrecognition, inference from physiologic data, and manual selection have beenreviewed. Output techniques that were discussed include emoticons, expressiveavatars, haptic devices, and kinetic typography.The next chapter introduces the Epoc headset and describes some of its capabilities.This headset is used as the emotional input device for the EmoChat system discussedin a subsequent chapter, and can be thought of as using automated facial expressionrecognition in combination with physiologic data to infer and convey emotion. Thenext chapter also presents a study that investigates the validity of the Epoc affectclassifier. 22

4 Study 1: Validating the Emotiv Epoc Headset4.1 IntroductionThis study investigates the validity of the Epoc headset in terms of how accurately itmeasures levels of excitement, engagement, and frustration. Self-reported measuresof excitement, engagement, and frustration are collected after games of Tetris areplayed at varied difficulty levels. The self-reported measures are compared with datafrom the headset to look for relationships.4.2 Overview of the Epoc HeadsetThe EmoChat application makes use of the Epoc headset for measuring affective stateand facial expression information. This headset, developed by Emotiv, was one ofthe first consumer-targeted BCI devices to become commercially available.Alternatives BCI devices that were considered include the Neurosky Mindset, and theOCZ Neural Impulse Actuator. The Epoc was selected because of the comparativelylarge number of electrodes (14) that it uses to sense electroencephalograph (EEG),and electromyography (EMG) signals, and the resulting capabilities. Additionally,the Epoc has a growing community of active developers who form a support networkfor other people using the software development kit to integrate headset capabilitieswith custom applications.Traditional EEG devices require the use of a conductive paste in order to reduceelectrical impedance and improve conductivity between the electrode and the scalp. 23

The Epoc device, however, replaces this conductive paste with saline-moistened feltpads, which reduces set up time and makes clean up much easier.A software development kit provides an application programming interface to allowintegration with homegrown applications, and a utility called EmoKey can be used toassociate any detection with any series of keystrokes for integration with legacyapplications. The developers have implemented three separate detection “suites”which monitor physiologic signals in different ways, and are reviewed below.4.2.1 Expressiv SuiteThis suite monitors EMG activity to detect facial expressions including left/rightwinks, blinks, brow furrowing/raising, left/right eye movement, jaw clenching,left/right smirks, smiles, and laughter. The detection sensitivity can be modifiedindependently for each feature and for different users. Universal detection signaturesare included for each feature, but signatures can also be trained to increase accuracy. Lower Face Movements Upper Face Movements Eye Movements Smirk Right Brow Raise Look Left Smirk Left Brow Furrow Look Right Smile Wink Right Laugh Wink Left Jaw Clench Blink Table 4.1 Facial expression features measured by the Epoc headset 24

4.2.2 Affectiv SuiteThe Affectiv suite monitors levels of basic affective states including instantaneousexcitement, average excitement, engagement, frustration, and meditation. Detectionalgorithms for each state are proprietary and have not been released to the public,therefore the given labels may be somewhat arbitrary, and may or may not accuratelyreflect affective state. The goal of the present study is to determine the accuracy ofthese detections with a longer-term goal of investigating whether this information canbe used to augment the instant messaging experience through the presentation ofemotional content.4.2.3 Cognitiv SuiteThis suite allows a user to train the software to recognize an arbitrary pattern ofelectrical activity measured by EEG/EMG that is associated with a specific,repeatable thought or visualization. The user may then reproduce this specific patternto act as the trigger for a binary switch. Skilled users may train and be monitored forup to 4 different thought patterns at once.4.3 The Need for ValidationThe Epoc affectiv suite purports to measure levels of excitement, engagement,frustration, and meditation; however, the algorithms used to infer these states areproprietary and closed-source. There has been little research that references the Epocheadset, perhaps because it is still new and relatively unknown. No studies thus far 25

have evaluated the accuracy of its affective inference algorithms. Cambpell andcolleagues used raw EEG data from the headset as input to a P300-based selectionengine (Campbell, et al., 2010), and several others have reviewed the device (Andrei,2010; Sherstyuk, Vincent, & Treskunov, 2009). Methods are provided to retrievewhat each affectiv suite score is at any given moment, but do not let one see how eachscore is calculated. It is understandable that Emotiv has chosen to keep this part oftheir intellectual property out of the public domain, but if these affectiv measurementsare to be used in any serious capacity by researchers or developers, evidence shouldbe provided to support the claim that reported affectiv suite excitement levels arereasonable estimates of actual subject excitement levels, that affectiv suiteengagement levels are reasonable estimates of actual subject engagement levels, andso on.4.4 Experimental DesignA study was designed to determine the accuracy of the Epoc affectiv suite bypresenting study participants with stimuli intended to elicit different levels ofaffective and physiologic responses (in the form of game play at varied levels), andmeasuring for correlation between output from the headset and self-reported affectiveexperience. Since the overall goal of this thesis work is to investigate how theinclusion of affective information enriches instant message communication, theexcitement, engagement, and frustration headset detections are validated here. It isthought that they are the most applicable to a study of affective communication. 26

The study design used in the present study is adapted from similar work by Chaneland colleagues, during which physiological metrics were monitored as participantsplayed a Tetris game (Chanel, Rebetez, Betrancourt, & Pun, 2008). The difficultylevel of the game was manipulated in order to elicit differing affective states. Self-report methods were also used to collect data about participants’ subjectiveexperience of affect. The goal of the study was to use these physiological metricswith machine learning techniques to classify affective experience into threecategories, including, anxiety, engagement, and boredom. It is thought that theanxiety category of the Chanel study may be a close analog to the frustrationcomponent of the Epoc affectiv suite.4.4.1 TetrisClone System DevelopmentA small Tetris application (TetrisClone) was developed to automate the experimentand to aid with data collection. The application was written in C# using MicrosoftVisual Studio 2008 and interfaces with the Emotiv Epoc headset through the suppliedAPI.The initialization screen for TetrisClone can be seen in fig. 4.1. This screen collectsthe test subject’s name and is used to start logging data coming from the Epocheadset. After logging begins, the right panel can be hidden so that it does notdistract the subject during the experiment. A screenshot of the TetrisClone applicationduring one of the trials is presented in fig. 4.2. 27

Fi igure 4.1 Initiali ization screen fo the TetrisClon application or ne Figure 4.2 TetrisClon application e ne during trials g4.4.2 Meas sures4.4.2.1 Que estionnairesAll participan complete a total of 3 surveys to cA nts e collect basic demographi icin nformation, self-reported levels of af s d ffect during the experim ment, and ope en-endedop pinions abou the causes of affect du ut s play. These questionnaires are uring game ppr rovided in ap ppendices A-C. Demog A graphics ques stions asked about age, g d gender,fa amiliarity wi Tetris, an skill at co ith nd omputer/cons sole games. Self-reporte levels of edaf ffect questio asked sub ons bjects to rate their exper e riences of ex xcitement, en ngagement,an frustratio on a 5 point likert-scale between t nd on trial games. Open ended questions das sked subject to describe what game events mad them feel excited, eng ts e e de gaged, andfr rustrated. 28

4.4.2.2 Hea adset DataThe TetrisClo applicati receives affective sta informati from the EpocT one ion ate ion eheadset at app proximately 2 Hz. and writes these t a hidden t box cont (visible w to text trolin the left pan of fig. 4.1). As a part n ne ticipant comp pletes one portion of the experiment e tan moves on to the next, the content of the text box control are output t a CSV file nd n ts t l to e.In this way, it becomes ea n t asier to deter rmine which groupings o headset d records h of dataar associated with which points in th experimen re d h he nt. Figure 4.3 Example ou 4 utput from the T TetrisClone app plicationThe output CSV files themT mselves con ntain time-sta amped, head dset-reported levels of dex xcitement (s short term), excitement (long term), e e engagement frustration, and t,meditation, however the present study is only conm p y ncerned with excitement (short term h t m),en ngagement, and frustrati compone ion ents. 29

4.5 Experimental ProceduresA total of (7) subjects participated in the experiment; however, data from one subjectwas incomplete due to problems with maintaining a consistently good signal qualityfrom the headset. This incomplete data is not included in the analysis. Theremaining subjects, (N=6), were all male aged 25-30 (mean=28.5), pretty to veryfamiliar with the Tetris game (mode=very familiar), occasionally to very frequentlyplayed computer or console games (mode=very frequently), but never to rarely playedTetris (mode=rarely). Participants rated themselves as being average to far aboveaverage skill (mode=above average) when it came to playing computer or consolegames.Subjects arrived, were seated in front of a laptop, asked to review and sign consentforms, and then completed the Demographics Questionnaire (Appendix A). Thesubjects were then fitted with the Epoc headset. Care was taken to ensure that eachheadset sensor reported strong contact quality in the Control Panel software. Theself-scaling feature of the Epoc headset required 15-20 minutes prior to datacollection. During this time, the subjects were asked to play several leisurely gamesof Tetris as a warm-up activity.Once the headset adjusted to the subjects, the Tetris game/Headset data recorder waslaunched. Subjects played through a series of three Tetris games to determine “skilllevel,” as calculated by averaging the highest levels reached during each game. TheTetrisClone application has no maximum level cap, although levels above 10 are so 30

difficult that progressing further is not practical. After each game the subjects restedfor 45 seconds to allow any heightened emotional states to return to baseline.Once skill level was determined, three experimental conditions were calculatedautomatically as follows: High Difficulty Level = Skill Level (+ 2) Med Difficulty Level = Skill Level Low Difficulty Level = Skill Level (– 2)The subjects then played through a random ordered set of 6 trials consisting of 2games at each difficulty level, e.g., [8,6,6,4,8,4]. Trials were randomized to accountfor order effects. During each trial, games lasted for 3 minutes at a constantdifficulty/speed. If the subjects reached the typical game over scenario, the playingfield was immediately reset and the subjects continued playing until the 3 minuteswere over. At the end of each round, the subjects complete a portion of theExperiment Questionnaire (Appendix B). The subjects rested for 45 seconds aftercompleting the questionnaire, but before beginning the next round, to allow emotionalstate to return to baseline. Headset logging stopped after all 6 rounds had beenplayed. 31

After the subjects finished all game play tasks the facilitator removed the headset, andsubjects completed the final Post-Experiment Questionnaire (Appendix C). Thesubject was paid for his time, signed a receipt, and was free to leave.4.6 Results and Analysis4.6.1 Headset-Reported versus Self-Reported Levels of AffectThe main goal for this study was to determine whether or not the Epoc headsetreports data that is congruent with self-reported data of the same features. This wasdone in order to establish the validity of the Epoc affectiv suite. Headset and self-report data were compared a number of different ways. Each trial yielded 3 minutesworth of headset data sampled at 2 Hz. (approximately 360 samples x 3 affectivefeatures per sample = 1080 individual data elements), which were to be comparedwith a single self-reported level of affect for each of the 3 affective features inquestion (excitement, engagement, and frustration). Headset data from each trial wasreduced to 3 individual data elements by taking the mean of all sampled values foreach of the three affective features. Headset means from each trial were then pairedwith the corresponding self-reported levels of affect for that trial. These data arereproduced in the table below. 32

Subject Trial Condition H_exc H_eng H_fru S_exc S_eng S_fruSubject_1 1 low 0.283975 0.488978 0.347653 2 2 1Subject_1 2 high 0.315356 0.559145 0.400994 4 5 3Subject_1 3 med 0.316626 0.52268 0.430862 5 4 4Subject_1 4 low 0.317267 0.573343 0.396643 4 5 1Subject_1 5 med 0.389669 0.543603 0.371616 5 5 2Subject_1 6 high 0.305706 0.596423 0.454396 5 5 3Subject_2 1 high 0.301968 0.654293 0.580068 3 3 3Subject_2 2 med 0.384194 0.660933 0.785552 3 3 3Subject_2 3 high 0.323292 0.559863 0.368594 3 4 4Subject_2 4 med 0.271679 0.505589 0.336272 2 3 2Subject_2 5 low 0.289168 0.604071 0.505809 2 2 2Subject_2 6 low 0.304198 0.530313 0.37589 3 2 2Subject_3 1 high 0.302396 0.406588 0.34533 3 3 3Subject_3 2 med 0.279739 0.409497 0.356766 3 3 2Subject_3 3 low 0.353323 0.402969 0.413634 2 2 1Subject_3 4 low 0.311233 0.425248 0.360592 2 3 1Subject_3 5 med 0.365323 0.427358 0.429031 3 4 2Subject_3 6 high 0.361752 0.543518 0.380738 4 4 4Subject_4 1 low 0.233371 0.557115 0.391817 2 3 1Subject_4 2 high 0.334173 0.485377 0.448109 3 4 4Subject_4 3 med 0.265334 0.532051 0.463684 2 2 3Subject_4 4 med 0.416719 0.519389 0.522566 2 2 1Subject_4 5 low 0.281803 0.445416 0.424542 2 3 1Subject_4 6 high 0.305135 0.508839 0.474708 3 3 3Subject_5 1 low 0.27187 0.597617 0.362354 3 4 2Subject_5 2 med 0.256403 0.744225 0.361584 4 4 2Subject_5 3 high 0.292769 0.659934 0.377663 4 5 3Subject_5 4 med 0.307077 0.650888 0.533751 4 5 1Subject_5 5 high 0.256848 0.710124 0.460227 2 2 2Subject_5 6 low 0.24097 0.594766 0.555613 2 4 1Subject_6 1 high 0.250623 0.563784 0.455011 3 4 3Subject_6 2 low 0.271444 0.59069 0.452902 5 5 1Subject_6 3 high 0.282375 0.55344 0.458738 2 3 4Subject_6 4 med 0.257751 0.573329 0.482535 4 4 1Subject_6 5 med 0.235875 0.558884 0.460652 3 4 2Subject_6 6 low 0.305082 0.622437 0.490638 4 4 1Table 4.2 Headset and self-reported levels of affect per subject, per trial 33

The non-parametric Spearman’s rho was selected as the correlation metric fordetermining statistical dependence between headset and self-reported levels of affectbecause of the mixed ordinal-interval data. The resulting correlation coefficients andsignificances were calculated with SPSS and are presented in the table below (N=36). Correlation Pair Coefficient Sig. (2-tailed) Excitement 0.261 0.125 Engagement .361 0.03 Frustration -0.033 0.849 Table 4.3 Spearman correlation between headset and self-reported levels of affect (N=36)This analysis suggests that of the three features of affect that were examined,engagement appears to significantly correlate (p=.03) between what is reported by theheadset and what is experienced by the subject. No significant correlation ofexcitement or frustration between headset and self-reported affect levels was found.The levels of headset reported excitement, engagement, and frustration presented intable 4.2 are average levels over entire 3 minute trials. It could be possible that self-reported affect levels collected after the trials might better relate to headset levelsaveraged over smaller subdivisions of the trial time. For example, a subject whoexperienced high levels of frustration during the last 15 seconds of game play, butlow levels of frustration at all other times may have self-reported a high level offrustration after the trial, even though the subject generally experienced low levels.To investigate whether headset data averaged over smaller subdivisions of trial timebetter correlated with self-reported data, headset data was averaged from time slices 34

of the last 15, 30, 60, and first 60 seconds of trial data. Spearman’s rho wascalculated for each new dataset by comparing with the same self-report data. Theseresults are provided in the table below (N=36), along with original correlation resultsfrom table 4.3. Correlation Pair Time Division Coefficient Sig. (2-tailed) Excitement all 3min 0.261 0.125 last 60s 0.133 0.439 last 30s 0.21 0.219 last 15s 0.174 0.31 first 60s 0.285 0.092 Engagement all 3min .361* 0.03 last 60s .340* 0.042 last 30s 0.291 0.085 last 15s 0.316 0.061 first 60s 0.229 0.179 Frustration all 3min -0.033 0.849 last 60s -0.102 0.554 last 30s -0.049 0.775 last 15s 0.005 0.977 first 60s 0.176 0.305 Table 4.4 Spearman correlation of headset and self-report data for varied time divisions (N=36)This analysis suggests that no new significant relationships between headset and self-report data are found when analyzing headset data from specific subdivisions of time(last 60, 30, 15 seconds, and first 60 seconds), however, it does appear that self-reported excitement and frustration levels correlate best with averaged headset datafrom the first 60s of each trial.The data were further analyzed by calculating grand means for each difficultycondition, and for both headset and self-reported levels of affect. Grand means arepresented in the table below. 35

Condition H_exc H_eng H_fru S_exc S_eng S_fru low 0.29 0.54 0.42 2.75 3.25 1.25 med 0.31 0.55 0.46 3.33 3.58 2.08 high 0.30 0.57 0.43 3.25 3.75 3.25 Table 4.5 Grand mean headset and self-reported levels of affect per difficulty levelSpearman’s rho was again used as the metric for determining statistical dependencebetween headset and self-reported affective levels of the grand mean data. Resultingcorrelation coefficients and significances were calculated with SPSS and arepresented in the table below (N=3). Correlation Pair Coefficient Sig. (2-tailed) Excitement 1.00 0.01 Engagement 1.00 0.01 Frustration 0.50 0.667 Table 4.6 Grand mean Spearman correlation between headset and self-reported levels of affect (N=3)This analysis suggests very high correlation of excitement and engagement betweenheadset and self-reported levels of affect (p=.01), however these results should beinterpreted with caution, considering that this is a correlation between means and thenumber of values being compared is small (N=3). No significant correlation offrustration between headset and self-report levels of affect were found.To visualize the relationship between grand means of headset and self-reported affectfeatures, line charts are presented below. 36

Fi igure 4.4 Comp parison of grand mean headset and self-repor t rted levels of ex xcitementFi igure 4.5 Comp parison of grand mean headset and self-repor t rted levels of en ngagementFi igure 4.6 Comp parison of grand mean headset and self-repor t rted levels of fru ustration 37

The significant correlation (p=.01) between headset and self-report levels ofexcitement and engagement are apparent in fig. 4.4 and fig. 4.5. Frustration is seen tocorrelate moderately well from low to medium difficulty conditions; however thedegree and direction of change from medium to high difficulty conditions are clearlyin disagreement. As noted above, cautious interpretation of figures 4.4-4.6 is prudentconsidering the use of grand means and small number of data points, but theemergence of relationships between headset and self-reported levels of excitementand engagement are suggested.4.6.2 Subjective Causes of Affect during GameplayAfter the game play portion of the experiment concluded, subjects were asked tocomplete a brief survey to collect their subjective opinions about what caused theirexperiences of excitement, engagement, and frustration during game play. Participantresponses were analyzed to identify general themes. The data were coded for thesethemes, which have been aggregated and are presented in the table below. 38

Q1. What types of events caused you to get excited during game play?Extracted Themes Participant Responses (S3) “Doing well”Competent game (S1) “Clearing many lines at once”performance (S1) “Seeing a particularly good move open up” (S5) “Speed increase of block decent”Game speed/difficulty (S2) “When blocks got faster” (S4) “Game speed” (S5) “Block failing to land where desired” (S3) “A poor move”Poor game performance (S4) “A mistake during otherwise good game play” (S6) “End of game when doing bad” (S2) “When the blocks would get too high” (S3) “A good sequence of pieces”Positively perceived game (S1) “Getting pieces I was hoping for”variables (S6) “Seeing blocks I needed to make 4 rows”Q2. What made the game engaging?Extracted Themes Participant Responses (S2) “The intensity”Game speed/difficulty (S1) “Speed increases” (S4) “You have to think fast” (S6) “Have to think about what’s happening on board”Cognitive load (S1) “Anticipating future moves” (S1) “Seeing game board get more full” (S3) “Small learning curve in general”Game simplicity (S1) “Simplicity” (S4) “Few input options, enough to stay interesting”Q3. What happened during the game that made you feel frustrated?Extracted Themes Participant Responses (S5) “Too many of same block in a row” (S3) “Bad sequence of blocks”Negatively perceived game (S1) “Getting the same piece over and over”variables (S6) “Not getting the blocks I needed” (S5) “Block did not fit pattern I was constructing” (S1) “Getting undesired pieces” (S5) “When a block would land out of place”Poor game performance (S3) “Poor move” (S3) “Game moving too quickly”Game speed/difficulty (S4) “Game speed increase”Table 4.7 Major themes identified in subjective affect survey results 39

4.7 Discussion4.7.1 Consistency in the Present StudyThe main goal of this study was to determine how accurately the Epoc headsetmeasures levels of excitement, engagement, and frustration in order to establish thevalidity of the Epoc affectiv suite. To this end, the TetrisClone application wasdeveloped and used to log headset output while subjects played games of Tetris atvaried difficulty levels. During the study, subjects were asked to self-report howexcited, engaged, and frustrated they felt for each game that they played.The responses to the self-report excitement, engagement, and frustration questionswere then statistically compared with the output from the headset. This analysissuggested that self-reported levels of engagement correlated well with levels reportedby the headset. To a lesser degree, the analysis suggested that excitement levelsmeasured by the headset correlated fairly well with self-reported levels. Frustrationlevels measured by the headset, however, did not appear to correlate with self-reported levels.Subjective responses about what made the game engaging seem to corroborate theself-report and headset data. General trends in the data described engagement asincreasing over low, medium, and high difficulty levels. The two main themesidentified in responses to “what makes the game engaging,” were gamespeed/difficulty and cognitive load. As level increases, game speed inherently alsoincreases. It makes sense that increased difficulty of the game should demand greater 40

concentration, more planning, and more efficient decision making—all suggestive ofincreased cognitive load. With respect to existing literature, the Chanel study, onwhich the experimental design of the present study was based, found a similar upwardlinear trend in participant arousal, however, the relationship between engagement andarousal has not been established.Excitement trends in self-report and headset data generally showed an increase fromlow to medium difficulty, then a slight decrease in the high difficulty condition.Responses to the question, “what types of events caused you to get excited duringgame play,” support this trend. General themes extracted from responses to thequestion include competent game performance, game speed/difficulty, poor gameperformance, and positively perceived game variables (such as getting a block typeyou were hoping for). Game speed increases as difficulty condition increases, so itscontribution to overall excitement level is always present in quantities that increasewith difficulty level. It might be assumed that competent game performance and poorgame performance have a balancing effect on one another, i.e., when one increasesthe other decreases, thereby creating a single contributing factor to excitement that isalways present, and arguably stable. The decisive contributor to excitement levelmay be the positively perceived game variables. It seems feasible that game variablesthat happen to be in the player’s favor should occur at a similar frequency, regardlessof difficulty level. It may also be feasible that these occurrences are less noticed inhigher difficulty levels due to increased cognitive load; a mind occupied by gametasks of greater importance. This lack of recognition of positive game variables may 41

be the reason that excitement increases from low to medium difficulty conditions, butthen decreases in the high condition. A similar trend reported by the Chanel studyoccurs in the valence dimension. Valence is shown to increase from low to mediumdifficulty conditions, then decrease in the high condition, although the relationshipbetween valence and excitement has not been established.4.7.2 Future DirectionIt might be beneficial to take a more granular approach to validating output from theEpoc headset to determine whether specific game events, e.g., a mistake or an optimalplacement, influence affect levels reported by the headset. The present study onlylooked at average headset output over large spans of time, but there is a great deal ofvariability in the data, some of which might have a relationship with game events.This more granular approach would require the ability to record specific game events,e.g., clearing a line, and cross referencing with data from the headset. This could beaccomplished by recording video of the game play. It might also yield interestingresults if headset data were tested for any correlation with other known physiologicalmeasures of affect such as GSR, or skin temperature.4.8 ConclusionWith the accuracy of at least some of the headset-reported levels of affect established,an instant message application was developed that uses output from the headset tocontrol a simple animated avatar and display data from the affectiv suite during 42

messaging sessions. This application is called EmoChat. In the next chapter, a studyinvestigates how EmoChat can be used to enrich emotional content during IMsessions. 43

5 Study 2: Emotional Instant Messaging with EmoChat5.1 IntroductionExisting instant messaging environments generally fail to capture non-verbal cuesthat greatly increase information transfer bandwidth during face-to-facecommunication. The present study is a step toward investigating the use of a low-cost EEG device as a means to incorporate lost non-verbal information duringcomputer-mediated communication.An Instant Messaging application (EmoChat) has been developed that integrates withthe Emotiv Epoc headset to capture facial movements that are used to animate anexpressive avatar. Output from the headset is also used to convey basic affectivestates of the user (levels of excitement, engagement, and frustration).The present study examines how emotional information is transferred differentlybetween users of the EmoChat application and a “traditional” instant messagingenvironment in terms of emotionality. This study addresses the following researchquestions: 1. Does the system facilitate communication that generally contains more emotional information? 2. Does the system provide a greater degree of richness (as defined in section 5.3.1)? 3. Is the emotional state of participants more accurately conveyed/interpreted? 4. How usable is a system that implements this technology? 44

5.2 EmoC5 Chat System Development m5.2.1 Over rviewEmoChat is a client/serve application that facilitE er n tates the exch hange of em motionalin nformation during instan message communicati d nt ion. The app plication was developed swith C# in Microsoft Visu Studio 2008.w M ualTraditional inT nstant messaging environ nments typic cally rely on manually ge eneratedem moticons in order to sha the emoti ape ional meanin of a mess ng sage. EmoC Chatin ntroduces a novel way to capture and convey em n o d motional mea aning by inte egrating with hth Emotiv Ep headset, a low cost, commercial available EEG device that is he poc , lly e eca apable of inf ferring facia expression and basic a ffective info al n ormation from raw EEG mdata.Fi igure 5.1 The EmoChat client application E t 45

Facial expression information that is captured by the Epoc headset is passed toEmoChat and used to animate a simple avatar with brow, eye, and mouth movements.Affective information captured by the headset is used to modify the value of a seriesof progress bar style widgets. The previous validation study suggested thatexcitement and engagement are reasonably estimated by the headset, however it wasdecided that other affective measures from the headset (frustration and meditation),would also be presented in the EmoChat application to give the users a chance todecide for themselves whether or not these measures are of any value.Although the application has been specifically designed to integrate with the Epocheadset, a headset is not required. All facial movements and affective levels may bemanually manipulated by the user at a very granular level, i.e., users may overridebrow control, but leave eye, mouth, and affect control determined by the headset.Manual override of facial and affect control is permitted whether or not a headset isbeing used. A summary of the facial movements and affective information conveyedby EmoChat is presented below:Eyebrow Eyes Mouth AffectStrong raise Blink Laugh ExcitementWeak raise Left wink Strong smile Average excitementNeutral Right wink Weak smile EngagementWeak furrow Neutral Neutral FrustrationStrong furrow Left look Clench (frown) Meditation Right lookTable 5.1 Facial movements and affective information used by EmoChat 46

5.2.2 Trad ditional Env vironmentA “traditional instant me l” essaging app plication was approxima by remo s ated oving avataran affect me nd eters from th original Em he moChat app plication, so t subjects would be that sre equired to co onvey emotio informa onal ation strictly through tex This trim y xt. mmed-downIM environment presents subjects wit standard c M th chat input an output pa nd anes.5.2.3 Application Arc chitectureThe EmoChat application architecture follows the client/serv model. A serverT n e e verap pplication listens for con nnections fro networke client app om ed plications. A After aco onnection is established, the server monitors for data transm , m r missions. On data is ncere eceived, the server retran nsmits the da to all oth connected clients. Th clients an ata her d he ndse erver may re eside on one or more com mputers netw worked acros a LAN or WAN. Thi ss r isco onfiguration allows the server to log all commun n s g nication even for offlin analysis nts newithout requiw iring any ext effort by client users. tra .Fi igure 5.2 EmoC Chat server app plication 47

EmoChat introduces a novel way to share emotional information during instantmessage communication by integrating with one of the first commercially availablebrain-computer interface devices. This application not only demonstrates a new andinteresting way to enrich computer-mediated communication, but also serves anevaluation tool for the Epoc EEG headset and its inference algorithms. The resultsfrom studies performed with EmoChat may contribute toward a foundation for futureresearch into BCI applications.5.3 Experimental DesignThe present study follows a crossover-repeated measures design (within-subjects).Paired subjects spend time chatting with one another using both the EmoChatapplication (HE condition), and a “traditional” instant messaging environment (TEcondition). The order in which subjects are presented with these environments isdetermined by random assignment to one of two groups. Group Condition Order I TE then HE II HE then TE Table 5.2 EmoChat experimental groupsAdditionally, roles of expresser (EX) and perceiver (PX) are divided between eachsubject pair. These roles are used to determine which version of the EmotionalTransfer Accuracy Questionnaire (ETAQ) is completed (discussed below). EX andPX roles are swapped by paired subjects between each experimental condition. 48

During each chat session, participants are asked to chat for 15 minutes about anytopic(s) of interest. Discussion guides for eliciting emotional responses wereconsidered but decided against. It was thought that since this was a repeatedmeasures design, two separate topic lists would need to be generated; one for eachexperimental condition. These two topic lists would need to elicit similar emotionalresponses in order to generate data that were comparable. The challenges posed bycreating two equivalent topic lists led to the decision to eliminate them entirely and toallow subjects to guide their own conversations. It was also thought that unguidedconversation would lead to a more natural exchange.5.3.1 Measures5.3.1.1 QuestionnairesSubjects complete a number of surveys during the experiment. Basic demographicinformation and responses to the International Positive Affect Negative AffectSchedule Short Form (I-PANAS-SF) are collected prior to the experiment. Afterparticipating in each condition, subjects complete another battery of questionnaires,including the Emotional Transfer Accuracy Questionnaire (ETAQ), and Richness ofExperience Questionnaire (REQ). After the HE condition subjects complete theSystem Usability Scale (SUS). Descriptions of each survey, and sources, whenapplicable, are presented below. Questionnaires are provided in appendices D-F,where permitted by copyright. 49

I-PANAS-SFPrior to the experiment, each subject is asked to complete Thompson’s InternationalPositive Affect Negative Affect Schedule Short Form (I-PANAS-SF) (Thompson,2007), a shortened, but psychometrically sound version of the original PANASdeveloped by Watson and colleagues (Watson, Clark, & Tellegen, 1988), in order toassess prevailing emotional state at the time of the experiment.ETAQSubjects from both experimental conditions are asked to complete a novelquestionnaire designed to assess how accurately emotional information was conveyedand perceived during the condition (Appendix E). Wording of the questions isslightly altered, depending on subject role. Subjects are asked to indicate, “how oftenduring this chat session [EX role: you / PX role: your partner] experienced thefollowing emotions,” and are presented with a list of 16 emotional adjectives; 4representative adjectives from each quadrant of the arousal/valence space defined byRussell’s circumplex model of affect (Russell, 1980). Subjects are asked to ratefrequency of experience on a five-point likert-scale from never/very seldom to veryoften (see Appendix D for the complete version of the emotional transfer accuracyquestionnaire).REQThe concept of “richness” has been applied in related studies of instant messagecommunication involving expressive avatars, and has been used to compare IM 50

applications having expressive avatar capabilities with those that do not (Fabri, 2006;Neviarouskaya, 2008). Fabri defines a high degree of “richness” as manifestingthrough greater task involvement, greater enjoyment, a higher sense of presence, anda higher sense of copresence (Fabri, 2006), while Neviarouskaya defines “richness”in terms of interactivity, involvement, copresence, enjoyment, affective intelligence,and overall satisfaction (Neviarouskaya, 2008). For the purposes of the present study,a combination of richness components from both Fabri and Neviarouskaya areadapted and measured, and include: task enjoyment, copresence, affectiveintelligence, and overall satisfaction. Questions related to each richness componentare presented and responses are collected on a 5-point symmetric likert-scale rangingfrom strongly disagree to strongly agree (See Appendix F for complete version ofrichness questionnaire). Subjects from both experimental conditions are asked tocomplete this richness questionnaire.SUSThe usability of any new system is an important measure that should be used toinform future design revisions. Although the usability of the prototype EmoChatsystem is expected to be lower than that of an established traditional instantmessaging environment, it is nonetheless a very worthwhile measure. The presentstudy implements Brooke’s “Quick and Dirty” system usability scale (SUS) to obtaina subjective account of usability for the HE condition only (Brooke, 1996) (seeAppendix F for a complete version of the SUS questionnaire). 51

5.3.1.2 Chat TranscriptsLogs of chat sessions are collected by the EmoChat server for offline analysis. Theselogs include time-stamped messages that are exchanged between participants and inthe HE condition, time-stamped facial expression and affect information that is eitherautomatically generated by the Epoc headset, or manually generated by usermanipulation of the EmoChat override controls.Of particular interest to the present study is the number of affect, and affect relatedterms that are used during each experimental condition. To facilitate the recognitionand frequency analysis of these specific terms, the Linguistic Inquiry and WordCount tool developed by Pennebaker and colleagues is used (Pennebaker, Booth, &Francis, 2007). This tool was created to aid in the measurement of specific types ofwords representing 74 different linguistic dimensions, and will be used to analyzechat logs from both experimental conditions.5.3.1.3 Unstructured InterviewsAfter subjects have completed the experiment, they join the facilitator as a pair for ashort informal interview to collect opinions about each experimental condition.Discussion topics and subjective experiences are documented for each pair ofsubjects. The data collected during the interviews is used to supplement othermeasures taken during the study. 52

5.4 Experimental SetupTwo laptop computers (Apple MacBook and IBM ThinkPad X41) were located in labareas separated by a closed door. The computers were both running Windows XPand connected to a private network via Ethernet cable and a Linksys WRT160Nrouter. An external standard-sized mouse was provided so that subjects would nothave to interact with the computers by using the built-in track pad (MacBook) orisometric joystick (ThinkPad). A wireless dongle for communicating with the Epocheadset was connected to the MacBook.5.5 Experimental ProceduresSubjects recruited for the study (N=10) were aged 22 to 32 years (mean=28.1).Gender of subjects was divided with 7 males and 3 females participating. Allsubjects reported having an intermediate or higher level of computer experience(mode=intermediate) and between basic and expert typing skills(mode=intermediate). Subjects’ highest level of education ranged from high schoolto bachelor’s degree (mode=bachelor’s degree). Reported use of instant messageapplications varied from never to every day (mode=every day).All subjects scored extremely low on the negative affect component of the PANAS-SF (mean=5.6 out of 25) and scored between 10 and 22 on the positive affectcomponent (mean=16.4 out of 25). 53

Subjects who had previously participated in the Epoc validation study (N=5) werepaired with subjects who had no prior experience with the headset. Paired subjectswere randomly assigned to one of the two experimental groups described in section(5.3). 6 subjects were assigned to experimental group I and 4 subjects were assignedto experimental group II.Subject pairs arrived and were asked to review and sign consent forms. One subject ineach pair had participated in the previous Epoc validation study. Subjects werebriefed on the tasks they were asked to complete, including participating in two 15minute chat sessions, and filling out several questionnaires. The pre-experimentquestionnaire was administered, which included basic demographic information andthe I-PANAS-SF (International Positive Affect Negative Affect Schedule ShortForm). Prior to the experiment, subject pairs were randomly assigned to one of theexperimental groups described above.Subjects were separated in their respective areas of the lab, and the facilitatorinitialized the chat application corresponding to the first experimental condition. Inthe (HE) condition, the facilitator fitted the (EX) subject with the Epoc headset, andthen briefly explained the functions of the EmoChat application independently to bothsubjects. The facilitator then asked the subjects to begin chatting, and left the room.After 15 minutes had elapsed, the facilitator informed the subjects that the first chatsession had concluded, and administered a battery of questionnaires containing theEmotional Transfer Accuracy Questionnaire (ETAQ) and the Richness of Experience 54

Questionnaire (REQ). In the (HE) condition, a third System Usability Scale (SUS)questionnaire was also administered. While the questionnaires were being completed,the facilitator initialized the chat application corresponding to the secondexperimental condition. Once both subjects completed their respectivequestionnaires, the facilitator asked the subjects to being the second 15 minute chatsession and left the room. At the conclusion of the second chat session, subjects wereasked to complete another battery of questionnaires including the ETAQ, theRichness of Experience Questionnaire, and in the (HE) condition, the SUS.After both subjects completed this final battery of questionnaires they joined thefacilitator for a short informal interview about their experiences with each chatapplication. Questions were asked, such as, “What did you think of each chatprogram,” “how often did you manually change the avatar,” and “which program didyou enjoy more,” however the interview was open ended and frequently led toparticipants discussing a very wide range of topics.5.6 Results and Analysis5.6.1 Emotional Transfer AccuracyThe results from the ETAQ were used to identify differences in emotional transferaccuracy between conditions. Subject pairs chatted with each other during bothexperimental conditions: traditional environment (TE) and the EmoChat environment(HE). During each condition, one subject was assigned the expresser role (EX), and 55

one subject was assigned the perceiver role (PX). After each chat session concluded,the subject with the EX role answered questions on the ETAQ about him/herself, e.g.,“How often did YOU experience the following emotions…,” while the subject withthe PX role answered questions about his/her partner, e.g., “How often did YOURPARTNER experience the following emotions.” This procedure yielded matched setsof responses for each experimental condition.The absolute value of the difference between the rank of each matched response wascalculated for each item on the ETAQ and then summed to give a total scoreindicating how accurate emotional transfer was for that condition. A score of 0indicates perfect transfer accuracy, i.e., EX and PX roles selected exactly the sameresponses for every item on the questionnaire. The ETAQ scores for each subjectpair are presented below. ETAQ Score (TE) ETAQ Score (HE) Subject-Pair 1 8 5 Subject-Pair 2 20 11 Subject-Pair 3 7 10 Subject-Pair 4 9 11 Subject-Pair 5 5 8 Mean 9.8 9 Table 5.3 ETAQ scores for each subject-pair and both experimental conditionsTo determine whether the mean scores for each condition were statistically different,the Wilcoxon signed rank test was used to calculate a significance value. Resultsfrom the test indicate that emotional transfer accuracy was not significantly differentbetween conditions (p=.891). 56

5.6.1.1 Effects of Avatar Features on Perception of AffectAdditional analysis was performed in the HE condition by testing for correlationbetween avatar features of EX role with perception of affect in the PX role (measuredby the ETAQ). This was a way to see how changes in the EmoChat affectiv metersand avatar expressions influenced the perception of emotions by the chat partner.Headset data recorded during the chat session were used to calculate average levels ofexcitement, frustration, engagement, and meditation displayed by the affect meters.Headset data were also used to calculate the proportion of time that avatars displayedmouth and brow facial expression features, e.g., smile, neutral mouth, raised brow,etc. Data calculated from the headset were tested for correlation with the non-headsetpartner’s responses on the ETAQ, e.g., “how often was your partner annoyed,” scoredon a 5 point likert-scale from seldom to very often. A summary of salientrelationships is provided below, and an abridged correlation matrix supporting theserelationships follows in table 5.4.Strong Correlation (r2 ≥ .95) 1. Participants with avatars displaying a lower ratio of smiles were perceived as annoyed more often. 2. Participants with avatars displaying higher average engagement levels were perceived as annoyed more often. 57

Moderate Correlation (r2 ≥ .80) 1. Participants with avatars displaying higher average excitement levels were perceived as pleased more often. 2. Participants with avatars displaying higher average frustration levels were perceived as satisfied more often. Affect Meters Lower Face (Mouth) Upper Face (Brow) eng excLT excST med frus clench neutral smile laugh furrow neutral raiseangry .866 .866 .577 -.866 .000 .289 .866 -.866 -.289 .866 -.289 -.577r-squared .750 .750 .333 .750 .000 .083 .750 .750 .083 .750 .083 .333annoyed .975 .564 .205 -.718 .359 .462 .821 -.975 -.154 .718 -.103 -.872r-squared .950 .318 .042 .516 .129 .213 .674 .950 .024 .516 .011 .761frustrated -.051 .154 -.103 -.667 .103 .667 -.154 .051 -.462 -.154 .667 -.205r-squared .003 .024 .011 .445 .011 .445 .024 .003 .213 .024 .445 .042bored .289 -.289 -.289 .577 .577 .000 .289 -.289 .000 .000 -.289 -.289r-squared .083 .083 .083 .333 .333 .000 .083 .083 .000 .000 .083 .083tired .289 .289 .289 .289 .289 .000 .577 -.289 -.577 .289 -.577 .000r-squared .083 .083 .083 .083 .083 .000 .333 .083 .333 .083 .333 .000interested .289 .577 .289 -.866 .000 .577 .289 -.289 -.577 .289 .289 -.289r-squared .083 .333 .083 .750 .000 .333 .083 .083 .333 .083 .083 .083astonished -.866 -.289 .000 .577 -.289 -.289 -.577 .866 -.289 -.577 .000 .866r-squared .750 .083 .000 .333 .083 .083 .333 .750 .083 .333 .000 .750excited -.211 .527 .527 -.580 -.580 .000 -.053 .211 -.316 .158 .053 .369r-squared .044 .278 .278 .336 .336 .000 .003 .044 .100 .025 .003 .136happy .000 .707 .707 -.354 -.354 .000 .354 .000 -.707 .354 -.354 .354r-squared .000 .500 .500 .125 .125 .000 .125 .000 .500 .125 .125 .125pleased .000 .738 .949 -.158 -.791 -.632 .369 .000 -.105 .580 -.791 .580r-squared .000 .544 .900 .025 .625 .400 .136 .000 .011 .336 .625 .336content -.369 .000 .158 .527 .053 -.105 .000 .369 -.632 -.211 -.316 .527r-squared .136 .000 .025 .278 .003 .011 .000 .136 .400 .044 .100 .278relaxed .205 -.051 .103 .616 .154 -.410 .359 -.205 .103 .205 -.667 .051r-squared .042 .003 .011 .379 .024 .168 .129 .042 .011 .042 .445 .003satisfied .224 -.224 -.447 .224 .894 .671 .224 -.224 -.671 -.224 .224 -.447r-squared .050 .050 .200 .050 .800 .450 .050 .050 .450 .050 .050 .200Table 5.4 Spearman correlation matrix between avatar features and perceived frequency of emotionalstates (N=5) 58

5.6.2 Richn of Exp ness perienceSubject respo onses to the Richness of Experience q R questionnair were used to re ddetermine wh hether “richn ness” was hig gher in the h headset cond dition than in the ntr raditional condition. Res sponses wer analyzed a number of different wa re ays. First,mean scores were calcula per condm w ated dition for eac question. A summary of these ch ysc cores compa ared between conditions is presented in the figur below. n d resFi igure 5.3 Mean scores from th richness ques n he stionnaire, ques stions 1-5 59

Fi igure 5.4 Mean scores from th richness ques n he stionnaire, ques stions 5-10It is apparent from the vis t sualization th mean sco for all b Q5 are hi hat ores but igher in theheadset envir ronment than in the traditional enviro n onment. To determine if thedifference in mean scores for each qu s uestion is sig gnificant, the Wilcoxon s e signed-rankte was perfo est ormed on the results. Th particular test for sign e his r nificance wa selected asin nstead of the paired t-test or ANOVA because th data do no have norm e A he ot maldistributions or equal vari o iances, and only two con o nditions are being compared. 60

TE-Mean HE-Mean Wilcoxon Sig. Q1 felt it was important to respond after each of my partners 4.1 4.2 - statements Q2 was interested in my partners 4.3 4.5 - responses Q3 enjoyed communicating with my 4.4 4.5 - partner Q4 had the sensation that my partner 3.9 4.3 - was aware of me Q5 felt as though I was sharing the 3.3 3.3 - same space with my partner Q6 had the sensation that my partner 4.1 4.2 - was responding to me Q7 was able to successfully convey 3.3 3.9 - my feelings with this application Q8 partner was able to successfully convey feelings with this 3 3.7 .05 application Q9 understood the emotions of my 2.8 3.6 .05 partner Q10 am satisfied with the experience of communication with this 3.7 4.1 - system Table 5.5 Wilcoxons signed rank test for significant difference between score means (N=10)This analysis suggests that responses to Q8 and Q9 are significantly different (p=.05)between experimental conditions. Means of the remaining questions from the survey,although typically higher in the HE condition, do not differ significantly betweenconditions.5.6.2.1 Differences between Headset and Non-Headset Users in EmoChatDuring the HE condition (when subjects were chatting with the EmoChat application)only one subject from each pair was wearing an Epoc headset. This provided anopportunity to identify differences between richness of experience when using aheadset with the application, which allows the automatic update of avatar expression 61

an affectiv meters, and when not usi a headse which req nd m w ing et, quires manua updating. alMean scores of REQ respM ponses can be visualized in the figur below. b d resFiigure 5.5 Comp parison of mean responses to REQ between su n R ubjects with he eadsets versus w without headsets s,in the EmoChat condition, Q1-5 (N=5) n 5Fiigure 5.6 Comp parison of mean responses to REQ between su n R ubjects with he eadsets versus w without headsets s,in the EmoChat condition, Q5-1 (N=5) n 10 62

The Wilcoxon signed-rank test for significant differences did not produce any p-values at or below .05 for this dataset, largely due to the small sample size (N=5);however, responses with the largest mean differences between user groups (Q1, Q8,and Q9) may still suggest meaningful relationships. Additional studies are needed todetermine any significance.Compared with the headset users, non-headset users typically felt that it was moreimportant to respond to each of their chat partners’ statements, and believed that theirpartners were more able to successfully convey feelings with the application. Non-headset users also typically indicated that they were better able to understand theemotions of their headset-wearing chat counterparts.5.6.3 Chat TranscriptsChat logs were analyzed by using the Linguistic Inquiry and Word Count (LIWC)tool developed by Pennebaker and colleagues (Pennebaker, et al., 2007). Thisapplication reports on the frequency of words belonging to specific linguisticcategories within the text that it processes, and has been used to analyze bothtranscribed spoken passages, and written text. Output from the application is entirelycontext-independent, meaning that the sentence “Quick brown foxes jump overdogs,” produces the exact same output as a nonsensical sentence using the samewords, like, “Brown over jump foxes dogs quick.” The present study was primarilyinterested in comparing the difference in number of affect-related terms that appear in 63

the transcripts from each experimental condition; however, additional linguisticcategories are considered in the analysis.Transcripts from each trial were analyzed with LIWC. Output from the tool consistedof frequencies of words from over 70 different categories. Output was analyzed withSPSS for significant differences between experimental conditions using Wilcoxon’ssigned-rank test. Results are provided below. Category HE-Mean TE-Mean Mean Dif. Wilcoxon sig. word count 1376.8 1566.4 -189.6 0.05 affect (%) 4.66 3.44 1.22 0.05 you (%) 1.608 1.072 0.536 0.05 relative (%) 5.988 7.086 -1.098 0.05 Table 5.6 Linguistic categories with significant differences between experimental conditionsThis analysis suggests that the HE condition corresponds with significantly lowerword count and fewer relativity-related words, and significantly more affect-relatedand you-related words. These metrics are described below in more detail.Word CountThis metric represents the total number of words in the processed text and includesboth words that are found in the program’s dictionary file, e.g., “together,” andnonsense words like, “orrrrly.” Most other LIWC generated metrics are calculated aspercentage of total word count. For all subject pairs that participated in theexperiment, total word count was smaller in the HE condition than in the TEcondition. 64

Affect ProcessesThe affect word category consists of 916 distinct emotion-related words, and issubdivided into positive and negative emotion words. The negative emotion categoryis further subdivided into anxiety, anger, and sadness categories. Example words foreach category within this hierarchy are presented below. Word Category Abbrev Examples Affective processes affect Happy, cried, abandon ---Positive emotion posemo Love, nice, sweet ---Negative emotion negemo Hurt, ugly, nasty ------Anxiety anx Worried, fearful, nervous ------Anger anger Hate, kill, annoyed ------Sadness sad Crying, grief, sad Table 5.7 LIWC affective processes hierarchyWords that are encountered by the LIWC processing engine are counted as part of allparent categories, thus an anger-related word would also be counted as a negativeemotion word, and an affective process word.Singularly, the ratios of positive and negative emotion words to total word count donot differ significantly between HE and TE conditions, but collectively as affectwords, ratios were higher in the HE condition for all subject pairs.You WordsThis category consists of 20 different second-person pronoun words, such as you,your, and thou. It is a child of the personal pronouns category, which itself is a childof the total pronouns category. You-word ratios were higher in the HE condition thanin the TE condition for all subject pairs. 65

Relativity WordsThis category consists of 638 relativity-related words. It is a top-level category withmotion, space, and time categories as children. The hierarchy is presented below withexamples. Word Category Abbrev Examples Relativity relative Area, bend, exit, stop ---Motion motion Arrive, car, go ---Space space Down, in, thin ---Time time End, until, season Table 5.8 LIWC relativity hierarchyRatios of child categories motion, space, and time were not significantly differentbetween HE and TE conditions, but ratios of the collective relativity-words categorywere smaller in the HE condition for all subject pairs.5.6.4 System UsabilityAfter subjects finished the HE test condition they were asked to rate the EmoChatapplication in terms of usability by completing the SUS questionnaire. The SUSproduces output on a scale from 0 to 100, with higher scores considered to indicategreater usability. Results for the EmoChat application were typically high (N=10),ranging from 72.5 to 87.5 (mean=76.25).5.6.5 Informal InterviewsSubject pairs jointly participated in informal interviews at the end of the experiment.They were asked to discuss differences between experimental conditions. Thefacilitator led the discussion and manually recorded interesting conversation topics.Notes from the interviews are presented below. 66

Subject1 / Subject2Experimental Group IRelationship: same sex long-time friendsBoth subjects indicated a general distrust of the avatar. Subject2, who was notwearing the headset, told me that he thought Subject1 may have been manuallymanipulating the movement of the avatar and affective bars. After briefly payingattention to the avatar early in the session both subjects eventually disregarded avatarand affective information presented in the application.The subjects mentioned that the lack of a “partner is entering a message…” indicatormade it more difficult to have a conversation. Subject2 is a particularly fast typistand felt as though he was dominating the conversation, since he couldn’t see when hispartner was entering text.Subject1 mentioned that the eye blink avatar behavior was very accurate. He alsosaid that he did not manually manipulate any of the Emo controls. Subject2 feltobligated to manipulate some of the avatar and affective meters after seeing themovements in Subject1’s Emo controls, however he said his manipulation choiceswere random and did not really consciously reflect a desire to convey specificemotional information.Subject1 mentioned that the location of the avatar and meters drew his focus awayfrom the chat window, and was part of the reason for his general disregard. 67

Subject3 / Subject4Experimental Group IRelationship: mixed sex romantic partnersSubjects legitimately enjoyed communicating with the headset system. At one point,Subject4 made a positive comment about Subject3’s appearance which induced avery large smile on Subject3’s face. This smile was reflected in the avatar. WhenSubject4 noticed the avatar with the large smile she seemed very pleased and giggled.Subject3 mentioned that the frustration meter may not have been entirely accurate,but that he did not immediately distrust it. He thought that it was possible that thefrustration meter was not necessarily reflecting frustration caused by theconversation, but instead caused by the recurrence of pain from a hand burn receivedearlier in the day.Subject3 mentioned that the excitement meter seemed to be the most dynamic andaccurate of the affective meters. At one point during the HE conversation, Subject4was telling Subject3 a joke. During waiting periods between lines of the joke,Subject3 noticed that excitement levels and frustration levels were increasing, whichhe thought was appropriate. When asked which condition the subjects preferred theystrongly indicated the headset condition. Subject4 noted that she was much moreinterested in communicating with Subject3 with the avatar and affective meterspresent, than not present. She said the experiment was actually fun. 68

Subject5 / Subject6Experimental Group IIRelationship: mixed sex short-term friendsSubject5 reported not manually manipulating the EmoControls, even though he wasinformed that they were available for use as he saw fit. Subject6 said she wished shehad a headset; she would have enjoyed the communication more and would havebeen able to focus more on the conversation and less on manually making the avatarface and affective meters accurately reflect her emotional state.Subject5 stated that he felt the avatar and affective meters were reasonably accurate,and that more than increased accuracy, he wished the presentation was moreaesthetically pleasing. As it stands, the avatar is very simple, and animations betweenstates are abrupt.Subject6 purposely tried influencing the affective levels reported by Subject5 bytelling jokes. Both Subject6 and Subject5 noticed that excitement and engagementlevels increased when Subject5 was told an entertaining joke. Subject5’s avatar alsoreflected his smile reaction to the jokes.Both Subject5 and Subject6 agreed that the (TE) condition was less fun that the (HE)condition. 69

Subject7 / Subject8Experimental Group IIRelationship: brothersSubject7 mentioned that several times during the conversation he switched to manualcontrol of the frustration meter, because he didn’t think it was returning to base linequickly enough.Subject8 noticed that every time he mentioned his new girlfriend, who his brotherSubject7 does not entirely approve of, Subject7’s frustration meter spiked.Subject8 mentioned that he felt that manually manipulating the avatar (since he didnot have a headset) caused him to pay less attention to the content of the dialog, andmore attention to making his avatar reflect his current emotional state. He said thathe wished he also had a headset.The issue of distrusting the avatar was brought up by Subject8 since participants werepermitted to manually manipulate facial expression and affective states. Hementioned he would have been happier if the partner was notified whenever manualEmoControls were invoked. 70

Subject9 / Subject10Experimental Group IRelationship: Mixed sex long-time friendsSubject10 mentioned that Subject9 seemed to be quite frustrated, but Subject9 said hewasn’t very frustrated. Subject10 also noticed increased in excitement andengagement when she asked Subject9 about a recent party they both attended.Subject9 said that he was unsure whether or not Subject10 was wearing a headset forthe experiment. Subject10 mentioned that she didn’t think she was supposed todivulge that information.Subject10 characterized the HE condition as “funny”. She appreciated the facialexpressions of her partner’s avatar. 71

5.6.5.1 Trends Identified During InterviewsSome subjects reported a general distrust of the avatar expressions and affectivemeters since subjects were permitted to manually manipulate these features withouttheir partner being aware. In one case, this distrust caused the chat partners tocompletely disregard changes in avatar features, but this was isolated. Most headsetusers indicated that they did not manually manipulate anything, opting to let theoutput from the headset control any changes to avatar states.Subject pairs in the EmoChat condition consisted of one partner with a headset andone partner without. The majority of non-headset users indicated that they wishedthey had also been using a headset. It was noted several times that having tomanually manipulate avatar features was time-consuming and took away from themain conversation. During manual manipulation, focus was shifted from the mainchat window to the override control pane of the application, and substantial effort wasrequired to alter avatar features to reflect present emotional state.Participants generally preferred using EmoChat over the traditional environment.Most agreed that EmoChat provided an environment that was interesting and fun touse. Several participants spontaneously started telling jokes to their headset-wearingchat partners because they wanted to see how it affected the avatar features. Thereseemed to be a genuine interest in the emotional information presented in changes toavatar facial expression and movement in the affective meters. 72

Frustration was perceived as inaccurate by several participants. One participant inparticular switched to manual manipulation of the frustration meter because hethought it was not returning to baseline quickly enough, and presumably did not wanthis partner to conclude that he was frustrated. Other participants noted that theirfrustration meter was spiking when they did not perceive themselves as beingparticularly frustrated. The meditation meter was mentioned only once during chatsessions and informal interviews. It was used for comedic effect when oneparticipant maxed out the level and stated that he was “meditating like a Jedi.” Onthe whole, participants felt that excitement and engagement meters were substantiallymore accurate than frustration and meditation meters.5.7 Discussion5.7.1 Summary of ResultsThis study has introduced a novel instant messaging application called EmoChat thatintegrates with the Emotiv Epoc headset to detect and convey brow, eye, and mouthmovements, and basic affective states of excitement, engagement, frustration, andmeditation.The main goals of the study were to compare EmoChat with a traditional instantmessaging environment in terms of the amount of emotional information contained inmessages, richness of experience, accuracy of emotional transfer, and usability, andto evaluate the effectiveness of using a brain-computer interface device like the Epocto facilitate emotional transfer in an IM environment. 73

Results suggest that communication with EmoChat contains a significantly higherpercentage of affect-related and you-related words, than communication with atraditional environment.In terms of richness, the data suggests that the level of affective intelligence washigher with EmoChat than with a traditional environment. Subjects rated their chatpartners as significantly better able to convey emotional information with EmoChat.Subjects also felt that using EmoChat helped them to understand the emotions of theirpartners significantly more than with a traditional environment. The data alsosuggests that other measures of richness, including involvement, enjoyment, sense ofcopresence, and overall satisfaction were marginally higher with EmoChat.Accuracy of emotional transfer was not determined to be significantly differentbetween chat environments; however, additional analysis was able to identifysignificant correlation between behavior of the avatar and emotion perceived by thechat partner.Usability ratings of the application on the SUS scale ranged from 72.5 to 87.5 with amean of 76.25. Subjects who used the headset with EmoChat reported slightly higherusability ratings (72.5 to 87.5, mean=79) than subjects without the headset (72.5 to75, mean=74). 74

5.7.2 Consistency with related workThe results of the present study are partially consistent with findings in relatedresearch, as described in the following subsections.5.7.2.1 InvolvementInvolvement in the present study was measured by responses to Q1-2 on the richnessquestionnaire. Responses to the questions were marginally, but not significantlyhigher in the EmoChat condition. This is consistent with results from Neviarouskayain that no significant differences were found; however, the present study does identifyhigher mean differences between conditions than the Neviarouskaya study.Involvement can also be estimated by characters per minute, total word count, ornumber of messages sent. The present study found that total word counts in theEmoChat condition were typically lower than in the traditional environment. Thisfinding is inconsistent with work by Fabri that reports considerably higher messagelength in an expressive avatar condition. A contributing factor to the lower total wordcount in the EmoChat condition might be that only one participant in the subject pairwas using a headset. The other participant was required to manually change avatarand affective features (if desired). Post-experiment interviews confirm that manysubjects felt like the avatar and affect meters were taking their focus away from thechat window, and might explain the lower total word count. 75

5.7.2.2 EnjoymentEnjoyment was measured by Q3 on the richness questionnaire, and was notsignificantly different between conditions. This result is consistent with findings byFabri and Neviarouskaya. Although results from the questionnaire did not suggest asignificant difference between the enjoyment of test conditions, during informalinterviews, users of the EmoChat application almost unanimously reported havingmore “fun” than with the traditional environment.5.7.2.3 CopresenceThe copresence dimension was measured by Q4-6 on the richness questionnaire.Analysis did not identify any significant difference between conditions, althoughmean differences were typically higher in the EmoChat condition. Fabri also noteshigher levels of copresence in his expressive avatar condition, although his differencewas significant. Copresence in the Neviarouskaya study is arguably higher in herexpressive avatar condition, but not significantly.5.7.2.4 Affective IntelligenceAffective intelligence in the present study was measured by Q7-9 on the richnessquestionnaire. This indicated how well the system was able to facilitate emotionalexpression. Responses were generally significantly higher in the EmoChat conditionthan in the traditional condition. This is inconsistent with the Neviarouskaya study,which was not able to identify significant differences between comparable conditions. 76

5.7.2.5 Overall SatisfactionOverall satisfaction was measured by Q10 on the richness questionnaire. Comparisonof responses between conditions did not identify significant differences; howevermean difference was marginally higher in the EmoChat condition. This is consistentwith the Neviarouskaya study in that no significant differences were found, howevermean differences in the present study are larger.5.7.2.6 UsabilityParticipants in the EmoChat condition were asked to rate the usability of the systemwith Brooke’s SUS. Responses ranged from 72.5 to 87.5 with a mean of 76.25. Thisrating is comparable to the usability of the expressive avatar system in the Fabri study(mean=78.4).5.7.3 Future DirectionAlthough the results of the present study are promising in terms of their support forthe hypothesis that EmoChat facilitates emotional communication more readily thantraditional IM environments, the relatively small sample size (N=10), createschallenges with respect to determining statistical differences between environments.The present study lays the foundation for related future studies that include a largernumber of participants.This study only investigated the use of EmoChat when one participant was using anEpoc headset. The other participant was required to manually manipulate avatar 77

features and affective state meters (if desired). It became clear during the study thatthe experience of participants who did not have the benefit of the automatic facialexpression and affect meter updates permitted by the headset was very different fromtheir counterparts’. A second study where both participants use headsets duringEmoChat sessions is likely to generate substantially different data.5.8 ConclusionThis chapter introduced the EmoChat application and compared it with a traditionalinstant messaging environment. The results of the comparison have suggested thatEmoChat facilitates communication that contains more affective information, andprovides a “richer” experience. The data has not demonstrated any significantdifference between the accuracy of the emotional information that is transferred,however, some correlation was found between the frequency of perceived affectivestates and incidence of avatar features including facial expressions and average levelsof affect meters. 78

6 Conclusion6.1 LimitationsThe studies presented in this work generally produced favorable results, howeverthere are some limitations that should be addressed. These limitations are presentedbelow.Since conversation topics were not assigned or suggested during study 2, participantswere free to discuss anything that came to mind during the session. Perhaps becauseof this freedom, in the EmoChat condition many subjects talked about their avatars,e.g., “I like your avatar,” or, “you look frustrated.” This may have contributed to thegreater frequency of affect and you-related words measured by the LIWC tool in theEmoChat condition.The novel questionnaire used to measure the accuracy of emotional transfer (theETAQ), was designed specifically for this experiment, and had not been establishedas reliable or valid prior to the study. Results from the survey were difficult tointerpret and compare between experimental conditions. Neither experimentalcondition seemed to produce consistent ETAQ results.The participant sample consisted entirely of young adults, aged 22-32, who all hadsubstantial computer and typing experience, most of whom used instant messagingapplications very frequently. This sample may not be representative of the general 79

population, so conclusions that are suggested by this study may not generalize beyondthe sample.The affectiv suite components of frustration and meditation were included as featuresin the EmoChat application, although they had not been shown to accurately reflectself-reported levels in the previous validation study. As noted in section 5.2.1, it wasa conscious decision to include these additional features to allow participants todecide for themselves whether or not headset-reported levels of frustration andmeditation were of any value. During post-experiment interviews, several subjectsmentioned that frustration levels reported by the headset may not have been accurate.This is in agreement with the previous validation study in which no strong correlationwas found between headset and self-reported levels of frustration. References tomeditation in chat transcripts and interviews indicate that the feature was generallydisregarded and not considered an important measure during interpersonalcommunication. The decision to include these non-validated features of the affectivsuite in the EmoChat application may have skewed the study results, particularly withrespect to affect-related word frequency in chat transcripts. Some participantsmentioned frustration during chat sessions, e.g., “are you frustrated?” This may havecaused affect-related word frequency to appear artificially high in the EmoChatcondition. Future studies may omit these non-validated affectiv suite features fromthe application entirely. 80

The EmoChat application used headset output from both the Emotiv affectiv andexpressive suites; however the previous study was only concerned with validating asubset of the affectiv suite. The affectiv suite inferences were expected to usecomplex classification algorithms to transform raw electroencephalogram (EEG) datainto levels of basic affective states. It was decided that the challenges involved withthis transformation warranted testing to determine accuracy. In contrast, it wasassumed that the facial expression recognition capabilities of the expressive suitewere based on existing research in the comparatively mature facial electromyography(EMG) field, and did not necessitate validation. The assumptions that were madeduring the previous validation study design may have been inappropriate, and it isnoted that a more thorough study should also investigate the accuracy of the Emotivexpressive suite if these detections are to be used in future work.The chat log analysis performed with the LIWC tool to identify significantdifferences in word-type frequency was primarily concerned with differences inaffect-related words between experimental conditions. For the sake of completeness,and to identify potential word frequency differences beyond the scope of the study, all72 word categories were tested between conditions. It is understood that applyingsuch a large number of tests may have produced results that incorrectly identifysignificant differences between conditions, when no true difference exists. It shouldbe expected that significant differences (p ≤ .05) are incorrectly determined to exist in5 out of every 100 tests performed. Applied to the present study, this means that of72 tests performed, 3.6 of those tests are expected to report differences that are falsely 81

identified as significant. The present study notes that 4 word categories were foundto be significantly different between conditions, but the large number of testsperformed merit careful interpretation of these results. The Bonferroni correctionmay have been applied in this situation as a potential workaround. This would havedictated that the standard significance level of (p=.05) be divided by the number oftests performed (72) to produce a corrected significance level of (p=.000694). Due tothe small initial sample size, testing with this corrected significance level would belikely to identify false-negatives, and may have failed to detect real significantdifferences. Ideally, the study should have just tested in-scope hypotheses, i.e.,differences in affect-related words only. It might be prudent, however, for any futurerelated studies that use this methodology to also test for differences in you andrelativity-related word frequency, and total word count between experimentalconditions. If significant differences are also found in future studies, then it becomesmore likely that these differences actually exist and are not appearing purely bychance. It may also be informative to test for frequency differences in the child word-categories of positive and negative emotion in addition to the parent affect-relatedcategory.6.2 Summary of ContributionsThis thesis work has presented two related studies. The overall contributions of eachstudy are discussed below. 82

The first study investigated the accuracy of the Emotiv Epoc headset in terms of itsability to correctly identify levels of excitement, engagement, and frustration. Resultsfrom the study suggest that the Epoc is reasonably capable of inferring levels ofengagement, and to a lesser degree, excitement. This information should beapplicable to any future research that involves the use of the Epoc headset affectivsuite. Additionally, the techniques used during study 1 may be applicable tovalidation studies of alternative systems that infer affective states from physiologicmetrics. Future research may wish to adapt the methods used in the present study toelicit differing affective responses through controlled variability of game playdifficulty.The second study introduced the EmoChat application, an instant messagingenvironment that integrates with the Epoc headset to convey facial expressioninformation, and levels of basic affective states during IM communication. EmoChatwas compared with a traditional instant messaging environment, and was found to bemore capable of communicating emotional information. The findings from this studyadd to existing bodies of research concerning affective computing, instant messaging,and brain-computer interfacing. The methods used here may be applicable to futurestudies that seek to evaluate the “affective intelligence” capabilities of an instantmessaging environment, regardless of how emotional information may be input to thesystem. Researchers are free to adapt the experimental framework presented in thisstudy, including procedures followed, measures used, and analyses performed. In amore abstract sense, another contribution of this work is that it establishes the 83

potential utility of commercial BCI devices in interaction design. This work maygenerate additional interest within the research community to assess how availableBCI-based tools can be applied to the field of human-computer interaction, outside ofthe traditional role of assistive technology.6.3 EmoChat Design ConsiderationsAlthough EmoChat was generally well received by users, modifications to theapplication may improve the overall experience of messaging. The majority of usersdid not feel the need to manually override any of the EmoChat controls (such aslocking a particular facial feature, or manually manipulating the affect meters), butthe possibility of doing so without disclosure of manual manipulation to the partnerled some participants to question whether or not they were being deceived, i.e., “ismy partner really smiling, or has he locked that control?” Future iterations ofEmoChat should either inform the partner when override controls are invoked, orprevent overriding the controls altogether.The general appearance of the avatar is very basic. One participant mentioned that hewould have had a more enjoyable experience if the aesthetic quality of the avatar hadbeen improved. Future versions of EmoChat should present more realistic avatarsthat perhaps smoothly animate between facial expressions, instead of the currentabrupt transitions. 84

Avatar placement in the current version of EmoChat is off to the side of the main chatwindow. One subject mentioned that having to divert his gaze from the chat windowto see changes in the avatar made doing so less appealing. Subsequent revisions ofthe application may investigate placement of the avatar within the main chat window,or slightly above, to make monitoring changes easier for the user.6.4 EmoChat Compared with Existing Methods of Emotional Instant MessagingExisting methods of conveying emotion in instant messaging environments werereviewed in chapter 3. This section provides a comparison between EmoChat andapplications with similar input and output techniques.6.4.1 Input TechniqueEmoChat uses a form of automated facial expression recognition that is thought torely on interpretation by the Epoc headset of electromyographic signals produced byfacial muscle movement. These electrical signals are detected by the Epoc and usedto infer changes in states of brow, mouth, and eye facial features. Existing facialexpression recognition algorithms typically rely on visual input from a camera orvideo stream and require that the user’s face remain situated in front of the camerawith no more than a minor angle of incidence. This means that if a user looks downat the keyboard to type a response, or looks away from the camera to attend tosomething else, the system may fail. By contrast, the Epoc headset transmits data 85

wirelessly to a USB dongle connected to the computer running EmoChat at ranges upto 15 feet. This provides the user with much greater freedom of movement, at theexpense of cost, setup time, and comfort associated with using an EEG headset.In addition to facial expression recognition, the Epoc is thought to useelectroencephalographic signals to infer levels of basic affective states includingexcitement, engagement, frustration, and meditation. To date, no other instantmessaging application has used EEG input in this manner. This may potentiallyprovide emotional information that is not associated with overt sentic modulation, andmay be capable of detecting very subtle emotional changes. This is in contrast tomost existing research that uses only overt symptoms of emotional experience to inferaffect, e.g., solely facial expression, haptic techniques, and explicit text or phrasematching. The addition of covert symptoms of emotional experience in the EmoChatsystem permits the communication of emotional information that might be consideredmore intimate.6.4.2 Output TechniqueEmoChat presents users with a very simple animated avatar that might be perceivedas far less realistic than some expressive avatars used in other instant messageapplications. Additionally, the technology used in EmoChat does not permitsimultaneous automatic changes to both mouth and brow features. This may preventthe avatar from expressing certain emotions that rely on simultaneous featurechanges, such as fear, or sadness. However, the application does provide override 86

controls to prevent facial features from changing based on headset input, and allowsusers to manually select a preferred feature. In this way, a user could manuallyoverride the mouth feature to force a persistent smile while allowing input from theheadset to modulate brow movement.This implementation of an animated avatar is in contrast with techniques presented insection 3.3.2. While existing implementations modulate multiple facial features toconvey a specific emotion, e.g., brow lowered, lips pursed, to indicate anger,EmoChat takes a more granular approach. Users are free to change facial featuresindependent of one another. The facial movement in the EmoChat avatar is notdetermined by which emotion is inferred or manually selected, but instead attempts todirectly mimic the expression of the user. The ability to combine multiple,independent facial features lets the EmoChat avatar display a larger number ofdistinct expressions than existing methods. By manipulating only brow and mouthmovements, for example, EmoChat can display up to 25 distinct facial expressions.A drawback to the avatar implementation in EmoChat is that it may not be able tofaithfully reproduce some facial expressions present in existing systems, includingdisgust, fear, or surprise, because the individual component images associated withchanges in brow and mouth state are limited.6.5 Future DirectionBeyond the future directions previously described in sections 4.7.2 and 5.7.3, thepresent work may provide impetus for additional related studies. 87

The existing EmoChat application is designed for a maximum of two users. It may beinteresting to study how similar emotion input and output techniques translate to alarger group chat environment, and how the group dynamic is changed by adding thistype of emotional information channel.Although EmoChat was well received, and users seemed genuinely interested in theemotional information it was providing, the real-world feasibility of such a systemhas not been established. A longitudinal study might suggest how, when, and why asystem like this might be used, and by whom.Presently, the EmoChat application does not make inferences as to where a particularuser’s experience of emotion lies in arousal-valence space. It may be possible to usedata recorded by the headset to classify emotional experience into some set ofdiscrete categories, e.g., positive/negative valence, or high/low arousal.6.6 ClosingThe work presented in this thesis suggests that computer-mediated communicationmay benefit from the integration of brain-computer interface devices like the EmotivEpoc for facilitating communication that contains more emotional information thantraditional environments. These results may generalize to the larger affectivecomputing domain if states like engagement and excitement can be accuratelyinferred. One might envision using this technology in a virtual environment settinglike second life, to endow an avatar with a greater degree of humanity, or in adistance-learning scenario, to monitor the interest level of students. It is up to the 88

scholars and proponents of human-centered computing and its related disciplines toinvestigate the best uses for this technology, perhaps, in applications that have not yetbeen conceived. The author believes that future iterations of BCI technology willincorporate greatly increased capabilities, opening up a vast number of newpossibilities for human-machine interaction, and he is proud to present the modestfindings of this thesis to the community. 89

Appendix A: Validation Study Demographics QuestionnaireName: _________________________________ Age: ________ Gender: ________1. How often do you play computer or console games? Very Never Rarely Occasionally Frequently Frequently2. How familiar are you with the game of Tetris? Not at All Somewhat Moderately Pretty Very Familiar Familiar Familiar Familiar Familiar3. How often do you play Tetris, or a similar game? Very Never Rarely Occasionally Frequently Frequently4. What is your typical skill level when it comes to playing computer or consolegames? Far Below Below Above Far Above Average Average Average Average Average5. What are some of your favorite computer or console games, if any?_______________________________________________________________________________________________________________________________________________________________________________________________________________ 90

Appendix B: Validation Study Experiment QuestionnaireRound [n] Not at Slightly Moderately Very Extremely AllHow Difficult was this 1 2 3 4 5round?How Fun was this round? 1 2 3 4 5On average, how Engageddid you feel during this 1 2 3 4 5round?On average, how Exciteddid you feel during this 1 2 3 4 5round?On average, howFrustrated did you feel 1 2 3 4 5during this round?(repeated for each round of play) 91

Appendix C: Validation Study Post-ExperimentQuestionnaire1. What types of events caused you to get excited during game play?_______________________________________________________________________________________________________________________________________________________________________________________________________________2. What makes a game like Tetris engaging?_______________________________________________________________________________________________________________________________________________________________________________________________________________3. What happened during the game that made you feel frustrated?_______________________________________________________________________________________________________________________________________________________________________________________________________________4. What makes Tetris fun?_______________________________________________________________________________________________________________________________________________________________________________________________________________5. Do you have any additional comments?__________________________________________________________________________________________________________________________________________ 92

Appendix D: EmoChat Demographics QuestionnaireName: _________________________________ Age: ________ Gender: ________1. Rate your level of computer experience. None Basic Intermediate Expert2. Rate your keyboard typing skills. None Basic Intermediate Expert3. Please indicate your highest level of education completed. High Some Bachelor’s Master’s PhD or None School College Degree Degree Higher4. How often do you use Instant Message tools, such as MSN, AIM, ICQ, etc.? Never Occasionally Regularly Every Day 93

Appendix E: EmoChat Emotional Transfer QuestionnaireDuring this chat session, how often did [EX role: you | PX role: your chat partner]experience the following emotions: Very Seldom Seldom Sometimes Often Very Often or Not at AllAfraid 1 2 3 4 5Angry 1 2 3 4 5Annoyed 1 2 3 4 5Astonished 1 2 3 4 5Bored 1 2 3 4 5Content 1 2 3 4 5Excited 1 2 3 4 5Frustrated 1 2 3 4 5Happy 1 2 3 4 5Interested 1 2 3 4 5Miserable 1 2 3 4 5Pleased 1 2 3 4 5Relaxed 1 2 3 4 5Sad 1 2 3 4 5Satisfied 1 2 3 4 5Tired 1 2 3 4 5 94

Appendix F: EmoChat Richness Questionnaire1. I felt it was important to respond after each of my partner’s statements. Strongly Strongly Disagree Neither Agree Disagree Agree2. I was interested in my partner’s responses. Strongly Strongly Disagree Neither Agree Disagree Agree3. I enjoyed communicating with my chat partner. Strongly Strongly Disagree Neither Agree Disagree Agree4. I had the sensation that my partner was aware of me. Strongly Strongly Disagree Neither Agree Disagree Agree5. I felt as though I was sharing the same space with my partner. Strongly Strongly Disagree Neither Agree Disagree Agree6. I had the sensation that my partner was responding to me. Strongly Strongly Disagree Neither Agree Disagree Agree7. I was able to successfully convey my feelings with this application. Strongly Strongly Disagree Neither Agree Disagree Agree8. My partner was able to successful convey his or her feelings with this application. Strongly Strongly Disagree Neither Agree Disagree Agree9. The emotional behavior of the avatars was appropriate. Strongly Strongly Disagree Neither Agree Disagree Agree10. I understood the emotions of my partner. Strongly Strongly Disagree Neither Agree Disagree Agree11. I am satisfied with the experience of communication with this system Strongly Strongly Disagree Neither Agree Disagree Agree 95

BibliographyAndrei, S. (2010). Toward Natural Selection in Virtual Reality. IEEE Computer Graphics and Applications, 30, 93-96,C93.Brooke, J. (1996). A "Quick and Dirty" Usability Scale. In Jordan, McLelland, Thomas & Weerdmeester (Eds.), Usability Evaluation in Industry (pp. 189-194): Taylor & Francis.Campbell, A., Choudhury, T., Hu, S., Lu, H., Mukerjee, M. K., Rabbi, M., et al. (2010). NeuroPhone: brain-mobile phone interface using a wireless EEG headset. Paper presented at the Proceedings of the second ACM SIGCOMM workshop on Networking, systems, and applications on mobile handhelds.Chanel, G., Rebetez, C., Betrancourt, M., & Pun, T. (2008). Boredom, engagement and anxiety as indicators for adaptation to difficulty in games. Paper presented at the Proceedings of the 12th international conference on Entertainment and media in the ubiquitous era.Ekman, P., & Oster, H. (1979). Facial Expressions of Emotion. Annual Review of Psychology, 30(1), 527-554.Fabri, M. (2006). Emotionally Expressive Avatars for Collaborative Virtual Environments. Leeds Metropolitan University.Fabri, M., & Moore, D. (2005, October 2005). Is Empathy the Key? Effective Communication via Instant Messaging. Paper presented at the in Proceedings of 11th EATA International Conference on Networking Entities, St. Poelten, Austria.James, W. (1884). What is an Emotion? Mind, 9(34), 188-205.Kaliouby, R. E., & Robinson, P. (2004). FAIM: integrating automated facial affect analysis in instant messaging. Paper presented at the Proceedings of the 9th international conference on Intelligent user interfaces. 96

Kayan, S., Fussell, S. R., & Setlock, L. D. (2006). Cultural differences in the use of instant messaging in Asia and North America. Paper presented at the Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work.Khalili, Z., & Moradi, M. H. (2009). Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of EEG. Paper presented at the Proceedings of the 2009 international joint conference on Neural Networks.Kiesler, S., Zubrow, D., Moses, A. M., & Geller, V. (1985). Affect in computer- mediated communication: an experiment in synchronous terminal-to- terminal discussion. Hum.-Comput. Interact., 1(1), 77-104.Lang, P. J. (1995). The Emotion Probe: Studies of Motivation and Attention. American Psychologist, 50(5), 372-385.Lo, S.-K. (2008). The Nonverbal Communication Functions of Emoticons in Computer-Mediated Communication. CyberPsychology & Behavior, 11(5), 595-597.Neviarouskaya, A. (2008). AffectIM: An Avatar-based Instant Messaging System Employing Rule-based Sensing. Unpublished MS, University of Tokyo, Tokyo.Neviarouskaya, A., Prendinger, H., & Ishizuka, M. (2007). Recognition of Affect Conveyed by Text Messaging in Online Communication Online Communities and Social Computing (Vol. 4564, pp. 141-150). Heidelberg: Springer Berlin.Oakley, I. O. M., S. (2003). Contact IM: Exploring asynchronous touch over distance. Paper presented at the CSCW02.Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005). Affective multimodal human-computer interaction. Paper presented at the Proceedings of the 13th annual ACM international conference on Multimedia.Pennebaker, J. W., Booth, R., & Francis, M. (2007). Linguistic Inquiry and Word Count: LIWC2007. 97

Picard, R. W. (1997). Affective computing: MIT Press.Reisenzein, R. (2006). Arnolds theory of emotion in historical perspective. Cognition and emotion, 20(7), 32.Rivera, K., Cooke, N. J., & Bauhs, J. A. (1996). The effects of emotional icons on remote communication. Paper presented at the Conference companion on Human factors in computing systems: common ground.Rovers, A. F., & Essen, H. A. v. (2004). HIM: a framework for haptic instant messaging. Paper presented at the CHI 04 extended abstracts on Human factors in computing systems.Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161-1178.Sanchez, J. A., Hernandez, N. P., Penagos, J. C., & Ostrovskaya, Y. (2006). Conveying mood and emotion in instant messaging by using a two- dimensional model for affective states. Paper presented at the Proceedings of VII Brazilian symposium on Human factors in computing systems.Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695-729.Sherstyuk, A., Vincent, D., & Treskunov, A. (2009). Towards Virtual Reality games. Paper presented at the Proceedings of the 8th International Conference on Virtual Reality Continuum and its Applications in Industry.Sloten, J. V., Verdonck, P., Nyssen, M., & Haueisen, J. (2008, November 23-27). On modeling users EEG response during a human-computer interaction: a mirror neuron system-based approach. Paper presented at the 4th European Conference of the International Federation for Medical and Biological Engineering, Antwerp, Belgium.Tetteroo, D. (2008). Communicating emotions in instant messaging, an overview. University of Twente, Enschede. 98

Thompson, E. R. (2007). Development and Validation of an Internationally Reliable Short-Form of the Positive and Negative Affect Schedule (PANAS). Journal of Cross-Cultural Psychology, 38(2), 227-242.Wang, H., Prendinger, H., & Igarashi, T. (2004). Communicating emotions in online chat using physiological sensors and animated text. Paper presented at the CHI 04 extended abstracts on Human factors in computing systems.Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063-1070.Yahoo. (2010). Emoticon survey results. Retrieved October 4, 2010, from http://www.ymessengerblog.com/blog/2007/07/10/emoticon-survey- results/Yeo, Z. (2008). Emotional instant messaging with KIM. Paper presented at the CHI 08 extended abstracts on Human factors in computing systems. 99

Emochat: Emotional instant messaging with the Epoc headset

by fwrigh2 on Dec 15, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Emochat: Emotional instant messaging with the Epoc headset — Document Transcript