Brain, Body, and Bytes CHI 2010 Workshop Presentations

BBB Workshop Overview Audrey Girouard, Erin Treacy Solovey, Regan Mandryk, Desney Tan, Lennart Nacke, Robert J.K. Jacob

BBB Workshop: Bringing Together BCI& Physiological Computing Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 3

Workshop Goals 1. Provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI). 2. Kick-start BBB research in HCI by identifying key research questions and application areas 3. Enhancing HCI methodologies by adding these new methods and techniques to the toolbox of HCI researchers Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 4

About Us Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 5

About you • 25 Participants • X EEG papers • Y fNIRS papers • Z EMG papers • 9 Brain papers • A GSR • 19 Body papers • B Eye tracking Brain Body 32% 68% Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 6

Workshop Discussion Questions <on your tables> Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 7

Workshop Discussion Questions What terms/acronyms should we use to describe our work? Psychophysiological computing seems a bit long… BBB? BBI? Other suggestions? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 8

Today’s Schedule 09:00-10:30 Introduction of organizers & participants, CHI madness style 10:30-11:00 Break 11:00-12:30 Group Discussion (assign groups + topic) 12:30-14:00 Lunch 14:00-15:30 Small group discussion (1h of discussion + 30 minutes of reporting) 15:30-16:00 Break 16:00-17:00 Report from small groups, recap, group discussion – barriers to publication in HCI Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 9

Staying connected Website http://www.eecs.tufts.edu/~agirou01/workshop/ Twitter http://twitter.com/bbbCHI2010 Twitter #bbbCHI2010 Hashtag Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and- Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786 Flickr http://www.flickr.com/groups/bbbchi/ Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological- user-interaction Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 10

And now… 2 Minute Introductions • About your research: – Application or domain space? (e.g. video games) – Signal you are measuring? (e.g. fNIRS) – Device/hardware you are using? (e.g. ISS, Inc) • What would you like to get out of workshop? – What problems are you facing? – What do you see in common with others here? – What device would you like to know more about? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 11

List of Presentations 1. Adriane Randolph 16. Katarzyna Wac 2. Krzysztof Z. Gajos 17. Feng Tian 3. Saraswathi Bellur 18. Veronica Zammitto 4. Danny Plass-Oude Bos 19. Johanna Octavia 5. Kai Kuikkaniemi 20. Bob Wray 6. Mitchel Benovoy 21. Marjolein van der Zwaag 7. Yee Chieh Chew 8. Martha Crosby 9. Jan Kallenbach 10. Eva Oliveira 11. Lucia Filgueiras 12. Marc Grootjen 13. Melody Moore Jackson 14. Johann Schrammel 15. Daniel Sjölie Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 12

Brain-Computer Interfaces as Child’s Play: Using Mental Training to Improve Motor Skills in Children Adriane B. Randolph, Ph.D. Ashley Ingraham Kennesaw State University

Outline • Research Position – Examine ties between sensorimotor area and physical ability to uncover a training tool to aid children’s motor skill development • Pilot Study – 3 able-bodied adults (2 Male, 1 Female) – Acquire baseline Wii tennis score – Conduct mental training with BCI2000 mu training protocol for 5 days – Retest Wii tennis score for improvements Participant Session One Session Five – Mixed results AZ 98 60 CM 70 64 MW 103 118 • Future Work – Extend study with 20 able-bodied adults (currently in progress) – Analyze link between training and baseline ability – Test with elementary-aged children Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 14

Screening for Mu Ashley prepares participant to use mental imagery of hand and foot movements to determine his best channels for use in mu BCI training before he plays a match of Wii Tennis. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 15

Thank you Adriane B. Randolph, Ph.D. Kennesaw State University BrainLab (770) 423-6083 brainlab@kennesaw.edu http://coles.kennesaw.edu/brainlab/ Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 16

Understanding How to Design Complex Brain-Controlled Applications Krzysztof Z. Gajos

How to Design Brain-Controlled Applications? • Basic operations Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 18

How to Design Brain-Controlled Applications? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 19

How to Design Brain-Controlled Applications? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 20

How to Design Brain-Controlled Applications? • Basic operations Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 21

How to Design Brain-Controlled Applications? • Interaction between primary task and BCI performance Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 22

How to Design Brain-Controlled Applications? • Error prevention, recovery, and actions with side effects Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 23

Thank you Krzysztof Gajos kgajos@eecs.harvard.edu http://www.eecs.harvard.edu/~kgajos/ Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 24

Psychophysiological Responses to Media Interfaces Saraswathi Bellur S. Shyam Sundar Media Effects Research Lab http://www.psu.edu/dept/medialab Pennsylvania State University

Biopac MP 35 & MP 150 ACQknowledge software  Skin Conductance (EDA)  Heart Rate (ECG)  Facial EMG  Brain Wave (EEG)  Eye Movement (EOG)  Application Areas Interface presentation Information Processing Usability Testing & Evaluation  User Experience Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 26

Physiological Measures: Affect, Arousal & Attention 3. ECG: Heart Rate recording 1. EEG: Bipolar recordings Occipital Alpha Blocking Orienting Response via BPM change 2. Skin Conductance (EDA) Tonic arousal levels 4. Facial EMG: Corrugator & Zygomatic recordings Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 27

Theoretical Frameworks • The Media Equation: Mediated Life = Real Life (Reeves & Nass, 1996). • Structural aspects of media technologies: Automatic or Controlled processing? Voluntary or Involuntary responses? • Specific cues triggered via interactions with four types of media affordances, MAIN model (Sundar, 2008) Animation M: MODALITY Customization Heuristics A: AGENCY Information Perceptual Scent Bandwidth I: INTERACTIVITY N: NAVIGABILITY Contingency UGC & Sourceness Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 28

Thank you Saras Bellur (saras@psu.edu) S. Shyam Sundar (sss12@psu.edu) Media Effects Research Lab http://www.psu.edu/dept/medialab Pennsylvania State University Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 29

BCI for Games Danny Plass-Oude Bos Boris Reuderink, Bram van de Laar, Hayrettin Gürkök, Christian Mühl, Mannes Poel, Anton Nijholt, Dirk Heylen HMI, University of Twente

Target Group: the general population Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 31

BCI: issues Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 32

BCI: new information, new features Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 33

Acceptance: Usability & Realistic Settings Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 34

Subjects  Users Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 35

Thank you d.plass@ewi.utwente.nl http://hmi.ewi.utwente.nl/~oudebos Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 36

Designing Biofeedback for Games and Playful Applications Kai Kuikkaniemi Helsinki Institute for Information Technology

Emoshooter Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 38

Emoshigeru - cinema games Real-time Public display Biosignal adaptation Massive co-located Shooter / Strategy Part of Shigeru-games local wlan Public display Connected to game server Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Presemo – presentation engine Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Emolisten – biosignal sonification Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Research statement • Social Games and Playful Applications, which Utilize Biosignals • Grey Area between Adaptation and Biofeedback – Implicit and Explicit Biofeedback • Biosignal Synchrony (Inter-Personal) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 42

Our team and projects Professor Marko Turpeinen Professor Timo Saari Ilkka Kosunen, Toni Laitinen, Petri Lievonen, Pauli Ojala Fun of Gaming (EU FP6) (2006-2009) Enactive Social Media and Games (Aalto) (2009) Emokeitai (Tekes – National Innovation Agency) (2009-2011) Thanks: CKIR / Niklas Ravaja Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 43

Psychophysiological signal analysis and classification Mitchel Benovoy Centre for Intelligent Machine, McGill University

Outline • Signals • Hardware • Application domains Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 45

Signals Galvanic Skin Response Phalange Temperature 1.5 ∆ Cond. (µS) 1 0.5 0 -0.5 0.5 1 1.5 2 2.5 Sample 4 x 10 Electromyogram 42 Blood Volume Pulse – HR & Amp. Respiration – Rate & Amp. 40 2.2 2 38 Refl. Exp. 1.8 36 1.6 34 1.4 32 1.2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 7 1 0.8 Amp. Amp. 6 0.6 0.4 5 0.2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 105 30 HR (bpm) RR (bpm) 100 25 95 20 90 15 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 46

Hardware • Thought Technology ProCom Infinity Images via www.thoughttechnology.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 47

Application Domains • Affective Computing – Interaction evaluation (UI, devices, latency) – Music & emotion research – Psychoneurology Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 48

Thank you benovoym@cim.mcgill.ca Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 49

BCI and Creativity Yee Chieh (Denise) Chew Georgia Institute of Technology

Outline • fNIR and Creative Expression • EEG and Music Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 51

Character Templates for fNIR Created with a mouse, these are templates that subjects traced using fNIR Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 52

Thank you ychew@gatech.edu Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 53

Physiological Measures Used for Identification of Cognitive States and Continuous Authentication Curtis S. Ikehara & Martha E. Crosby University of Hawaii at Manoa

Outline • Overview of our Research Using Physiological sensors – Assessing Cognitive States – Continuous Authentication • Possible Applications Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 55

Multi-modal Physiological Sensors System Eye Tracker (Gaze Position, Fixation Number, Fixation Duration, Repeat Fixations, Search Patterns, MTF Pupil Size, Blink Rate, Blink Duration) (Moving Target Fractions) HUMAN Pressure Mouse (Pressures on Mouse Button and Body) Laptop (Data Collection, Analysis & Display) EDA / TSA Finger Sensors (Electrodermal and Temperature Sensor Activity) Oximeter (Pulse, Blood Oxygen) Custom Electronics (Analog to Digital Converter) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 56 56

A Composite of Over 20 Mouse Clicks of Three People (Click Signature) US Patent # 7,245,218 Subject 4 Subject 5 Subject 7 57 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Possible Applications • The student’s confidence can be assessed by evaluating the distortions of the mouse click signature. • High Value Vehicle Control – Pressure and other physiological sensors can be attached to a steering (e.g. aircraft yoke or steering wheel) . – Sensors can be used to indicate that: • The authorized pilot or driver is in control. • The cognitive state of the pilot is consistent with the vehicle’s activity. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 58

Thank you Curtis S. Ikehara Martha E. Crosby cikehara@hawaii.edu crosby@hawaii.edu University of Hawaii at Manoa 1680 East-West Road - POST 317 Honolulu, Hawaii 96822 USA Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 59

Measuring Interaction Experiences: Integration of Multiple Psychophysiological Methods Jan Kallenbach Aalto University Helsinki, Finland

Outline User Experience = Choice & Interaction + Cost-Benefit Evaluations Measuring Interaction Experiences?  Software Framework Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 61

Multimodal Measurements Client Application for UX Research Sampling frequency: max 1000Hz Psychophysiological Measures: Gaze, Pupil Size, EDA, EMG, EEG, … Behavioral Measures: Keyboard, Mouse, Slider, … System Measures: Window Events Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 62

Thank you Jan Kallenbach Research Scholar Aalto University School of Science & Technology Department of Media Technology P.O. Box 15500 FIN - 00076 Aalto Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 63

Towards enhanced video access and recommendation through emotions Eva Oliveira Nuno Ribeiro Teresa Chambel IPCA | LASIGE

We Want • To access to videos based on emotions – user actual emotional (psychophysiological) state – video emotional content – director’s perspective • To recommend videos based on emotions – create a set of rules from every affective classification perspective (user and video, director) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 65

Affective video recommendation system requirements () • Every video must be classified affectively from the user and from the content perspective. • Every user has an emotional profile constituted by all his implicit (physiological) classification of every classified movie. • User physiological signals can change the way we watch the videos Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 66

User Classification Issues • Automatic Classification – Users emotional state • emotional model – Classification techniques • Result output • Accuracy • Lab contexts Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 67

Current State • Capturing users phychophysiological signals (BIOPAC System) while watching movies This is my exciting picture • Next step: several classification techniques to convert signal result into emotional categories Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 68

Thank you • Eva Oliveira : • eva.oliveira@gmail.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 69

Your users can’t talk! How can you tell they liked your system? Lucia Filgueiras and Claudia Tambascia Escola Politecnica Universidade de São Paulo

What we want to do • Develop products for cerebral palsy users • Perform participatory design • Understand non-verbal communication Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 71

Direct measures and usability evaluation HRV and workload measures Eye tracking in iTV apps in control room operability evaluation evaluation Comparison of eye tracker NASA TLX, SWAT, Heart Rate findings to conventional Variability usability tests results Variabilidade da frequência cardíaca OP13 LF nu / HF nu LSC Média 18,0 Operador analisa sistema após vários Índice do Balanço Simpático? Parassimpático Ação do operador para alarmes: temperatura alta (10:29); Ação do operador para 16,0 correção nível alto (10:32) e segurança correção 14,0 (10:32; 10:37; 10:43) 14,1 12,0 Alarme de 11,7 temperarura alta: 10,0 10:15 9,1 8,3 8,5 8,0 7,3 6,6 6,0 5,9 5,2 5,3 5,6 4,6 5,0 4,0 3,9 3,8 4,0 4,4 4,1 2,9 3,3 2,6 2,6 2,6 2,3 2,4 2,0 0,0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 :3 :4 :4 :5 :5 :0 :0 :1 :1 :2 :2 :3 :3 :4 :4 :5 :5 :0 :0 :1 :1 :2 :2 :3 :3 09 09 09 09 09 10 10 10 10 10 10 10 10 10 10 10 10 11 11 11 11 11 11 11 11 HORA Heart Rate Variability of an operator in a unstable unit situation Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Our questions: • How personalized are body expressions for cerebral palsy patients? • Are there general body expressions that can mean the same for everyone? • Is it possible to determine a generic way of measuring these expressions? • Is there any other liaison besides gestures and facial expressions , like an unconscious connection of limbic systems that can be responsible for the mutual sensitive communication? • Is there a reference body signal that can be used as a reliable parameter to evaluate body expressions? • How can we establish a correlation between body expressions and patient´s opinion of approval or disapproval of a design solution? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 73

Thank you! Contact us: Lucia V. L. Filgueiras Escola Politecnica, Universidade de São Paulo lucia.filgueiras@poli.usp.br Claudia A.Tambascia Fundação CPqD claudiat@cpqd.com.br http://lts-i.pcs.usp.br Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 74

Human Computer Symbiosis by Mutual Understanding Marc Grootjen EagleScience-TNO

Outline • Symbiosis & mutual understanding • Information models • Interfaces Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 76

Maximum task performance Symbiotic entity Disturbance Mutual human understanding computer Tasks (goals) Human-computer symbiosis with mutual understanding as key issue for maximum task performance Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 77

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 78

Domesticating Brain- Computer Interfaces: Systems for the Home Melody Moore Jackson BrainLab School of Interactive Computing Georgia Tech

Extended Home-based BCI study • fNIR-based BCI for communication • 32 People with ALS (so far) over 1 year • 78% (25) could operate the device on initial visit with accuracy > 70% • Accuracy to 100%, average 76.5% • Issues: – Motivation – Data Collection – Remote Support Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 80

The Kokoro Gatari Brain Computer Interface based on Functional Near Infrared imaging Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 81

Thank you Melody Moore Jackson melody@cc.gatech.edu 404-277-1324 Ian McClendon imclend@cc.gatech.edu 404-641-7712 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 82

Get up, move on! Using Electromyography to Explore the Relationship of Experience and Motion Johann Schrammel Centre for Usability Research and Engineering

Outline • Compare experiences and emotions in conditions with and without movements with Electromyography(EMG) and other methods Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 84

Outline • Can EMG successfully be used to estimate a user‘s emotional experience in motion contexts (artefacts, feasibility, etc.)? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 85

Outline • Is a more embodied way of controlling interaction actually amplifying experience? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 86

Results • EMG is suitable for motion contexts. • Both EMG and self-report data indicate that natural body movement intensifies emotions. • It’s crucial that body movements are in accordance with activity. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 87

Thank you schrammel@cure.at Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 88

Reality-Based Brain- Computer Interaction Daniel Sjölie Umeå University, Umeå, Sweden

Reality-Based Brain-Computer Interaction • Complex, reality-based, interaction with the ability to adapt to the user in real-time – Using brain measurements to evaluate • Direct brain-computer interaction – The user can be unaware • We’re using a combination of virtual reality (VR) and functional MRI (fMRI) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 90

HCI and cognition • An improved understanding of the brain in this interactive and complex context is needed • Build on theories of cognition as a foundation for the development of advanced HCI – As interaction moves towards being reality-based we move closer to “interacting with reality”, and thus to general cognition • Cognitive neuroscience and computational models of the brain are important tools Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 91

Working with fMRI and Virtual Reality Working with fMRI requires specialist competence and forces us to deal with powerful magnetic fields and restricted mobility, but the ability to measure the whole brain is an invaluable research tool. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 92

Thank you Daniel Sjölie, daniel@cs.umu.se Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 93

emoBAN: Improving Quality of Life via Psychophysiological Mobile Computing Katarzyna Wac, Anind K. Dey Carnegie Mellon University, HCII

emoBAN idea • “what always speaks silently is the body” (Brown, 1990) – measure QoL via psychophysiology Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 95

emoBAN: Usable Useful Functional • Body Area Network – sensors • vital signs, e.g., ECG, temp, resp • context, e.g., location, time, activity – actuators • feedback via, e.g., AV, tactile, light Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 96

Thank you Dr Katarzyna Wac Carnegie Mellon University Human-Computer Interaction Institute katewac@cmu.cs.edu www.cui.unige.ch/~wac Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 97

Leveraging Psychophysical Data in Monitoring and Analyzing the States of Badminton Players Feng Tian, Wencan Luo, Jing Dai, Reza Naima, Yongmin Cheng, Hongan Wang, Guozhong Dai Institute of Software, Chinese Academy of Sciences Physical Education Collage, Beijing Sport University Berkeley Institute of Design, UC. Berkeley China Badminton Team, General Administration of Sport of China 4/20/2010

Traces and physical actions - Camera, accelerometers Psychophysical data - Berkeley Tricorder ECG, EMG, Pulse Oximetry, BioImpedance Respiration Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010

Exploring Quantitative Methods for Evaluating Sports Games Veronica Zammitto Simon Fraser University Magy Seif El-Nasr Simon Fraser University Paul Newton Electronic Arts Canada

Current Study: NBA Live 10 • Eye tracking – Eye Tech TM3 • Psycho-physiological data – ProComp5 Infiniti – EMG (corrugator supercilii and zygomatic major), EKG (heart rate), GSR and BVP • Gameplay Telemetry Data – event-based RPC telemetry collection system • Triangulation Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 102

What was the user the Data Triangulation - NBA Live 10 emotional response? looking at? Game user experience Event, when Was positive? Where user’s Was negative? attention was? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 103

Data Triangulation - NBA Live 10 Game user experience Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 104

Thank you Veronica Zammitto vzammitt@sfu.ca Magy Seif El-Nasr magy@sfu.ca Paul Newton pnewton@ea.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 105

Exploring Psycho-physiological Measures for the Design and Behavior of Socially-Aware Avatars in Ubicomp Environments Johanna Renny Octavia Hasselt University, Belgium

Avatars in Ubicomp Environment  User’s representation  Mediator between system and user (companion)  To design a socially-aware avatar that adapts its behavior according to user’s emotion  Any incorrect actions of the avatar can lead to loss of identification of the user Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 107

Our informal experiment  ProComp Infiniti device, measured GSR, EMG and BVP  Higher physiological values when playing against a friend  How different social relationships between players influence those values? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 108

Thank you Expertise Centre for Digital Media (EDM) Web: www.edm.uhasselt.be Email: johanna.octavia@uhasselt.be Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 109

Tailoring Practice for Individualized Learning via Cognitive State Bob Wray Soar Technology, Inc.

Why I need Psychophysiological Computing • Practice environments are becoming common-place • Adaptive tailoring of practice is important for individualized experience/learning • Need more than traditional adaptations (e.g., scaffolding) – Adapting to engage learner / maintain interest – Adaptation to reducing “gaming” in environment – Adaptation to elicit learner affect/arousal (“real”) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 111

Research Testbed Narrative, conversational, inter-cultural training application Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 112

Thank you Bob Wray Soar Technology wray@soartech.com, 919.967.5079 Brian Magerko Georgia Institute of Technology magerko@git.edu Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 113

Guidelines to biosignal driven HCI Marjolein van der Zwaag Philips Research Europe, University of Groningen

Guidelines to biosignal driven HCI Marjolein van der Zwaag Egon L. van den Broek Joris H. Janssen

Outline • Signal(s) measured: psycho-physiological • Application domain: Affective computing • Device used: Nexus 10, mind media • Guidelines for processing biosignals – Triangulation – Identification of users – Biosignal characteristics – Temporal construction Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 116

Unobtrusive affective computing Music direct your mood concept Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 117

Thank you Marjolein.van.der.zwaag@philips.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 118

Workshop Discussion Questions What is brain/body computing good for? How can we think systematically about this space? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 120

Workshop Discussion Questions What are the major themes of HCI that can be explored using brain/body interaction? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 121

Workshop Discussion Questions What specific brain & body sensing modalities are interesting and how can we use them? What are most innovative technologies in this space? How do these compare with existing practices (NASA-TLX, Likert-style questionnaires, etc)? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 122

Workshop Discussion Questions How can BCI and PC be defined within an HCI context? How is a HCI context different from neuropsychology, biomedicine or biomedical engineering? How does HCI contribute to psychophysiological computing (or more specifically neural engineering, bio- engineering, etc)? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 123

Workshop Discussion Questions If we could measure anything on brain and body, what would we want to measure and why? What cognitive states and emotions have been investigated with sensors? What methodologies are best for what signals? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 124

Workshop Discussion Questions How can accurate physiological, psychological and neurophysiological measurements improve HCI? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 125

Workshop Discussion Questions With what software techniques and hardware technologies could these signals be optimally processed? How can software account for the combination of user actions taken to accomplish a task versus actions that would surprise the user, yet optimize their experience? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 126

Workshop Discussion Questions What are the issues that must be considered when using brain/body computing for direct control, passive sensing, adaptation, etc? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 127

Groups • How to go from raw data to something else – Emotions: TABLE 1 – Data Fusion: TABLE 4 • How to use these things in interactive interfaces: TABLE 3 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 128

BBB Workshop: Bringing Together BCI& Physiological Computing Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 129

Staying connected Website http://www.eecs.tufts.edu/~agirou01/workshop/ Twitter http://twitter.com/bbbCHI2010 Twitter #bbbCHI2010 Hashtag Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and- Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786 Flickr http://www.flickr.com/groups/bbbchi/ Grouplist http://groups.google.com/group/brain-body-interfaces/ Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological- user-interaction Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 130

BBB at CHI2010 Brains and Brawn session: Tuesday 11:30-1pm Making Muscle-Computer Interfaces More Practical T. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, James A. Landay University of Washington, USA Microsoft Corporation, USA A Novel Brain-Computer Interface Using a Multi-Touch Surface Beste F. Yuksel, Michael Donnerer, James Tompkin, Anthony Steed. University College London, UK The Influence of Implicit and Explicit Biofeedback in First-Person Shooter Games Kai Kuikkaniemi, Toni Laitinen, Marko Turpeinen, Timo Saari, Ilkka Kosunen, Niklas Ravaja, Helsinki Institute for Information Technology, Finland Center for Knowledge and Innovation Research, Finland Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual Environment Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, Lars Nyberg Umeå University, Sweden Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 131

THANK YOU • Keep in touch! Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 132

http://www.eecs.tufts.edu	448
http://www.acagamic.com	39
http://www.slideshare.net	22
http://translate.googleusercontent.com	5

Brain, Body, and Bytes CHI 2010 Workshop Presentations

by Lennart Nacke on Apr 20, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 514

Statistics

Brain, Body, and Bytes CHI 2010 Workshop Presentations — Presentation Transcript