Brain, Body, and Bytes CHI 2010 Workshop Presentations

5,450 views

Published on

Brain and Body Interfaces (BBI) were discussed in this workshop. Its goal was to provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI). Find out more at the workshop website: http://www.eecs.tufts.edu/~agirou01/workshop/

Published in: Education
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,450
On SlideShare
0
From Embeds
0
Number of Embeds
1,052
Actions
Shares
0
Downloads
103
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Brain, Body, and Bytes CHI 2010 Workshop Presentations

  1. 1. BBB Workshop Overview Audrey Girouard, Erin Treacy Solovey, Regan Mandryk, Desney Tan, Lennart Nacke, Robert J.K. Jacob
  2. 2. BBB Workshop: Bringing Together BCI& Physiological Computing Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 3
  3. 3. Workshop Goals 1. Provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI). 2. Kick-start BBB research in HCI by identifying key research questions and application areas 3. Enhancing HCI methodologies by adding these new methods and techniques to the toolbox of HCI researchers Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 4
  4. 4. About Us Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 5
  5. 5. About you • 25 Participants • X EEG papers • Y fNIRS papers • Z EMG papers • 9 Brain papers • A GSR • 19 Body papers • B Eye tracking Brain Body 32% 68% Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 6
  6. 6. Workshop Discussion Questions <on your tables> Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 7
  7. 7. Workshop Discussion Questions What terms/acronyms should we use to describe our work? Psychophysiological computing seems a bit long… BBB? BBI? Other suggestions? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 8
  8. 8. Today’s Schedule 09:00-10:30 Introduction of organizers & participants, CHI madness style 10:30-11:00 Break 11:00-12:30 Group Discussion (assign groups + topic) 12:30-14:00 Lunch 14:00-15:30 Small group discussion (1h of discussion + 30 minutes of reporting) 15:30-16:00 Break 16:00-17:00 Report from small groups, recap, group discussion – barriers to publication in HCI Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 9
  9. 9. Staying connected Website http://www.eecs.tufts.edu/~agirou01/workshop/ Twitter http://twitter.com/bbbCHI2010 Twitter #bbbCHI2010 Hashtag Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and- Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786 Flickr http://www.flickr.com/groups/bbbchi/ Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological- user-interaction Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 10
  10. 10. And now… 2 Minute Introductions • About your research: – Application or domain space? (e.g. video games) – Signal you are measuring? (e.g. fNIRS) – Device/hardware you are using? (e.g. ISS, Inc) • What would you like to get out of workshop? – What problems are you facing? – What do you see in common with others here? – What device would you like to know more about? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 11
  11. 11. List of Presentations 1. Adriane Randolph 16. Katarzyna Wac 2. Krzysztof Z. Gajos 17. Feng Tian 3. Saraswathi Bellur 18. Veronica Zammitto 4. Danny Plass-Oude Bos 19. Johanna Octavia 5. Kai Kuikkaniemi 20. Bob Wray 6. Mitchel Benovoy 21. Marjolein van der Zwaag 7. Yee Chieh Chew 8. Martha Crosby 9. Jan Kallenbach 10. Eva Oliveira 11. Lucia Filgueiras 12. Marc Grootjen 13. Melody Moore Jackson 14. Johann Schrammel 15. Daniel Sjölie Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 12
  12. 12. Brain-Computer Interfaces as Child’s Play: Using Mental Training to Improve Motor Skills in Children Adriane B. Randolph, Ph.D. Ashley Ingraham Kennesaw State University
  13. 13. Outline • Research Position – Examine ties between sensorimotor area and physical ability to uncover a training tool to aid children’s motor skill development • Pilot Study – 3 able-bodied adults (2 Male, 1 Female) – Acquire baseline Wii tennis score – Conduct mental training with BCI2000 mu training protocol for 5 days – Retest Wii tennis score for improvements Participant Session One Session Five – Mixed results AZ 98 60 CM 70 64 MW 103 118 • Future Work – Extend study with 20 able-bodied adults (currently in progress) – Analyze link between training and baseline ability – Test with elementary-aged children Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 14
  14. 14. Screening for Mu Ashley prepares participant to use mental imagery of hand and foot movements to determine his best channels for use in mu BCI training before he plays a match of Wii Tennis. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 15
  15. 15. Thank you Adriane B. Randolph, Ph.D. Kennesaw State University BrainLab (770) 423-6083 brainlab@kennesaw.edu http://coles.kennesaw.edu/brainlab/ Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 16
  16. 16. Understanding How to Design Complex Brain-Controlled Applications Krzysztof Z. Gajos
  17. 17. How to Design Brain-Controlled Applications? • Basic operations Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 18
  18. 18. How to Design Brain-Controlled Applications? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 19
  19. 19. How to Design Brain-Controlled Applications? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 20
  20. 20. How to Design Brain-Controlled Applications? • Basic operations Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 21
  21. 21. How to Design Brain-Controlled Applications? • Interaction between primary task and BCI performance Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 22
  22. 22. How to Design Brain-Controlled Applications? • Error prevention, recovery, and actions with side effects Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 23
  23. 23. Thank you Krzysztof Gajos kgajos@eecs.harvard.edu http://www.eecs.harvard.edu/~kgajos/ Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 24
  24. 24. Psychophysiological Responses to Media Interfaces Saraswathi Bellur S. Shyam Sundar Media Effects Research Lab http://www.psu.edu/dept/medialab Pennsylvania State University
  25. 25. Biopac MP 35 & MP 150 ACQknowledge software  Skin Conductance (EDA)  Heart Rate (ECG)  Facial EMG  Brain Wave (EEG)  Eye Movement (EOG)  Application Areas Interface presentation Information Processing Usability Testing & Evaluation  User Experience Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 26
  26. 26. Physiological Measures: Affect, Arousal & Attention 3. ECG: Heart Rate recording 1. EEG: Bipolar recordings Occipital Alpha Blocking Orienting Response via BPM change 2. Skin Conductance (EDA) Tonic arousal levels 4. Facial EMG: Corrugator & Zygomatic recordings Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 27
  27. 27. Theoretical Frameworks • The Media Equation: Mediated Life = Real Life (Reeves & Nass, 1996). • Structural aspects of media technologies: Automatic or Controlled processing? Voluntary or Involuntary responses? • Specific cues triggered via interactions with four types of media affordances, MAIN model (Sundar, 2008) Animation M: MODALITY Customization Heuristics A: AGENCY Information Perceptual Scent Bandwidth I: INTERACTIVITY N: NAVIGABILITY Contingency UGC & Sourceness Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 28
  28. 28. Thank you Saras Bellur (saras@psu.edu) S. Shyam Sundar (sss12@psu.edu) Media Effects Research Lab http://www.psu.edu/dept/medialab Pennsylvania State University Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 29
  29. 29. BCI for Games Danny Plass-Oude Bos Boris Reuderink, Bram van de Laar, Hayrettin Gürkök, Christian Mühl, Mannes Poel, Anton Nijholt, Dirk Heylen HMI, University of Twente
  30. 30. Target Group: the general population Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 31
  31. 31. BCI: issues Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 32
  32. 32. BCI: new information, new features Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 33
  33. 33. Acceptance: Usability & Realistic Settings Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 34
  34. 34. Subjects  Users Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 35
  35. 35. Thank you d.plass@ewi.utwente.nl http://hmi.ewi.utwente.nl/~oudebos Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 36
  36. 36. Designing Biofeedback for Games and Playful Applications Kai Kuikkaniemi Helsinki Institute for Information Technology
  37. 37. Emoshooter Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 38
  38. 38. Emoshigeru - cinema games Real-time Public display Biosignal adaptation Massive co-located Shooter / Strategy Part of Shigeru-games local wlan Public display Connected to game server Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  39. 39. Presemo – presentation engine Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  40. 40. Emolisten – biosignal sonification Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  41. 41. Research statement • Social Games and Playful Applications, which Utilize Biosignals • Grey Area between Adaptation and Biofeedback – Implicit and Explicit Biofeedback • Biosignal Synchrony (Inter-Personal) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 42
  42. 42. Our team and projects Professor Marko Turpeinen Professor Timo Saari Ilkka Kosunen, Toni Laitinen, Petri Lievonen, Pauli Ojala Fun of Gaming (EU FP6) (2006-2009) Enactive Social Media and Games (Aalto) (2009) Emokeitai (Tekes – National Innovation Agency) (2009-2011) Thanks: CKIR / Niklas Ravaja Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 43
  43. 43. Psychophysiological signal analysis and classification Mitchel Benovoy Centre for Intelligent Machine, McGill University
  44. 44. Outline • Signals • Hardware • Application domains Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 45
  45. 45. Signals Galvanic Skin Response Phalange Temperature 1.5 ∆ Cond. (µS) 1 0.5 0 -0.5 0.5 1 1.5 2 2.5 Sample 4 x 10 Electromyogram 42 Blood Volume Pulse – HR & Amp. Respiration – Rate & Amp. 40 2.2 2 38 Refl. Exp. 1.8 36 1.6 34 1.4 32 1.2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 7 1 0.8 Amp. Amp. 6 0.6 0.4 5 0.2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 105 30 HR (bpm) RR (bpm) 100 25 95 20 90 15 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 46
  46. 46. Hardware • Thought Technology ProCom Infinity Images via www.thoughttechnology.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 47
  47. 47. Application Domains • Affective Computing – Interaction evaluation (UI, devices, latency) – Music & emotion research – Psychoneurology Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 48
  48. 48. Thank you benovoym@cim.mcgill.ca Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 49
  49. 49. BCI and Creativity Yee Chieh (Denise) Chew Georgia Institute of Technology
  50. 50. Outline • fNIR and Creative Expression • EEG and Music Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 51
  51. 51. Character Templates for fNIR Created with a mouse, these are templates that subjects traced using fNIR Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 52
  52. 52. Thank you ychew@gatech.edu Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 53
  53. 53. Physiological Measures Used for Identification of Cognitive States and Continuous Authentication Curtis S. Ikehara & Martha E. Crosby University of Hawaii at Manoa
  54. 54. Outline • Overview of our Research Using Physiological sensors – Assessing Cognitive States – Continuous Authentication • Possible Applications Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 55
  55. 55. Multi-modal Physiological Sensors System Eye Tracker (Gaze Position, Fixation Number, Fixation Duration, Repeat Fixations, Search Patterns, MTF Pupil Size, Blink Rate, Blink Duration) (Moving Target Fractions) HUMAN Pressure Mouse (Pressures on Mouse Button and Body) Laptop (Data Collection, Analysis & Display) EDA / TSA Finger Sensors (Electrodermal and Temperature Sensor Activity) Oximeter (Pulse, Blood Oxygen) Custom Electronics (Analog to Digital Converter) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 56 56
  56. 56. A Composite of Over 20 Mouse Clicks of Three People (Click Signature) US Patent # 7,245,218 Subject 4 Subject 5 Subject 7 57 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  57. 57. Possible Applications • The student’s confidence can be assessed by evaluating the distortions of the mouse click signature. • High Value Vehicle Control – Pressure and other physiological sensors can be attached to a steering (e.g. aircraft yoke or steering wheel) . – Sensors can be used to indicate that: • The authorized pilot or driver is in control. • The cognitive state of the pilot is consistent with the vehicle’s activity. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 58
  58. 58. Thank you Curtis S. Ikehara Martha E. Crosby cikehara@hawaii.edu crosby@hawaii.edu University of Hawaii at Manoa 1680 East-West Road - POST 317 Honolulu, Hawaii 96822 USA Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 59
  59. 59. Measuring Interaction Experiences: Integration of Multiple Psychophysiological Methods Jan Kallenbach Aalto University Helsinki, Finland
  60. 60. Outline User Experience = Choice & Interaction + Cost-Benefit Evaluations Measuring Interaction Experiences?  Software Framework Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 61
  61. 61. Multimodal Measurements Client Application for UX Research Sampling frequency: max 1000Hz Psychophysiological Measures: Gaze, Pupil Size, EDA, EMG, EEG, … Behavioral Measures: Keyboard, Mouse, Slider, … System Measures: Window Events Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 62
  62. 62. Thank you Jan Kallenbach Research Scholar Aalto University School of Science & Technology Department of Media Technology P.O. Box 15500 FIN - 00076 Aalto Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 63
  63. 63. Towards enhanced video access and recommendation through emotions Eva Oliveira Nuno Ribeiro Teresa Chambel IPCA | LASIGE
  64. 64. We Want • To access to videos based on emotions – user actual emotional (psychophysiological) state – video emotional content – director’s perspective • To recommend videos based on emotions – create a set of rules from every affective classification perspective (user and video, director) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 65
  65. 65. Affective video recommendation system requirements () • Every video must be classified affectively from the user and from the content perspective. • Every user has an emotional profile constituted by all his implicit (physiological) classification of every classified movie. • User physiological signals can change the way we watch the videos Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 66
  66. 66. User Classification Issues • Automatic Classification – Users emotional state • emotional model – Classification techniques • Result output • Accuracy • Lab contexts Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 67
  67. 67. Current State • Capturing users phychophysiological signals (BIOPAC System) while watching movies This is my exciting picture • Next step: several classification techniques to convert signal result into emotional categories Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 68
  68. 68. Thank you • Eva Oliveira : • eva.oliveira@gmail.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 69
  69. 69. Your users can’t talk! How can you tell they liked your system? Lucia Filgueiras and Claudia Tambascia Escola Politecnica Universidade de São Paulo
  70. 70. What we want to do • Develop products for cerebral palsy users • Perform participatory design • Understand non-verbal communication Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 71
  71. 71. Direct measures and usability evaluation HRV and workload measures Eye tracking in iTV apps in control room operability evaluation evaluation Comparison of eye tracker NASA TLX, SWAT, Heart Rate findings to conventional Variability usability tests results Variabilidade da frequência cardíaca OP13 LF nu / HF nu LSC Média 18,0 Operador analisa sistema após vários Índice do Balanço Simpático? Parassimpático Ação do operador para alarmes: temperatura alta (10:29); Ação do operador para 16,0 correção nível alto (10:32) e segurança correção 14,0 (10:32; 10:37; 10:43) 14,1 12,0 Alarme de 11,7 temperarura alta: 10,0 10:15 9,1 8,3 8,5 8,0 7,3 6,6 6,0 5,9 5,2 5,3 5,6 4,6 5,0 4,0 3,9 3,8 4,0 4,4 4,1 2,9 3,3 2,6 2,6 2,6 2,3 2,4 2,0 0,0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 :3 :4 :4 :5 :5 :0 :0 :1 :1 :2 :2 :3 :3 :4 :4 :5 :5 :0 :0 :1 :1 :2 :2 :3 :3 09 09 09 09 09 10 10 10 10 10 10 10 10 10 10 10 10 11 11 11 11 11 11 11 11 HORA Heart Rate Variability of an operator in a unstable unit situation Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  72. 72. Our questions: • How personalized are body expressions for cerebral palsy patients? • Are there general body expressions that can mean the same for everyone? • Is it possible to determine a generic way of measuring these expressions? • Is there any other liaison besides gestures and facial expressions , like an unconscious connection of limbic systems that can be responsible for the mutual sensitive communication? • Is there a reference body signal that can be used as a reliable parameter to evaluate body expressions? • How can we establish a correlation between body expressions and patient´s opinion of approval or disapproval of a design solution? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 73
  73. 73. Thank you! Contact us: Lucia V. L. Filgueiras Escola Politecnica, Universidade de São Paulo lucia.filgueiras@poli.usp.br Claudia A.Tambascia Fundação CPqD claudiat@cpqd.com.br http://lts-i.pcs.usp.br Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 74
  74. 74. Human Computer Symbiosis by Mutual Understanding Marc Grootjen EagleScience-TNO
  75. 75. Outline • Symbiosis & mutual understanding • Information models • Interfaces Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 76
  76. 76. Maximum task performance Symbiotic entity Disturbance Mutual human understanding computer Tasks (goals) Human-computer symbiosis with mutual understanding as key issue for maximum task performance Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 77
  77. 77. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 78
  78. 78. Domesticating Brain- Computer Interfaces: Systems for the Home Melody Moore Jackson BrainLab School of Interactive Computing Georgia Tech
  79. 79. Extended Home-based BCI study • fNIR-based BCI for communication • 32 People with ALS (so far) over 1 year • 78% (25) could operate the device on initial visit with accuracy > 70% • Accuracy to 100%, average 76.5% • Issues: – Motivation – Data Collection – Remote Support Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 80
  80. 80. The Kokoro Gatari Brain Computer Interface based on Functional Near Infrared imaging Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 81
  81. 81. Thank you Melody Moore Jackson melody@cc.gatech.edu 404-277-1324 Ian McClendon imclend@cc.gatech.edu 404-641-7712 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 82
  82. 82. Get up, move on! Using Electromyography to Explore the Relationship of Experience and Motion Johann Schrammel Centre for Usability Research and Engineering
  83. 83. Outline • Compare experiences and emotions in conditions with and without movements with Electromyography(EMG) and other methods Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 84
  84. 84. Outline • Can EMG successfully be used to estimate a user‘s emotional experience in motion contexts (artefacts, feasibility, etc.)? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 85
  85. 85. Outline • Is a more embodied way of controlling interaction actually amplifying experience? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 86
  86. 86. Results • EMG is suitable for motion contexts. • Both EMG and self-report data indicate that natural body movement intensifies emotions. • It’s crucial that body movements are in accordance with activity. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 87
  87. 87. Thank you schrammel@cure.at Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 88
  88. 88. Reality-Based Brain- Computer Interaction Daniel Sjölie Umeå University, Umeå, Sweden
  89. 89. Reality-Based Brain-Computer Interaction • Complex, reality-based, interaction with the ability to adapt to the user in real-time – Using brain measurements to evaluate • Direct brain-computer interaction – The user can be unaware • We’re using a combination of virtual reality (VR) and functional MRI (fMRI) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 90
  90. 90. HCI and cognition • An improved understanding of the brain in this interactive and complex context is needed • Build on theories of cognition as a foundation for the development of advanced HCI – As interaction moves towards being reality-based we move closer to “interacting with reality”, and thus to general cognition • Cognitive neuroscience and computational models of the brain are important tools Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 91
  91. 91. Working with fMRI and Virtual Reality Working with fMRI requires specialist competence and forces us to deal with powerful magnetic fields and restricted mobility, but the ability to measure the whole brain is an invaluable research tool. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 92
  92. 92. Thank you Daniel Sjölie, daniel@cs.umu.se Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 93
  93. 93. emoBAN: Improving Quality of Life via Psychophysiological Mobile Computing Katarzyna Wac, Anind K. Dey Carnegie Mellon University, HCII
  94. 94. emoBAN idea • “what always speaks silently is the body” (Brown, 1990) – measure QoL via psychophysiology Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 95
  95. 95. emoBAN: Usable Useful Functional • Body Area Network – sensors • vital signs, e.g., ECG, temp, resp • context, e.g., location, time, activity – actuators • feedback via, e.g., AV, tactile, light Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 96
  96. 96. Thank you Dr Katarzyna Wac Carnegie Mellon University Human-Computer Interaction Institute katewac@cmu.cs.edu www.cui.unige.ch/~wac Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 97
  97. 97. Leveraging Psychophysical Data in Monitoring and Analyzing the States of Badminton Players Feng Tian, Wencan Luo, Jing Dai, Reza Naima, Yongmin Cheng, Hongan Wang, Guozhong Dai Institute of Software, Chinese Academy of Sciences Physical Education Collage, Beijing Sport University Berkeley Institute of Design, UC. Berkeley China Badminton Team, General Administration of Sport of China 4/20/2010
  98. 98. Traces and physical actions - Camera, accelerometers Psychophysical data - Berkeley Tricorder ECG, EMG, Pulse Oximetry, BioImpedance Respiration Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  99. 99. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
  100. 100. Exploring Quantitative Methods for Evaluating Sports Games Veronica Zammitto Simon Fraser University Magy Seif El-Nasr Simon Fraser University Paul Newton Electronic Arts Canada
  101. 101. Current Study: NBA Live 10 • Eye tracking – Eye Tech TM3 • Psycho-physiological data – ProComp5 Infiniti – EMG (corrugator supercilii and zygomatic major), EKG (heart rate), GSR and BVP • Gameplay Telemetry Data – event-based RPC telemetry collection system • Triangulation Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 102
  102. 102. What was the user the Data Triangulation - NBA Live 10 emotional response? looking at? Game user experience Event, when Was positive? Where user’s Was negative? attention was? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 103
  103. 103. Data Triangulation - NBA Live 10 Game user experience Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 104
  104. 104. Thank you Veronica Zammitto vzammitt@sfu.ca Magy Seif El-Nasr magy@sfu.ca Paul Newton pnewton@ea.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 105
  105. 105. Exploring Psycho-physiological Measures for the Design and Behavior of Socially-Aware Avatars in Ubicomp Environments Johanna Renny Octavia Hasselt University, Belgium
  106. 106. Avatars in Ubicomp Environment  User’s representation  Mediator between system and user (companion)  To design a socially-aware avatar that adapts its behavior according to user’s emotion  Any incorrect actions of the avatar can lead to loss of identification of the user Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 107
  107. 107. Our informal experiment  ProComp Infiniti device, measured GSR, EMG and BVP  Higher physiological values when playing against a friend  How different social relationships between players influence those values? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 108
  108. 108. Thank you Expertise Centre for Digital Media (EDM) Web: www.edm.uhasselt.be Email: johanna.octavia@uhasselt.be Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 109
  109. 109. Tailoring Practice for Individualized Learning via Cognitive State Bob Wray Soar Technology, Inc.
  110. 110. Why I need Psychophysiological Computing • Practice environments are becoming common-place • Adaptive tailoring of practice is important for individualized experience/learning • Need more than traditional adaptations (e.g., scaffolding) – Adapting to engage learner / maintain interest – Adaptation to reducing “gaming” in environment – Adaptation to elicit learner affect/arousal (“real”) Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 111
  111. 111. Research Testbed Narrative, conversational, inter-cultural training application Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 112
  112. 112. Thank you Bob Wray Soar Technology wray@soartech.com, 919.967.5079 Brian Magerko Georgia Institute of Technology magerko@git.edu Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 113
  113. 113. Guidelines to biosignal driven HCI Marjolein van der Zwaag Philips Research Europe, University of Groningen
  114. 114. Guidelines to biosignal driven HCI Marjolein van der Zwaag Egon L. van den Broek Joris H. Janssen
  115. 115. Outline • Signal(s) measured: psycho-physiological • Application domain: Affective computing • Device used: Nexus 10, mind media • Guidelines for processing biosignals – Triangulation – Identification of users – Biosignal characteristics – Temporal construction Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 116
  116. 116. Unobtrusive affective computing Music direct your mood concept Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 117
  117. 117. Thank you Marjolein.van.der.zwaag@philips.com Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 118
  118. 118. Workshop Discussion Questions What is brain/body computing good for? How can we think systematically about this space? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 120
  119. 119. Workshop Discussion Questions What are the major themes of HCI that can be explored using brain/body interaction? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 121
  120. 120. Workshop Discussion Questions What specific brain & body sensing modalities are interesting and how can we use them? What are most innovative technologies in this space? How do these compare with existing practices (NASA-TLX, Likert-style questionnaires, etc)? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 122
  121. 121. Workshop Discussion Questions How can BCI and PC be defined within an HCI context? How is a HCI context different from neuropsychology, biomedicine or biomedical engineering? How does HCI contribute to psychophysiological computing (or more specifically neural engineering, bio- engineering, etc)? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 123
  122. 122. Workshop Discussion Questions If we could measure anything on brain and body, what would we want to measure and why? What cognitive states and emotions have been investigated with sensors? What methodologies are best for what signals? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 124
  123. 123. Workshop Discussion Questions How can accurate physiological, psychological and neurophysiological measurements improve HCI? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 125
  124. 124. Workshop Discussion Questions With what software techniques and hardware technologies could these signals be optimally processed? How can software account for the combination of user actions taken to accomplish a task versus actions that would surprise the user, yet optimize their experience? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 126
  125. 125. Workshop Discussion Questions What are the issues that must be considered when using brain/body computing for direct control, passive sensing, adaptation, etc? Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 127
  126. 126. Groups • How to go from raw data to something else – Emotions: TABLE 1 – Data Fusion: TABLE 4 • How to use these things in interactive interfaces: TABLE 3 Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 128
  127. 127. BBB Workshop: Bringing Together BCI& Physiological Computing Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 129
  128. 128. Staying connected Website http://www.eecs.tufts.edu/~agirou01/workshop/ Twitter http://twitter.com/bbbCHI2010 Twitter #bbbCHI2010 Hashtag Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and- Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786 Flickr http://www.flickr.com/groups/bbbchi/ Grouplist http://groups.google.com/group/brain-body-interfaces/ Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological- user-interaction Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 130
  129. 129. BBB at CHI2010 Brains and Brawn session: Tuesday 11:30-1pm Making Muscle-Computer Interfaces More Practical T. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, James A. Landay University of Washington, USA Microsoft Corporation, USA A Novel Brain-Computer Interface Using a Multi-Touch Surface Beste F. Yuksel, Michael Donnerer, James Tompkin, Anthony Steed. University College London, UK The Influence of Implicit and Explicit Biofeedback in First-Person Shooter Games Kai Kuikkaniemi, Toni Laitinen, Marko Turpeinen, Timo Saari, Ilkka Kosunen, Niklas Ravaja, Helsinki Institute for Information Technology, Finland Center for Knowledge and Innovation Research, Finland Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual Environment Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, Lars Nyberg Umeå University, Sweden Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 131
  130. 130. THANK YOU • Keep in touch! Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 132

×