How to Do Multimodal Detection      of Affective States?      Javier Gonzalez-Sanchez, Maria-Elena Chavez-Echeagaray,     ...
MotivationJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
MotivationJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Agenda1. Concepts and Theory.2. Sensing Devices.3. Software Framework, Data Filtering and Integration.4. Analyzing Data.5....
I. Concepts and Theory
Concepts          How to Do Multimodal Detection                of Affective States?physiological   physical             i...
Concepts physical appearance                                                                                   measurement...
TheoryJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Theory                        !"#$%&(#)*%&$&+),*-&.+/$&+)0102*%)        =*+0/+.))                       ;*,-*:$&+)        ...
2. Sensing Devices
BCIWireless Emotiv® EEG Headset.The device reports data with intervals of 125 ms (8 Hz).The output includes 14 values (7 c...
emotions                  EEG data                 facial gestures           DemoWireless Emotiv® EEG Headset             ...
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
EEG data           This work was supported by Office of Naval Research under Grant N00014-10-1-0143
emotions           This work was supported by Office of Naval Research under Grant N00014-10-1-0143
Sensing DevicesTobii® Eye TrackerThe device reports data with intervals of 100 ms (10Hz).The output provides data concerni...
About             Tobii® Eye TrackerThis work was supported by Office of Naval Research under Grant N00014-10-1-0143
Sensing DevicesMindReader Software from MIT Media Lab.It infers affective states from head gestures and facialexpressions ...
DemoMindReader Software from MIT Media Lab       This work was supported by Office of Naval Research under Grant N00014-10-...
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
Sensing DevicesHardware designed by MIT Media Lab.It measures Arousal.It is a skin electrical conductance sensor to measur...
DemoSkin electrical conductance Sensor   This work was supported by Office of Naval Research under Grant N00014-10-1-0143
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
Sensing DevicesHardware designed by MIT Media Lab.Pressure SensingIt is a Serial device that reports data in intervals of ...
Demo     Mouse Pressure SensorThis work was supported by Office of Naval Research under Grant N00014-10-1-0143
This work was supported by Office of Naval Research under Grant N00014-10-1-0143
Sensing Devices                                                                                  !"#$%&(#)*%&$&+),*-&.+/$&...
3. Software Framework, Data   Filtering and Integration
Framework                                                                                                      40 students...
Framework Agent Federation* B. Horling, and V. Lesser, “A survey of multi-agent organizational paradigms,” The Knowledge E...
FrameworkAgentFederation Gonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Ag...
FrameworkGonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Softwa...
4. Analyzing Data: Tools and        Techniques
Tool: Weka                                 explore               clasification                   clustering                ...
Tool: WekaThe experiment wasrun over 21 subjects,undergrad and gradstudents of ArizonaState Universityranging between 18 t...
Tool: WekaJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Tool: Eureqa                                            mathematical relationships in dataDubcˇa ́kova ́, R. Eureqa-softwa...
Tool: EureqaJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Techniques: Netsengagement and EEG raw       !"#$%"$#&()*$&+,-./(012&%3-%4(5&"#673.(                 >#&6-(.%=3?&+%/(.=,96...
Tool: ABE                                      gaze-x, gaze-y, time                                                       ...
Technique:Sparse LearningJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
5. Sharing Experiences
Experiencesnon- invasive devices     setup time                        Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeag...
Experiences movility | interference       data analysis                                                       Third-      ...
ExperiencesJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
6. Conclusions
Reference AVirtual Worlds Best Practices in Education 2011 Conference                  http://javiergs.com?p=1317         ...
Reference B8th Annual Games for Change Festival        http://javiergs.com?p=1433  Javier Gonzalez-Sanchez | Maria-Elena C...
http://lsrl.lab.asu.edu          {javiergs, helenchavez}@asu.eduThis work was supported by Office of Naval Research under G...
Upcoming SlideShare
Loading in...5
×

201107 ICALT

1,545

Published on

Considering that the human-element as crucial in designing and implementing interactive intelligent systems, this tutorial provides a description and hands-on demonstration on detection of affective states and a description of devices, methodologies and data processing, as well as their impact in instructional design. The information that a computer senses in order to automate the detection of affective states, includes an extensive set of data, it could ranges from brain-waves signals and biofeedback readings from face-based or gesture emotion recognition and posture or pressure sensing. The work presented in this tutorial, is not about the development of the algorithms or hardware that make this works, our concerns are about the encapsulation of preexisting systems (we are actually using all of them) that implements those algorithms and uses these hardware to improve Learning.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,545
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

201107 ICALT

  1. 1. How to Do Multimodal Detection of Affective States? Javier Gonzalez-Sanchez, Maria-Elena Chavez-Echeagaray, David Gibson, Robert Atkinson, Winslow Burleson Learning Science Research Lab School of Computing, Informatics, and Decision Systems Engineering Arizona State University This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  2. 2. MotivationJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  3. 3. MotivationJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  4. 4. Agenda1. Concepts and Theory.2. Sensing Devices.3. Software Framework, Data Filtering and Integration.4. Analyzing Data.5. Sharing Experiences.6. Conclusions. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  5. 5. I. Concepts and Theory
  6. 6. Concepts How to Do Multimodal Detection of Affective States?physiological physical instinctual reaction to stimulation feelings, emotions How do you feel? Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  7. 7. Concepts physical appearance measurementphysiological measures identify the presence of ... self-report How to Do Multimodal Detection of Affective States? Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  8. 8. TheoryJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  9. 9. Theory !"#$%&(#)*%&$&+),*-&.+/$&+)0102*%) =*+0/+.)) ;*,-*:$&+) ?+2*.,($&+) >*6/-*0) %*-<(+/0%0) @#.&,/2<%) 4,(/+5(6*0) 71*)%&6*%*+20) 8(-/(#)*9:,*00/&+0) ;<10/&#&./-(#)0/.+(#0)30*,) A(5)(2() 4*#/*B0) =2(2*) Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  10. 10. 2. Sensing Devices
  11. 11. BCIWireless Emotiv® EEG Headset.The device reports data with intervals of 125 ms (8 Hz).The output includes 14 values (7 channels on each brainhemisphere: AF3, F7, F3, FC5, T7, P7, O1, O2, P8,T8, FC6, F4, F8, and AF4) and two values of theacceleration of the head when leaning (gyrox and gyroy).This report Engagement, Boredom, Excitement,Frustration, Meditation.And also facial activity: blink, wink (left and right), look(left and right), raise brow, furrow brow, smile,clench, smirk (left and right), and laugh. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  12. 12. emotions EEG data facial gestures DemoWireless Emotiv® EEG Headset Emotiv Systems $299 This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  13. 13. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  14. 14. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  15. 15. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  16. 16. EEG data This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  17. 17. emotions This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  18. 18. Sensing DevicesTobii® Eye TrackerThe device reports data with intervals of 100 ms (10Hz).The output provides data concerning attention direction(gaze-x, gaze-y), time of focus and pupil dilation. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  19. 19. About Tobii® Eye TrackerThis work was supported by Office of Naval Research under Grant N00014-10-1-0143
  20. 20. Sensing DevicesMindReader Software from MIT Media Lab.It infers affective states from head gestures and facialexpressions in a video stream in real-time at data intervalsof 100 ms approximately (10 Hz).With this system it is possible to infer: agreeing,concentrating, disagreeing, interested, thinkingand unsure. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  21. 21. DemoMindReader Software from MIT Media Lab This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  22. 22. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  23. 23. Sensing DevicesHardware designed by MIT Media Lab.It measures Arousal.It is a skin electrical conductance sensor to measures theelectrical conductance of the skin, which varies with itsmoisture level that depends on the sweat glands, which arecontrolled by the sympathetic, and parasympathetic nervoussystems.It is a Wireless Bluetooth device that reports data in intervalsof 500 ms approximately (2Hz). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  24. 24. DemoSkin electrical conductance Sensor This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  25. 25. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  26. 26. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  27. 27. Sensing DevicesHardware designed by MIT Media Lab.Pressure SensingIt is a Serial device that reports data in intervals of 150 msapproximately (6Hz). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  28. 28. Demo Mouse Pressure SensorThis work was supported by Office of Naval Research under Grant N00014-10-1-0143
  29. 29. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  30. 30. Sensing Devices !"#$%&(#)*%&$&+),*-&.+/$&+)0102*%) =*+0/+.)) ;*,-*:$&+) ?+2*.,($&+) >*6/-*0) %*-<(+/0%0) @#.&,/2<%) 4,(/+5(6*0) 71*)%&6*%*+20) 8(-/(#)*9:,*00/&+0) ;<10/&#&./-(#)0/.+(#0) 30*,) A(5)(2() 4*#/*B0) =2(2*)Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  31. 31. 3. Software Framework, Data Filtering and Integration
  32. 32. Framework 40 students independently Data Logger Agent Data Visualizer Agent Agent CentreMultimodal Tutoring System 37 student concurrently Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  33. 33. Framework Agent Federation* B. Horling, and V. Lesser, “A survey of multi-agent organizational paradigms,” The Knowledge Engineering Review, Cambridge University Press, 2005, vol. 19, pp. 281-316, doi: 10.1017/S0269888905000317. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  34. 34. FrameworkAgentFederation Gonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Software Architecture for a Multimodal Emotion Recognition Framework," in Proceedings of Working IEEE/IFIP Conference on Software Architecture (June 2011). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  35. 35. FrameworkGonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Software Architecturefor a Multimodal Emotion Recognition Framework," in Proceedings of Working IEEE/IFIP Conference on Software Architecture (June2011). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  36. 36. 4. Analyzing Data: Tools and Techniques
  37. 37. Tool: Weka explore clasification clustering pre-processing visualizationMark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H. Witten (2009); The WEKA Data Mining Software:An Update; SIGKDD Explorations, Volume 11, Issue 1. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  38. 38. Tool: WekaThe experiment wasrun over 21 subjects,undergrad and gradstudents of ArizonaState Universityranging between 18 to25 years. For thepurpose of ourexperiment weconsider all levels ofexpertise from noviceto expert users ofGuitar Hero, we alsoconsider regular andno regular gamers , andwe also consider bothgenders. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  39. 39. Tool: WekaJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  40. 40. Tool: Eureqa mathematical relationships in dataDubcˇa ́kova ́, R. Eureqa-software review. Genetic programming and evolvable machines. Genet. Program. Evol. Mach. (2010) online first.doi:10.1007/s10710- 010-9124-z . Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  41. 41. Tool: EureqaJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  42. 42. Techniques: Netsengagement and EEG raw !"#$%"$#&()*$&+,-./(012&%3-%4(5&"#673.( >#&6-(.%=3?&+%/(.=,96-@("=3(%=&--3.( &-1(83"9,#:.(;#&<=.( "=&"(%,-"#6A$"3(96"=(3-@&@3?3-"( ! !* Dr. David C. Gibson Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  43. 43. Tool: ABE gaze-x, gaze-y, time frustration threshold = 0.75Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  44. 44. Technique:Sparse LearningJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  45. 45. 5. Sharing Experiences
  46. 46. Experiencesnon- invasive devices setup time Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  47. 47. Experiences movility | interference data analysis Third- party The experiment was run Systemsover 21 subjects, undergradand grad students of ArizonaState University rangingbetween 18 to 25 years. For gaze-x, gaze-y, timethe purpose of ourexperiment we consider alllevels of expertise fromnovice to expert users of frustration threshold = 0.75Guitar Hero, we also consider regular and no regular Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  48. 48. ExperiencesJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  49. 49. 6. Conclusions
  50. 50. Reference AVirtual Worlds Best Practices in Education 2011 Conference http://javiergs.com?p=1317 Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  51. 51. Reference B8th Annual Games for Change Festival http://javiergs.com?p=1433 Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  52. 52. http://lsrl.lab.asu.edu {javiergs, helenchavez}@asu.eduThis work was supported by Office of Naval Research under Grant N00014-10-1-0143

×