Your SlideShare is downloading. ×

201107 ICALT

1,522

Published on

Considering that the human-element as crucial in designing and implementing interactive intelligent systems, this tutorial provides a description and hands-on demonstration on detection of affective …

Considering that the human-element as crucial in designing and implementing interactive intelligent systems, this tutorial provides a description and hands-on demonstration on detection of affective states and a description of devices, methodologies and data processing, as well as their impact in instructional design. The information that a computer senses in order to automate the detection of affective states, includes an extensive set of data, it could ranges from brain-waves signals and biofeedback readings from face-based or gesture emotion recognition and posture or pressure sensing. The work presented in this tutorial, is not about the development of the algorithms or hardware that make this works, our concerns are about the encapsulation of preexisting systems (we are actually using all of them) that implements those algorithms and uses these hardware to improve Learning.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,522
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. How to Do Multimodal Detection of Affective States? Javier Gonzalez-Sanchez, Maria-Elena Chavez-Echeagaray, David Gibson, Robert Atkinson, Winslow Burleson Learning Science Research Lab School of Computing, Informatics, and Decision Systems Engineering Arizona State University This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 2. MotivationJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 3. MotivationJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 4. Agenda1. Concepts and Theory.2. Sensing Devices.3. Software Framework, Data Filtering and Integration.4. Analyzing Data.5. Sharing Experiences.6. Conclusions. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 5. I. Concepts and Theory
  • 6. Concepts How to Do Multimodal Detection of Affective States?physiological physical instinctual reaction to stimulation feelings, emotions How do you feel? Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 7. Concepts physical appearance measurementphysiological measures identify the presence of ... self-report How to Do Multimodal Detection of Affective States? Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 8. TheoryJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 9. Theory !"#$%&(#)*%&$&+),*-&.+/$&+)0102*%) =*+0/+.)) ;*,-*:$&+) ?+2*.,($&+) >*6/-*0) %*-<(+/0%0) @#.&,/2<%) 4,(/+5(6*0) 71*)%&6*%*+20) 8(-/(#)*9:,*00/&+0) ;<10/&#&./-(#)0/.+(#0)30*,) A(5)(2() 4*#/*B0) =2(2*) Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 10. 2. Sensing Devices
  • 11. BCIWireless Emotiv® EEG Headset.The device reports data with intervals of 125 ms (8 Hz).The output includes 14 values (7 channels on each brainhemisphere: AF3, F7, F3, FC5, T7, P7, O1, O2, P8,T8, FC6, F4, F8, and AF4) and two values of theacceleration of the head when leaning (gyrox and gyroy).This report Engagement, Boredom, Excitement,Frustration, Meditation.And also facial activity: blink, wink (left and right), look(left and right), raise brow, furrow brow, smile,clench, smirk (left and right), and laugh. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 12. emotions EEG data facial gestures DemoWireless Emotiv® EEG Headset Emotiv Systems $299 This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 13. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 14. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 15. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 16. EEG data This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 17. emotions This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 18. Sensing DevicesTobii® Eye TrackerThe device reports data with intervals of 100 ms (10Hz).The output provides data concerning attention direction(gaze-x, gaze-y), time of focus and pupil dilation. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 19. About Tobii® Eye TrackerThis work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 20. Sensing DevicesMindReader Software from MIT Media Lab.It infers affective states from head gestures and facialexpressions in a video stream in real-time at data intervalsof 100 ms approximately (10 Hz).With this system it is possible to infer: agreeing,concentrating, disagreeing, interested, thinkingand unsure. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 21. DemoMindReader Software from MIT Media Lab This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 22. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 23. Sensing DevicesHardware designed by MIT Media Lab.It measures Arousal.It is a skin electrical conductance sensor to measures theelectrical conductance of the skin, which varies with itsmoisture level that depends on the sweat glands, which arecontrolled by the sympathetic, and parasympathetic nervoussystems.It is a Wireless Bluetooth device that reports data in intervalsof 500 ms approximately (2Hz). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 24. DemoSkin electrical conductance Sensor This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 25. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 26. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 27. Sensing DevicesHardware designed by MIT Media Lab.Pressure SensingIt is a Serial device that reports data in intervals of 150 msapproximately (6Hz). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 28. Demo Mouse Pressure SensorThis work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 29. This work was supported by Office of Naval Research under Grant N00014-10-1-0143
  • 30. Sensing Devices !"#$%&(#)*%&$&+),*-&.+/$&+)0102*%) =*+0/+.)) ;*,-*:$&+) ?+2*.,($&+) >*6/-*0) %*-<(+/0%0) @#.&,/2<%) 4,(/+5(6*0) 71*)%&6*%*+20) 8(-/(#)*9:,*00/&+0) ;<10/&#&./-(#)0/.+(#0) 30*,) A(5)(2() 4*#/*B0) =2(2*)Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 31. 3. Software Framework, Data Filtering and Integration
  • 32. Framework 40 students independently Data Logger Agent Data Visualizer Agent Agent CentreMultimodal Tutoring System 37 student concurrently Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 33. Framework Agent Federation* B. Horling, and V. Lesser, “A survey of multi-agent organizational paradigms,” The Knowledge Engineering Review, Cambridge University Press, 2005, vol. 19, pp. 281-316, doi: 10.1017/S0269888905000317. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 34. FrameworkAgentFederation Gonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Software Architecture for a Multimodal Emotion Recognition Framework," in Proceedings of Working IEEE/IFIP Conference on Software Architecture (June 2011). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 35. FrameworkGonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Software Architecturefor a Multimodal Emotion Recognition Framework," in Proceedings of Working IEEE/IFIP Conference on Software Architecture (June2011). Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 36. 4. Analyzing Data: Tools and Techniques
  • 37. Tool: Weka explore clasification clustering pre-processing visualizationMark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H. Witten (2009); The WEKA Data Mining Software:An Update; SIGKDD Explorations, Volume 11, Issue 1. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 38. Tool: WekaThe experiment wasrun over 21 subjects,undergrad and gradstudents of ArizonaState Universityranging between 18 to25 years. For thepurpose of ourexperiment weconsider all levels ofexpertise from noviceto expert users ofGuitar Hero, we alsoconsider regular andno regular gamers , andwe also consider bothgenders. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 39. Tool: WekaJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 40. Tool: Eureqa mathematical relationships in dataDubcˇa ́kova ́, R. Eureqa-software review. Genetic programming and evolvable machines. Genet. Program. Evol. Mach. (2010) online first.doi:10.1007/s10710- 010-9124-z . Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 41. Tool: EureqaJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 42. Techniques: Netsengagement and EEG raw !"#$%"$#&()*$&+,-./(012&%3-%4(5&"#673.( >#&6-(.%=3?&+%/(.=,96-@("=3(%=&--3.( &-1(83"9,#:.(;#&<=.( "=&"(%,-"#6A$"3(96"=(3-@&@3?3-"( ! !* Dr. David C. Gibson Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 43. Tool: ABE gaze-x, gaze-y, time frustration threshold = 0.75Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 44. Technique:Sparse LearningJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 45. 5. Sharing Experiences
  • 46. Experiencesnon- invasive devices setup time Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 47. Experiences movility | interference data analysis Third- party The experiment was run Systemsover 21 subjects, undergradand grad students of ArizonaState University rangingbetween 18 to 25 years. For gaze-x, gaze-y, timethe purpose of ourexperiment we consider alllevels of expertise fromnovice to expert users of frustration threshold = 0.75Guitar Hero, we also consider regular and no regular Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 48. ExperiencesJavier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 49. 6. Conclusions
  • 50. Reference AVirtual Worlds Best Practices in Education 2011 Conference http://javiergs.com?p=1317 Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 51. Reference B8th Annual Games for Change Festival http://javiergs.com?p=1433 Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
  • 52. http://lsrl.lab.asu.edu {javiergs, helenchavez}@asu.eduThis work was supported by Office of Naval Research under Grant N00014-10-1-0143

×