This research studied the emotional impacts of digital media by collecting biometric data from 20 participants playing Guitar Hero. Sensors measured electroencephalography (EEG), eye tracking, galvanic skin response, head motion and facial expressions. The data was analyzed using structural equation modeling and network analysis to identify patterns in physical, emotional and cognitive attributes during easy and hard songs. Near real-time displays and time slice analyses provided operational models of brain states during tasks. The findings suggest these methods could be used to personalize media experiences, study emotion and learning, and improve affective tutoring systems.

1. Emotional Impacts of Digital MediaRobert Atkinson Maria-Elena Chavez-EcheagarayRobert M. ChristophersonDavid Gibson Javier Gonzalez-SanchezThis research was supported by Office of Naval Research under Grant N00014-10-1-0143 awarded to Dr. Robert Atkinson

2. Research QuestionsAt what moments does a digital media user experience identifiable physical, emotional and cognitive attributes?Are those facilitating or impeding moment-by-moment learning and performance?

3. Research QuestionsWhat patterns are found within & between sensors?How do these patterns relate to baseline and experimental activities?

4. SensorsWireless EEGFacial muscles, emotional clusters, raw EEGWireless Galvanic Skin ConductanceArousal levelEye TrackerGaze-point, duration, mouse-clicksHapticsButton presses, head tilt

5. Anatomy of the System

6. Anatomy of theSystemWireless Emotiv® EPOC Headset(report data with intervals of 125 ms). The output includes 14 sensors or channels (7 on each brain hemisphere: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4) and two values of the acceleration of the head when leaning (gyrox and gyroy). Also it reports Engagement, Boredom, Excitement , Frustration, Meditation.Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray

7. Anatomy of theSystemTobii®Eye Tracker report data with intervals of 100 ms.provides data concerning attention direction and time of focus during individual use of a computer.Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray

8. Anatomy of theSystemMindReader Software from MIT Media Lab. This is about inferring a person mental state from non-verbal cues. Visual system infers mental states from head gestures and facial expressions in a video stream in real-time at data intervals of 100 ms approximately. This system infer six emotions: agreeing, concentrating, disagreeing, interested, thinking and unsure.Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray

9. Anatomy of theSystemSkin electrical conductance sensordesigned by MIT Media Lab for Arousal Sensing. Reports conductance data in intervals of 500 msThis sensor measures the electrical conductance of the skin, which varies with its moisture level that depends on the sweat glands, which are controlled by the sympathetic, and parasympathetic nervous systems.Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray

10. Method20 users play Guitar Hero sessionsEasy and hard songs10 minutes of recording per sessionOver 76000 records per user per session

11. Video

12. Data

13. Thinking StatesRise inuncertainty and interestDuring thinkingAgreement & concentration drop

14. AnalysisInductive structural equation modeling

15. AnalysisNetwork analysisAdjacency tablesCentralityDigraphs

16. AnalysisDigraphs illustrate structural relationships in the causative factors during a time slice.

17. FindingsNear real-time displays are possible that relate to hidden psychological variablesTime slices of performance can be analyzed to give statistically accurate operational models of brain states during specific tasks