This presentation was done during the SUNY Oswego Technology Conference in 2013 to showcase the utilization of the Microsoft Kinect for education research.

1. Techniques for Recording and
Analyzing Posture and Gesture as
a Means of Inferring Students’
Emotional States
1

2. Learning & Emotion Lab
• Focuses on uncovering the relationships between students;
learning and their emotional (i.e., affective states. The
research goals include refining psychological theory and
developing educational applications, such as emotionally
adaptive learning environments.
• Dr. Roger S. Taylor
• Zachary Bradley
• Matthew Doyle
2

3. 3
Measuring Affect: Through Facial
Expression - Boredom
AU 27 + 64 – Mouth Stretch
+ Eyes Down
AU 43 + 64 – Eye Closure
+ Eyes Down
Neutral AU 27 AU 64AU 43 AU 55

4. 4
Examples of Facial Expression
Boredom
Happiness/Excitement
Confusion
Calmness/Neutral

5. 5
Affect Sensors: New Setup
• Facial Displays
• Webcam
• Human & Computer Coding
• Posture
• Replace Pressure Sensitive Chair with Kinect
• Affect Map
• Java Implementation
• Mobile Devices (hopefully coming soon)

6. Current Research Projects
Posture Analysis
• Determining students’ emotional states through
their postures.
Facial Analysis
• Determining students’ emotional states through
their facial expressions.
Self-Report Assessment
• Determining students’ emotional states through
the Affect Map self-report instrument.
6

7. Utilizing the Kinect Motion
Sensing Input Device to Record
Students Affective States
7

8. Assessment Techniques
• Internal - Observations Reference Frame –
Self-reports – Affect Map
• External - Observations Reference Frame –
Behavioral Traces – Kinect Device
8

9. 9
Affect: Theoretical Background
• Core Affect Theory (Russell, 1980, 2003; Russell & Barrett, 1999
• Dimension 1: Activation Represented vertically, with higher levels of energy
toward the top and lower levels of energy toward the bottom
• Dimension 2: Valence Represented horizontally, with “positive” feelings
(pleasant) toward the right and “negative” feelings (unpleasant) toward the left

10. 10
• Measurement about every minute
• Activation: Level of Energy (Vertical)
• Valence: Level of Pleasure (Horizontal)
Affect Map
High Activation (+)
Low Activation
LowValence
HighValence(+)

11. Kinect Device
• Motion sensing input device
• Records participants postures and gestures
and exports joint positional coordinates X,
Y & Z to a .CSV
11

12. Kinect
12

13. Methods
• Participants: 37 SUNY Oswego
undergraduates, 2 males, 35 females
• Materials: Algebraic Problems; levels of
difficulty = Easy, Medium & Hard
• Procedure: Participants attempted to answer as
many algebraic equations as possible in one half
hour.
13

14. Open Sesame
14

15. Data Visualization
• Students’ emotional states over time
• Distance from screen (meters)
• Emotive State (Valence | Activation)
• Problem difficulty (Easy | Medium | Hard)
15

16. Deducer Output
16

17. Thank you!
If you have any questions, please contact:
Dr. Roger S. Taylor|
roger.taylor@oswego.edu
Matthew C. Doyle|
mdoyle@oswego.edu
Or visit:
www.learningandemotionlab.org
17