The document discusses a privacy-preserving system named STC-Live designed to detect students' emotional states in online synchronous learning environments. It highlights the importance of non-verbal cues for engagement and the challenges in emotion recognition while maintaining user privacy. The system architecture integrates machine learning to ensure effective monitoring without compromising personal data.

1. PRIVACY-
PRESERVING AND
SCALABLE AFFECT
DETECTION IN
ONLINE
SYNCHRONOUS
LEARNING
Felix Böttger1,
Ufuk Cetinkaya2,
Daniele Di Mitri2,
Sebastian Gombert2,
Krist Shingjergji1,
Deniz Iren1,
Roland Klemke1
1 Open University ofThe Netherlands
2 DIPF | Leibniz Institute for Research and Information in Education, Germany

2. Motivation
■ Most educational institutions
transitioned to online learning during
the pandemic
■ Video conferencing tools and learning
management systems make education
accessible and flexible
■ Communication is unnatural, students
get distracted under isolation and
teachers have a lack of understanding
of how the students are doing.
Zzz…

3. How can we enable teachers to sense the affective states
of the classroom in online synchronized learning
environments in a privacy-preserving way?
Research question:

4. Background
anger
disgust
sadness
happiness
fear
surprise
6 emotions by Ekman
engagement
confusion
anxiety
boredom
anger
frustration
concentration
….
LearningCentered
Affective States
Emotions
Detected
Non-verbal cues:
• gestures
• body postures
• micro-expressions
• ….
Activities
• looking away
• actively listening
• ….

5. Background
Emotions recognition
■ Psycological (self-reporting)
– Questionnaires e.g.,Academic Emotions Questionnaire by Pekrun
– Systems e.g., emot-control
■ Physiological (sensors)
– Signals from skin, heart, etc.
■ Behavioral
– Natural language
– Facial expressions

6. Background
Privacy
■ Data privacy is the most critical factor of users’ trust
■ Data needs to be anonymized as far as possible
■ Privacy in emotion recognition
– Minimize possibility extracting sensitive information from neural
networks
– (debatable) latent vector representations of neural networks

7. Sense the Classroom - Live (STC-Live)
A pragmatic approach to detect
students’ affect in a privacy-
preserving and highly scalable
manner

8. System architecture
Tool
Teacher Side
Tool
Server Side
Tool
Student Side
Presentation
Layer
Application
Layer
Data
Layer
Video
(Webcam)
Server Status
Display
(CLI)
Dashboard
(Information Display and
SessionControl)
Neural
Networks
WebSocket WebSocket WebSocket
Database Interface
WebSocket Processing
Machine Learning
Outputs Database Files Session Data

9. Student-side component
Log in with a key
generated from the
back-end
&
choose a nickname
Periodically take an
image from the
webcam’s video feed
Tool
Student Side
Video
(Webcam)
Neural
Networks
WebSocket
Machine Learning
Outputs

10. Student-side component
Emotion detection
image Detection
of face and
landmarks
faceAPI faceAPI
angry
Detect
prominent
emotion
Tool
Student Side
Neural
Networks
WebSocket
Machine Learning
Outputs
Video
(Webcam)

11. Student-side component
Action Unit detection
image
Detection of face
and landmarks
faceAPI faceAPI
Tool
Student Side
Neural
Networks
WebSocket
Machine Learning
Outputs
Video
(Webcam)
Use eyes’ centers for
alignment
mask face
Generate hogs
AU1
AU2
...
AU28
AU43
Action
Unit
Detection

12. Student-side component
Json file containing:
1. Emotion
2. Timestamp
3. Landmakrs (vector 68x2)
4. Hogs (5408x1)
Tool
Student Side
Neural
Networks
WebSocket
Machine Learning
Outputs
Video
(Webcam)

13. Server back-end component
■ Node.js program, stores the contained
data in a MongoDB database
■ Only host has access to the session
data
■ Able to handle several hundreds of
participants in multiple sessions
Tool
Server Side
Server Status
Display
(CLI)
Database Interface
WebSocket WebSocket
Database Files

14. Teacher’s side
Tool
Teacher Side
Dashboard
(Information Display and
SessionControl)
WebSocket Processing
Session Data

15. Teacher’s side
Tool
Teacher Side
Dashboard
(Information Display and
SessionControl)
WebSocket Processing
Session Data

16. System evaluation
 We created a benchmark scenario with the embedded Machine
Learning Pipeline
 Benchmark: consisted of 1000 executions with a new image
being passed to the pipeline every second
 We collect the time of extraction for emotions, landmarks and
HOG-Values
 AU-Detection is performed on the server-side

17. System evaluation
 Recorded a video clip of a face moving around
 Tested the performance using this pre-recorded video clip2
with varying hardware, operating systems, and browsers
 Tested the platform on all hardware configurations that were
available
 Feasible on lower-end hardware status with interval of one
second
 -> Shorter status on higher-end hardware

18. Results

19. Results

20. Ethical concerns
Students
 Emotions are personal
 Should ask for consent
 Provide transparency
Teachers
 Data not used for monitoring
teachers’ performance
 Not undermine teachers’
independence

21. Limitations
Machine learning algorithms are
subject to different accuracies
Privacy makes it challenging to
evaluate system’s performance
Limited to emotions and facial
expressions (Action Units)
Relevance to education to be studied
Acceptance of teachers and students

22. Conclusion
■ Explore the ways to alleviate the challenges posed by non-verbal
communication limitations of synchronized online learning.
■ STC-live prototype
■ detect students’ affect without transferring images
■ communicate the information to the teachers

23. Thanks for your attention!

24. References
Paul Ekman. “An argument for basic emotions”. In: Cognition & emotion 6.3-4 (1992), pp. 169–200.
Sidney D’Mello. “A selective meta-analysis on the relative incidence of discrete affective states during learning with technology.” In: Journal of Educational Psychology 105.4 (2013), p.
1082.
Reinhard Pekrun et al. “Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative re- search”. In: Educational psychologist
37.2 (2002), pp. 91–105.
Reinhard Pekrun et al. “Achievement emotions and academic performance: Longitudinal models of reciprocal effects”. In: Child development 88.5 (2017), pp. 1653–1670.
Reinhard Pekrun et al. “Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ)”. In: Con- temporary educational psychology 36.1
(2011), pp. 36–48.
Sidney D’Mello and Art Graesser. “Affect detection from human-computer dialogue with an intelligent tutoring system”. In: International Workshop on Intelligent Virtual Agents.
Springer. 2006, pp. 54–67.
Michalis Feidakis, Thanasis Daradoumis, and Santi Caball ́e. “Emotion measurement in intelligent tutoring systems: what, when and how to measure”. In: 2011 Third International
Conference on Intelligent Networking and Collaborative Systems. IEEE. 2011, pp. 807–812.
Michalis Feidakis et al. “Measuring the Impact of Emotion Awareness on e-learning Situations”. In: 2013 Seventh international conference on complex, intelligent, and software
intensive systems. IEEE. 2013, pp. 391– 396.
Maren Scheffel et al. “Quality indicators for learning analytics”. In: Journal of Educational Technology & Society 17.4 (2014), pp. 117–132.
Mimansa Jaiswal and Emily Mower Provost. “Privacy Enhanced Multi- modal Neural Representations for Emotion Recognition”. In: The Thirty- Fourth AAAI Conference on Artificial
Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artifi- cial
Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, 2020, pp. 7985–7993. url: https://ojs.aaai.org/index. php/AAAI/article/view/6307.
Vansh Narula, Kexin Feng, and Theodora Chaspari. “Preserving Privacy in Image-Based Emotion Recognition through User Anonymization”. In: Proceedings of the 2020 International
Conference on Multimodal Interac- tion. New York, NY, USA: Association for Computing Machinery, 2020, 452–460. isbn: 9781450375818. url: https://doi.org/10.1145/3382507. 3418833.
Nicholas Carlini et al. “The Secret Sharer: Evaluating and Testing Un- intended Memorization in Neural Networks”. In: Proceedings of the 28th USENIX Conference on Security
Symposium. SEC’19. Santa Clara, CA, USA: USENIX Association, 2019, 267–284. isbn: 9781939133069.