Citation: Amarasinghe I, Vujovic M, Hernández-Leo D. Towards teacher orchestration load-aware teacher-facing dashboards. In: Proc. CrossMMLA in practice: collecting, annotating and analyzing multimodal data across spaces, 10th International Learning and Analytics Conference (LAK 2020); 2020 Mar 24. Aachen: CEUR; 2020. p. 7-10.

1. Towards Teacher Orchestration
Load-aware Teacher-facing Dashboards
Ishari Amarasinghe, Milica Vujovic and Davinia Hernández-Leo
{ishari. amarasinghe, milica.vujovic, davinia.hernandez-leo}@upf.edu
TIDE, ICT Department, Universitat Pompeu Fabra, Barcelona

2. 2
Introduction
● Orchestration
○ “How a teacher manages, in real time, multi-layered activities in a multi-constraints context” [Dillenbourg, 2013, p. 1]
■ Monitor the situation
■ Decide what adaptations are necessary
■ Perform the adaptations
○ Challenges of TEL practice under the multiple constraints of authentic educational settings [Prieto, et.al. 2018]
● Learning Analytics (LA)
○ “Measurement, collection, analysis and reporting of data about learners and their contexts, for the purposes of
understanding and optimizing learning and the environment in which it occurs” [Ferguson, 2012]
○ Teacher-facing dashboards
■ Deployed within Computer Supported Collaborative Learning (CSCL) environments as a supporting tool
■ Objective : building awareness and facilitating teachers’ productive intervention towards groups that require
immediate attention [van Leeuwen, 2015]

3. 3
Introduction Contd.
● Orchestration
○ “How a teacher manages, in real time, multi-layered activities in a multi-constraints context” [ Dillenbourg, 2013, p. 1]
○ At an abstract level, orchestration can be described as a regulation process
■ Monitor the situation
■ Decide what adaptations are necessary
■ Perform the adaptations
● Learning Analytics (LA)
○ “Measurement, collection, analysis and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the
environment in which it occurs” [ Ferguson, 2012]
○ Teacher-facing dashboards
■ Deployed within CSCL environments as a supporting tool
■ Objective : building awareness and facilitating teachers’ productive intervention towards groups that require immediate attention [van Leeuwen, 2015]
● How the addition of teacher-facing dashboard applications influence orchestration load of the teachers?
● - Design Perspective
● - Orchestration load as an analogy to cognitive load [Prieto, et.al. 2018]

4. 4
Study Design
● Participants
○ 2 female teachers from a Public University in Spain
○ Both experienced in: CSCL and teacher-facing dashboards
○ Students in respective classes participated with informed consent
● Procedure
○ CSCL activities in classrooms were scripted using PyramidApp [Manathunga & Hernández-Leo, 2018]
○ Each teacher conducted 3 pyramid activities (duration: 9 mins. for each activity)
○ Three Conditions:
■ No Dashboard Condition
■ Dashboard Condition
● Dashboard Condition I (with warnings)
● Dashboard Condition II (no warnings)

5. 5
Study Design Contd.
Fig. 1. The information presented in the “Responses Related” tab

6. 6
Study Design Contd.
Fig. 2. The information presented in the “Participation Related” tab

7. 7
Study Design Contd.
Fig. 3. Dashboard Alerts

8. 8
Estimating Orchestration Load using multimodal data
Fig. 4. Data Collection Instruments

9. 9
Physiological measures using EDA
● EDA (also known as galvanic skin response - GSR) reflects the activities that are dependent on emotional and
physiological activation of sympathetic nervous system.
● Detection of changes in affective states
● Changes of the affective state could be triggered by any kind of change in emotional and physiological
reaction

10. ● EDA data was collected using Shimmer3 GSR+ sensor
● Electrodes were placed on the wrist, while sensor was placed in the holder mounted to the teacher’s arm using
armband
● Collected signal was recorder using sensor memory and exported to the computer for further processing
● Skin conductance response data was exported in XLS file as txt and imported in Matlab where low-pass filter was
applied
● Data was plotted and exported for visual inspections
Physiological measures using EDA
Fig. 5. A teacher wearing the Shimmer3 GSR+ sensor during a classroom session (left) and data collection in a co-located collaborative learning setting (right)

11. Initial Findings

12. 12
● Teacher A - Visual inspection of peaks in three conditions
Estimating Orchestration Load using physiological
and subjective measures Contd.

13. 13
Estimating Orchestration Load using physiological and
subjective measures
● Teacher A - No Dashboard Condition
“...Thinking and decision making was somewhat demanding..I am more relaxed when I
use the dashboard and I can monitor the progression of the activity...”
“..I really felt I was in control, alerts were very helpful, I could relax and
read on student’s submissions, discussions etc...”
“...Very difficult to obtain the whole picture..I was stressed regarding the planned
time as some students were taking more time and frustrated for not having means
to control the script progressions...”

14. 14
● Teacher B - Visual inspection of signal change trend in two conditions
Estimating Orchestration Load using physiological
and subjective measures Contd.

15. 15
Estimating Orchestration Load using physiological and
subjective measures
● Teacher B - No Dashboard Condition
“...When the activity is done without the use of the
dashboard, I felt that I did not have control over the
activity.But the activity was not demanding...”
“...I am more relaxed when I use the dashboard and I can monitor the
progression of the activity....I really like to receive alerts..they are helpful to
react to certain moments of the activity...”

16. 16
● Compared EDA signal for two teachers
Estimating Orchestration Load using physiological
and subjective measures Contd.
Teacher A Teacher B

17. 17
Conclusions & Future Work
● According to the results we have so far we can conclude that:
○ Differences between conditions are obvious and this research presents promising first
findings
○ According to the qualitative comments and EDA measures it can be interpreted that:
■ teachers were less comfortable in no warnings dashboard condition
■ teachers were much comfortable in dashboard with warnings condition
● In future, we are planning to enrich our analysis further with other data sources and more
teachers
● To propose orchestration load aware design guidelines for teacher-facing dashboards

18. References
● Dillenbourg, P., Nussbaum, M., Dimitriadis, Y., & Roschelle, J. (2013). Design for classroom orchestration. Computers
& Education, 69(0), 485-492.
● Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology
Enhanced Learning, 4(5-6), 304-317.
● Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and
theoretical research. Advances in psychology, 52, (pp. 139-183).
● Manathunga, K., & Hernández-Leo, D. (2018). Authoring and enactment of mobile pyramid-based collaborative
learning activities. British Journal of Educational Technology, 49(2), 262-275.
● van Leeuwen, A. (2015). Learning analytics to support teachers during synchronous CSCL: Balancing between
overview and overload. Journal of learning Analytics, 2(2), 138-162.
● Prieto, L. P., Sharma, K., Kidzinski, Ł., & Dillenbourg, P. (2018). Orchestration Load Indicators and Patterns: In-the-Wild
Studies Using Mobile Eye-Tracking, IEEE Transactions on Learning Technologies,11(2), 216-229.