Presentation at the Learning Analytics Summer Institute. Bilbao (Spain). June 27th, 2016 From the conceptualization to the evaluation of blended learning scenarios, teachers address multiple tasks, sometimes being overwhelmed on account of the required time and associated burden. To support teachers in this endeavor, we propose to connect the pedagogical decisions made at design time with the analysis of the participants’ interactions. In this paper, we evaluate the proposal in two authentic scenarios where we analysed whether the script-aware monitoring process provided the participant teachers with relevant information for the orchestration of blended learning scenarios. The participant teachers valued the proposal positively in terms of representativeness, novelty, relevance, required effort, and perceived usefulness. Additionally, they stated that it was helpful for the orchestration of the learning scenarios.

1. Design-aware analytics
supporting teachers’
monitoring of blended
learning scenarios: Two
experiences in higher
education
María Jesús Rodríguez Triana, Alejandra Martínez-Monés, Juan Ignacio
Asensio-Pérez
GSIC – EMIC, Universidad de Valladolid
amartine@infor.uva.es

2. DLEs
[MacNeil & Kraan,
2010]
Context
Blended Learning
[Graham, 2005]
CSCL
[Stahl et al., 2006]
+
Multiple locations
Multiple interactivity types
Multiple tools
Multiple social levels
Multiple interactivity types
June 27, 2016LASI Spain 2016
2

3. Challenges to LA
— The problem of “dirty & incomplete” data
— Analysis should be aware (and make the user aware)
of the weaknesses of data
— The need for contextualization
— Data should be analyzed in their context
— Teachers’ workload and interpretation of the results
— Results must be manageable and easy to understand
— Ethical issues regarding e.g., data validity and
adverse impact
June 27, 2016LASI Spain 2016
3

4. Overall approach
— Analyze whether the current state of interaction
(monitored at enactment time) matches the desired
state of interaction (defined in the script at design
time).
June 27, 2016LASI Spain 2016
4
Scripting
Monitoring

5. Proposals
June 27, 2016LASI Spain 2016
5

6. Methodological approach
Design-Based Research
— Design-Based Research
— Iterative research
— Situated in educational contexts
— Paternship between researcher and practitioners
— Focus on design and testing a significant intervention
— Use of mixed-methods
— Strong emphasis on applying research to a real
problem à Close connection to pedagogical practice
June 27, 2016LASI Spain 2016
6

7. DBR – Research process
June 27, 2016LASI Spain 2016
7

8. What we did in practice (1/3)
June 27, 2016LASI Spain 2016
8

9. What we did in practice (1/3)
June 27, 2016LASI Spain 2016
9

10. What we did in practice (2/3)
June 27, 2016LASI Spain 2016
10

11. GLIMPSE - Heuristics
— Select an action to be included in the analysis
— Define and compare the current and desired state of
the interaction regarding:
— participation,
— collaboration,
— group formation and
— expected use of resources.
June 27, 2016LASI Spain 2016
11

12. What we did in practice (3/3)
June 27, 2016LASI Spain 2016
12

13. Monitoring report
September 5, 2014
13
Warnings
Integrated data sources
Students’
self-­‐reports
GLUE!-­‐CAS
blogs
Teacher’s
observa>ons
Groups and
participants
The system has not
registered any access to
the Final research
proposal by
StudentName6
A warning is raised to
advise the teacher

14. Evaluation
Focusing on the teacher’s perspective
June 27, 2016LASI Spain 2016
14

15. Anticipated data reduction
[Miles & Huberman, 1994]
Evaluation
June 27, 2016LASI Spain 2016
15

16. Anticipated data reduction
[Miles & Huberman, 1994]
Evaluation
June 27, 2016LASI Spain 2016
16

17. Informants & data gathering techniques
Data sources
June 27, 2016LASI Spain 2016
17

18. Anticipated data reduction
[Miles & Huberman, 1994]
Evaluation
June 27, 2016LASI Spain 2016
18

19. — The interpretation of the reports was
easy and efficient.
— The information was generally
accurate, reflecting a realistic view of
the process. A small percentage of
false positives and of problems that
passed unnoticed, but the teachers
did not consider them to be critical.
— The quality of data is very limited
(i.e., accesses to docs, not actual
reading…). The approach promotes
to interpret the data using extra
information teachers might have
available. This points to lines of
potential improvement.
June 27, 2016LASI Spain 2016
19
Script-aware monitoring
process

20. • The process provided teachers with relevant feedback to
improve the awareness on the learning situation and to
support the regulation tasks.
• More advanced solutions should be found to support the
gathering of data directly provided by teachers and
students (to enrich computer-mediated evidence)
• New data sources and indicators are required to minimize
the deviations and to take into account the quality of the
participation
Script-aware monitoring
process
— Achievements and limitations
June 27, 2016LASI Spain 2016
20

21. Conclusions / Remarks
— The alignment between scripting and monitoring
helped to improve both processes
— Teachers intervene in the definition of the analysis
with an affordable effort, and were able to interpret
the results
— The approach is minimalistic: shows initial
evidences that teachers check with their available
information
— Positive impact on data validity, responsibility, and
diminution of potential adverse impact
June 27, 2016LASI Spain 2016
21

22. Thank you!
More info:
https://www.gsic.uva.es/personal/chus
E-mail:
amartine@infor.uva.es
maria.rodrigueztriana@epfl.ch
June 27, 2016LASI Spain 2016
22