This document discusses aligning design for learning (D4L) with learning analytics (LA) in technology-enhanced learning ecosystems. It proposes that D4L knowledge can be captured through conceptual tools like principles, patterns and processes. D4L forms part of teacher inquiry and should inform and be informed by LA through teaching analytics. Multiple frameworks could explain the relationships between D4L, LA, the learning environment and stakeholders like teachers. Bringing these perspectives together could help support teachers as designers and orchestrators through sustainable systems.

1. Design for effective orchestration and
pedagogical interventions aligned with
learning analytics in technology-
enhanced learning ecosystems
Prof. Yannis Dimitriadis
GSIC/EMIC group
University of Valladolid, Spain
University of Trondheim
8 Nov 2017

2. Deep Acknowledgements
to collaborators and GSIC/EMIC
2

3. Overview (I)
Theme and context
3
Understand and support
Design for Learning (D4L) aligned with
Learning Analytics (LA) in Technology
Enhanced Learning (TEL) ecosystems
Real world teachers design and orchestrate
innovative pedagogical scenarios in complex
TEL contexts

4. Overview (II)
Some questions to answer
4
n What can be designed and by whom?
n How D4L knowledge can be captured and
expressed?
n How D4L can be understood and supported
within the teacher inquiry process and
aligned with LA?
n What are the frameworks that can help us
understand better D4L and LA?
n Can we formulate design knowledge and
recommendations for D4L and LA alignment?

5. D4L in complex TEL ecosystems
(at a classroom level)
5

6. (Design and...) Orchestration
(Prieto et al., 2015)
6

7. Design for Learning (D4L)- I
or Learning Design (LD)
n Can we (pedagogically) inform and
(technologically) support teachers (and other
stakeholders) in designing and orchestrating
effective (and efficient?) learning situations
n Teachers as designers … generate products …
within processes …
7

8. Design for Learning – II
“In media res framework”
n What can be designed for learning? (Goodyear &
Dimitriadis, 2013)
n The learning (performed by students) and support
(made by teachers) tasks
n The “physical” environment
n Spaces, tools, infrastructures, artifacts-resources (to be
consumed and/or produced)
n The social architecture
n Groupings, interactions with external agents
n Design is indirect (tasks vs. activities)
n Learners may change-interpret tasks in learntime
8

9. Design for Learning – III
Focus on student activity (ACAD)
9
Activity-Centred Analysis and Design (ACAD) framework
(Carvalho & Goodyear, 2014)

10. Design for Learning - IV
In media res: Some notes
10
n In the new real contexts
n Teachers cannot resolve all emerging problems
n There are many internal and external conditions
n An equilibrium should be found between
n Learners’ autonomy (“agency”)
n Structuring and guiding (“structure, scripting”)
n The process of D4L is non trivial
n Meet several objectives that are eventually
contradictory

11. Design for Learning - V
In media res: Some notes
11
n Who is going to control the learning process?
n The teacher / designer
n The student / learner
n The devices / resources / virtual agents of the
learning environment
n D4L should look forward (forward-oriented design)
in order to support the different phases of the
design and orchestration lifecycle
n Configuration, orchestration, reflection, redesign
...

12. Design for Learning – VI
In media res: Some notes
12
n Design for Configuration:
n Prepare, particularize, or modify what had been
designed beforehand
n Adapt to specific requirements of the context
n Design for Orchestration
n Support teachers (and students) at learn time

13. Design for Learning - VII
In media res: Some notes
n Design for Reflection
n Be sure that monitoring can take place
n Offer “awareness” at learn time
n Allow for regulation and scaffolding
n Inform evaluation of the intervention and
assessment of learning
n Design for Redesign
n Take design decisions so that posterior
modifications of the design (pedagogical
interventions) can be made easily

14. D4L knowledge
Mediating conceptual tools
14
n Design for Learning (D4L) articulated
knowledge may be captured and expressed
n as conceptual mediating tool not only for
researchers but also for teachers
n in the form of D4L principles, patterns and
procedures at multiple levels
n and eventually operationalized in terms of ICT
tools and environments, professional
development actions

15. Design Principles (Kali, 2006)
http://www.edu-design-principles.org/dp/aboutDPD.php
15

16. D4L Patterns
Learning flow, assesment (artefact flow)
16

17. Recurrent routines
Atomic patterns, teacher moves
17

18. Levels of D4L knowledge
18
n From design principles to patterns and routines,
that are put in place through design processes

19. Teaching as a design science
(Laurillard, 2012)
n “In media res”: Design forms part of the
“normal flow” of educational activities
n Teaching as a design science
n Teaching is not only an art. It has a formally defined
goal
n It builds on design principles, rather than theories,
and heuristics of practice than explanations
n See also Design Based Research, co-design, …
n Design principles, patterns, routines externalize
knowledge and allow for discussion and sharing …
(and eventually operationalization)

20. But teaching design knowledge
(TPACK) is
………n Typically informed by
n Own and shared experience and intuition
n And partially by
n Learning scientists
n Teacher education curriculum
n Typically expressed in
n Lessons learnt
n Tacit knowledge
n Knowledge in pieces
n Activated by “internal schemas” by experts
n Characterized by cognitive (and orchestration) load

21. Supporting the D4L lifecycle
Objectives,
Activities …
Students,
resources,
ICT tools…
How to
Enhance?
D4L knowledge embedded
in a process / lifecycle

22. • Conceptualize
• Author
• Implement
• Share
• Evaluate
• Explore
Integrated Learning Design Environment http://ilde2.upf.edu
The METIS ILDE
The ILDE environment
Embeds D4L knowledge and tools

23. The ILDE environment
An architectural view
23
• Conceptualize
• Author
• Implement
• Share
• Evaluate
• Explore
Integrated Learning Design Environment http://ilde2.upf.edu/

24. Across spaces …
An architectural view
24

25. Teacher inquiry – TI
and D4L
n TI: “a set of research practices by which
teachers examine their practice and its effects
on students’ learning, in order to enhance
their professional knowledge and improve
practice” (Clarke & Erickson, 2003)
n TI and D4L process / lifecycle
– Inquiry on teaching (and learning) practices,
augmenting, activating, putting in practice,
assessing, sharing … the D4L knowledge
n Need for Teaching Analytics …
25

26. Teacher Inquiry into Student Learning
TISL
n TISL: “Systematic, intentional, design-
oriented approach to teachers’ technology-
supported inquiry into students’ learning…
n … digital tools … to capture, analyse,
interpret, share and evaluate student data…
n … focus on teacher-centred, practice-based,
evidence-oriented research activity (Clark et
al, 2011)
n Need for Learning Analytics…
26

27. LA meets TISL, TA and D4L
n Learning Analytics
– “the measurement, collection, analysis and reporting of
data about learners and their contexts, for purposes of
understanding and optimizing learning and the
environments in which it occurs” (SOLAR, 2011)
n Learning Analytics (LA) provide a data-informed
approach (process and products)
– to Teacher Inquiry into Student Learning (TISL)
– in conjunction with Teaching Analytics (TA), i.e. the
process and products of Design for Learning (D4L)
– using and augmenting D4L knowledge
27

28. Dependencies and interplay …
(LE, D4L, LA, TA, TISL, actors…)
n Evidence provided by LA (process and product)
informs existing D4L (process and products)
– Allows orchestration, reflection, redesign and
implementation of pedagogical interventions
n Evidence by LA - TA augments D4L knowledge
– Design principles, patterns and procedures
n (Process and product) of D4L informs and
modulates LA (process and product)
– Affects types of data to monitor and collect, indicators
to calculate and visualize, etc., more appropriate for
design and orchestration of the concrete D4L 28

29. Multiple interdependencies …
n LA can provide such a “structured evidence” for
“optimizing” the learning experiences and allow for
informed pedagogical decisions, interventions
– i.e. LA should provide “actionable information”
n Then, D4L (process and product) should be used in
conjunction / alignment (bi-direccional relation)
with LA
n What are the D4L decisions that may build the necessary
LA solutions (LA-aware design, or design for Analytics)?
– How LA may inform D4L (LA-aware design)?
29

30. However in most LA projects …
n Analytics are created and presented by other
stakeholders (researchers, systems designers,
institutions) to the teacher (or the student) in a
dashboard, without
– Making explicit connection to D4L process and product
– Involving “the teachers or the students in the loop”
– Considering teacher-student agency and trust (versus
external, institutional control through LA (Griffiths, 2017))
n Post-hoc analyses are performed to understand
processes (student retention …)
– Aiming to extract patterns, identify the most influencing
variables in predictive models, analyze strategies, etc. 30

31. Our research objectives …
n Employ existing (and create) new theoretical
frameworks that may explain the phenomena
n Study TISL processes and extract D4L knowledge
n Encapsulate the joint D4L+LA+TA decisions in new
D4L knowledge: principles, patterns, and processes
n Support the D4L and LA processes within TISL
n Embed the new D4L knowledge in conceptual and
digital tools, architectures and systems, professional
development actions
n Provide recommendations for more effective and
efficient design and orchestration 31

32. Theoretical frameworks
n Design for Learning (ACAD and In Media Res)
n Distributed cognition (DCon)
n (Participatory) Sense Making (from noun to verbs)
n Cognitive and Orchestration Load and internal
schemas
n Orchestration and Orchestrating Learning Analytics
(OrLA)
Several frameworks that may come together to
express the complex relations
32

33. Theoretical frameworks
(Alhamad & Thompson, 2017)
33

34. Orchestration + ACAD + DCON
(Martínez et al., 2017)
34

35. Orchestrating LA (OrLA)
(Prieto et al., 2017)
35

36. (Participatory) sense making
(Dervin, 1998)
n Knowledge is the sense made at a particular point in
time-space by someone
n Sense making is directly involved in decision (design
and orchestration) processes (D4L) around a
learning environment (LE) informed by learning
analytics (LA)
n Sense making as “verb” vs. “noun” in research
practices in order to elicit these LD-LA-LE sense
making
– What happened that brought you here today?
– If you could wave a magic wand, how would we help you?
– … 36

37. D4L+LA design principles
(Wise, 2014)
n Learning Analytics Implementation Design
approach (LAID): “mediating artifact between the
analytics presented and the localized course context”
– DP1 Coordination: which analytics, what productive patterns
and what measures, or “logistics”, i.e. when and how, and
whether it should be free or guided
– DP2 Comparison: implemented as an “absolute” fixed
standard or exact target, or relative with respect to peers, other
events in the same course or in different/similar courses, or with
respect to prior activity of the same students
– DP3 Customization: multiple needs and paths to use LA,
implemented as adaptive (by system/agent) or adaptable (by
users) 37

38. D4L + LA design principles
(Harrer, 2015)
n Complementary principles:
– DP4 Scope: which information of the Learning Environment
(LE) is relevant and useful for analysis and interpretation
– DP5 Representation Consistency: between the information
employed in the Learning Environment (LE) and the one used
for visualization of the LA by students (and teachers).
38

39. LA and D4L as products
(Nguyen et al, 2016) (Bakharia et al., 2016)
39

40. Interplay between LA and LE
(Leeuwen, 2015,2016,2017)
40

41. The LA+LE (and D4L) interplay
(Leeuwen, 2015) (Wise, 2015) (Chen, 2015)
n Important design decisions
– Embedded vs. extracted analytics
– Continuous vs. cumulative data
n Considering interplay between LA and LE (and D4L)
during design and orchestration
n Measuring and keeping low cognitive / orchestration
load and augmenting internal schemas
n Conceiving the D4L-LA-LE system and process in
terms of
– architecture (components, degree of coupling and reuse)
– user interaction patterns
41

42. D4L allows better predictive models
(Gasevic, 2016) (Er, 2017) (Sanz, 2017)
n “There is a need to create models for academic
success prediction for individual courses,
incorporating instructional conditions into the
analysis model”
n Features drawn from the D4L product can be
derived that create better prediction models
n Better in-situ prediction models allow better
pedagogical interventions and orchestration
42
BLOCK 6
Introduction to
module
Discussion
Forum
Peer Review for
IA3 in Week 5
Quiz 7 Peer Review for
GA2 in Week 5
Peer Review for
OA3 in Week 5
Optional
Activity
BLOCK 5
Introduction
[BLK5_0]
Lecture Content
[BLK5_1]
Discussion Forum
[BLK5_2]
Detailed
Description
[BLK5_31]
Introduction &
Submission
[BLK5_30]
Peer Review for
GA1 in Week 4
Peer Review for
OA2 in Week 4
Optional
Activity
Discussion
Forum
[BLK5_32]
Group Assignment 2 [GA2]
BLOCK 4
Introduction
[BLK4_0]
Lecture Content
[BLK4_1]
Discussion Forum
[BLK4_2]
Peer Review for
IA2 in Week 3
[BLK4_4]
Review Video
[BLK4_5]
BLOCK 3
Introduction
[BLK3_0]
Lecture Content
[BLK3_1]
Discussion Forum
[BLK3_2]
Detailed
Description
[BLK3_31]
Optional Activity
[BLK3_4]
Discussion
Forum
[BLK3_32]
Group Assignment 1 [GA1]
Review Video
[BLK3_5]
Review Video
[OA2]
[OA3]
[OA4]
[IA3]
Introduction &
Submission
[BLK4_3]
Introduction &
Submission
[BLK3_30]

43. Bringing the teacher in the loop
(Rodríguez et al., 2017)
n Customization of the D4L and orchestration
increases efficacy and teacher agency and trust
n LA solutions based on concrete D4L information
provided by the teacher: course checkpoints, script
and activity constraints, orchestration problems
43

44. Current personal work
Sabbatical year (2017-2018)
n Data analysis
– Lessons learnt and student networks in MOOCs
– Personalized feedback in 1st year HE flipped classroom
– Needs and case studies in the LOOP project
– Teacher views and feedback in the LocoAnalyst project
n Connections to
– Self, co-, socially shared regulation
– Orchestration Graphs (Lausanne)
– WISE teacher community (Berkeley)
44

45. Some conclusions
n Design for Learning (D4L) knowledge can be
– Encapsulated in principles, patterns, processes
– Embedded in practices, tools and PD actions
n D4L (and orchestration)
– Forms part of Teacher Inquiry into Student Learning
(TISL)
– Informs (and gets informed by) the Teacher (TA) -
Learning Analytics (LA) products and processes
– May be explained by multiple theoretical frameworks
– Affects system design and research practices
45

46. And future directions
n Bring together multiple stakeholders
n Use trans-disciplinary approaches
n Put more emphasis on ethnographic studies
n Analyze better the multiple dependencies
between D4L, LA and TA, LE within TISL
n Support teachers as designers and orchestrators
n Create sustainable technological architectures
46

47. OrLA workshop at LAK18
n LAK18, Sydney, Monday 5 March 2018
n Workshop: Orchestrating Learning Analytics:
Learning Analytics Adoption at the Classroom Level
n Organisers: Luis P. Prieto, María Jesús Rodríguez-
Triana, Roberto Martinez-Maldonado, Dragan
Gasevic and Yannis Dimitriadis
n Call for papers (deadline 18 December 2018):
https://sites.google.com/view/orla-ws-2018/call-for-submissions
47