Keynote talk at CollabTech2022 (November 9, 2022): Design and orchestration of technology-enhanced collaborative learning can be very challenging for teachers or even instructional designers. This keynote presentation deals with design for effective and efficient collaborative learning, and how teachers as designers and orchestrators may be supported in complex ecosystems. We present the main challenges and solutions regarding conceptual and technological tools which may be developed, building on, and adapting to existing design knowledge. The talk will provide an overview of patterns, approaches, tools, and systems that should respect teachers’ agency while taking advantage of complex computational approaches, typically based on artificial intelligence. We pay special attention to recent research on how learning analytics solutions may be designed and implemented using human-centered approaches, and how socially shared regulated learning may be better supported. Several illustrating examples will be shown drawing on the literature and the research work of the presented during the last 25 years. Some prominent pending issues will be posed that may guide future research in supporting teachers as designers and orchestrators.

1. Teachers as designers and
orchestrators of technology-
enhanced collaborative learning:
Challenges and solutions
Prof. Yannis Dimitriadis
GSIC/EMIC group
University of Valladolid, Spain
CollabTech2022
10 November 2022

2. Why collaboration matters (I)
n Collaboration as a philosophical and political value
– Value to consider in human-centered value-sensitive design
n Collaboration as a learning/education mechanism
– Mechanism to be understood, activated and optimized
n Collaboration as a design target
– Product and process to be designed for learning by teachers
n Collaboration as a construct/concept
– Construct to be estimated in Socially Shared Regulated
Learning (SSRL) and co-RL
n Collaboration as a social mechanism
– Mechanism to be orchestrated in real-world classrooms
2

3. Why collaboration matters (II)
n Collaboration as a philosophical and political value
– Value to consider in human-centered value-sensitive design
– How can we design considering ethical/political values
regarding the involved stakeholders?
n Collaboration as a learning/education mechanism
– Mechanism to be understood, activated and optimized
– Is collaborative learning effective or how can it be more
effective?
n Collaboration as a design target
– Product and process to be designed for learning by teachers
– How can teachers (and other instructional designers)
design for learning?
3

4. Why collaboration matters (III)
n Collaboration as a construct/concept
– Construct to be estimated in Socially Shared Regulated
Learning (SSRL) and co-RL
– How can we measure theoretical concepts so that
pedagogical interventions can be made?
n Collaboration as a social mechanism
– Mechanism to be orchestrated in real-world classrooms
– What are the ways that teachers can put collaboration in
practice and manage it in real-world contexts?
4

5. A view of the TEL/CSCL ecosystem
How to design for learning in it?
5

6. Support teachers for collaboration
A contemporary technology-enhanced
learning (TEL) ecosystem is complex, and
collaboration is complex to achieve
What are the conceptual and technological
tools to support teachers as designers and
orchestrators for Computer Supported
Collaborative Learning (CSCL)?
6

7. What is this talk about (I)
n How to design for effective and efficient
(computer supported) collaborative learning?
n How teachers as designers and
orchestrators may be supported?
– What are the main challenges and solutions regarding
conceptual and technological support tools?
n building on and adapting to existing design knowledge ….
n Overview of patterns, approaches, tools, and
systems that should respect teachers’ agency
7

8. What is this talk about (II)
n Take advantage of complex computational
approaches, typically based on artificial
intelligence
n Special attention to recent research on
– How learning analytics solutions may be designed and
implemented using human-centered approaches?
– How socially shared regulated learning (SSRL) and co-
regulation may be better supported?
n Illustrating examples from the last 25 years
n Prominent issues to guide future research
8

9. Contents (I)
n Some fundamental questions
– Is (CS)CL effective in terms of learning outcomes?
– Are there any (successful) patterns/scripts for (CS)CL?
– What and how to design for (collaborative) learning?
– How can teachers design and orchestrate for (CS)CL?
n How to manage technology in CSCL contexts?
– Collaboration management cycle
– Technology for Collaboration
– Distributed Scaffolding in complex CSCL ecosystems
– Learning Analytics and AI for CSCL
– From data to collaboration analytics
– Human-Centered and Value-Sensitive Design 9

10. Contents (II)
n Examples of models, tools, architectures, applications
– Learning Design authoring and deployment tools
– Dialogic Feedback: Framework and system
– Detection and support of SSRL
– Group Formation in MOOCs
– Prediction and interventions in MOOCs
– Detection of participatory roles
n Challenges and future directions
n Take home messages
10

11. Is (CS)CL effective?
n Meta-analytical evidence (Kollar et al., in prep) shows
that CSCL is an effective way to foster learning, with
moderate effect sizes on learning outcomes
– Part of these effects can be attributed to the provision of
computer support
– Scaffold collaboration increases the effects of CSCL even
more, for example through the provision of CSCL scripts
– All these effects, however, seem to be moderated by distal
variables such as the participants’ educational level, the
domain of study, or the intervention duration
n More meta-studies necessary for effects on learning
processes, qualitative studies, technical papers
11
Kollar, I., Greisel, M., Özbek, T., Spang, L. & Vogel, F. (in prep.). Computer-supported collaborative learning.
To be included in A. Gegenfurtner & I. Kollar (Eds.), Designing effective digital learning environments. Routledge.

12. Patterns/scripts for (CS)CL?
n Effective and efficient (CS)CL unlikely to occur
– External obstacles (curriculum, competencies, cost,
practical issues)
– Psychological, social and technological issues
(group formation and roles, need for collaboration,
technology complexity, mindsets)
n Pedagogical patterns/scripts to increase the
chances for success
n Support to design for successful (CS)CL
12

13. (CS)CL (design) patterns
Think-Pair-Share
Think-Pair-Share (TPS) pattern
– It structures collaboration and promotes
participation in large classes
They comment or take a
classroom “vote”
They pair and discuss
their ideas about the
question
Each participant has
time to think about the
question
13

14. (CS)CL (design) patterns
Jigsaw/Puzzle
Individual or initial group
Teacher
Introductory
individual (or initial
group) activity
Collaborative
activity around the
sub-problem
Collaborative
activity around the
problem and
solution proposal
Jigsaw pattern
– It allows for individual accountability and
positive interdependence

15. (CS)CL Flow/assessment patterns
15
n Learning Flow
n Assessment

16. Aligning both sets of patterns
16

17. Usefulness of patterns
17
n Are these patterns useful for practitioners?
n Evidence for learning flow patterns for OER
repurposing to collaborative activities
n Appropriation in Higher Education teachers
(Professional Development workshops)
n But not always
n If not included in appropriate technological tools
n Too abstract for Primary Education practitioners
n Do not consider enactment issues

18. The need for atomic patterns
18
n Two independent studies by GSIC and SRI using Group
Scribbles in schools
n Studying enactment and disciplined improvisation
n Showed that other small-scale, informal, contextual,
actionable patterns emerged

19. A learning design in action
Orchestration graphs
19

20. Use of pedagogical/design pattterns
20
What for?
1. Understand, compile and represent design
knowledge (principles, patterns, procedures)
2. Empower and support teachers as designers
(professional development)
3. Inform and embed design in tools and
architectures (systems perspective)

21. An orchestration framework
21
L.P. Prieto, Y. Dimitriadis, J.I. Asensio, C-K. Looi, “Orchestration in Learning Technology Research: Evaluation of a Conceptual
Framework”, Research in Learning Technology (2015), 23: 28019 - http://dx.doi.org/10.3402/rlt.v23.28019 .

22. Learning Design (LD)
or Design for Learning (D4L) - I
n Learning is the objective of education
– Can we (pedagogically) inform and (technologically)
support teachers (and other stakeholders) in creating
effective (and efficient?) learning situations
n Main metaphor:
– “Teachers as designers”
n Main issue:
– Is it possible that teachers (and other stakeholders)
work as other traditional designers (e.g., architects,
engineers)?
– Can design form part of the normal flow of educational
activities? 22

23. Design for Learning – II
“In media res framework”
n What can be designed for learning?
– The learning (performed by students) and support
(made by teachers) tasks
n The “physical” environment
n Spaces, tools, infrastructures, artifacts-resources (to be
consumed and/or produced)
n The social architecture
n Groupings, interactions with external agents
n Design is indirect (tasks vs. activities)
n Learners may change-interpret tasks in learntime
23
P. Goodyear, Y. Dimitriadis, “In medias res: reframing design for learning”, Research in Learning Technology, Research in Learning
Technology Supplement (2013); 21: 19909 - http://dx.doi.org/10.3402/rlt.v21i0.19909 .

24. Design for Learning – III
“Activity-Centered Analysis and Design” (ACAD)
24
Goodyear, P., Carvalho, L. & Yeoman, P. Activity-Centred Analysis and Design (ACAD): Core purposes, distinctive qualities and
current developments. Education Tech Research Dev 69, 445–464 (2021). https://doi.org/10.1007/s11423-020-09926-7

25. Design for Learning
Some notes on the framework
25
n In the new real contexts
n Teachers cannot resolve all emerging problems
n There are many internal and external conditions
n An equilibrium should be found between
n Learners’ autonomy (“agency”)
n Structuring and guiding (“structure, scripting”)
n The process of D4L is nontrivial
n Meet several objectives that are eventually
contradictory

26. Design for Learning (D4L)
Some notes on the framework
26
n Who is going to control the learning process?
n The teacher / designer
n The student / learner
n The devices / resources / virtual agents of the
learning environment
n D4L should look forward (forward-oriented design) in
order to support the different phases
n Configuration, orchestration, reflection, redesign ...

27. Forward oriented design - I
27
n Design for Configuration:
n Prepare, particularize, or modify what had been
designed beforehand
n Adapt to specific requirements of the context
n Design for Orchestration
n Support teachers (and students) at learn time
Y. Dimitriadis, P. Goodyear, “Forward-oriented design for learning: Illustrating the approach”, Research in Learning Technology
Supplement (2013), 21: 20290 - http://dx.doi.org/10.3402/rlt.v21i0.20290

28. Forward oriented design - II
n Design for Reflection
n Be sure that monitoring can take place
n Offer “awareness” at learn time
n Allow for regulation and scaffolding
n Inform evaluation of the intervention and
assessment of learning
n Design for Redesign
n Take design decisions so that posterior
modifications of the design can be made easily

29. TinkerLamp
Design for orchestration
29
Do-Lenh, S., Jermann, P., Legge, A., Zufferey, G., Dillenbourg, P. (2012). TinkerLamp 2.0: Designing and Evaluating Orchestration Technologies for the
Classroom. In: Ravenscroft, A., Lindstaedt, S., Kloos, C.D., Hernández-Leo, D. (eds) 21st Century Learning for 21st Century Skills. EC-TEL 2012. Lecture Notes in
Computer Science, vol 7563. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33263-0_6

30. GroupScribbles
Design for social coordination
30

31. Spaces for collaboration
Design physical (and digital) environment
31

32. Group concept maps
Full set of forward-oriented design
32
R. Martínez, Y. Dimitriadis, A. Martínez, J. Kay, K. Yacef “Capturing and analysing verbal and physical collaborative learning interactions at
an enriched interactive tabletop”, International Journal of Computer Supported Collaborative Learning, 8(4), 455-485 (2013)

33. Teaching as a design science
n “In media res”: Design forms part of the “normal
flow” of educational activities
n Teaching as a design science
n Teaching is not only an art. It has a formally defined
goal
n It builds on design principles, rather than theories,
and heuristics of practice than explanations
n See also Design Based Research, co-design, …
n Design patterns externalize knowledge and allow for
discussion and sharing …
Laurillard, D. (2018). Teaching as a Design Science: Teachers Building, Testing, and Sharing Pedagogic Ideas. In: Voogt, J., Knezek, G., Christensen, R., Lai, KW. (eds)
Second Handbook of Information Technology in Primary and Secondary Education . Springer International Handbooks of Education. Springer, Cham.
https://doi.org/10.1007/978-3-319-71054-9_108

34. Supporting the D4L lifecycle
Objectives,
Activities …
Students,
resources,
ICT tools…
How to
Enhance?
Design for Learning lifecycle

35. • Conceptualize
• Author
• Implement
• Share
• Evaluate
• Explore
Integrated Learning Design Environment http://ilde2.upf.edu
The METIS ILDE
The ILDE environment

36. Example at UVA:
“Landscape organization”
(Geography)

37. And the course in Moodle!

38. n ILDE
– It fits the needs of several educational contexts
– It is independent from the pedagogical approach
adopted
– It covers all the phases of the LD lifecycle
– It is usable by the ‘average’ teacher
n Pedagogy is the “king”
n Support is needed for enactment and communities
beyond workshops
Some METIS findings
Some findings on ILDE
J. I. Asensio, Y. Dimitriadis, F. Pozzi, D. Hernández, L.P. Prieto, D. Persico, S. Villagrá, “Towards Teaching as Design: exploring the interplay
between full lifecycle Learning Design tooling and Teacher Professional Development”. Computers & Education, 114, 92-116 (2017).

39. n First Order Barriers
– Lack of institutional support
– Lack of adequate teacher training
– Time/workload factors
– Conceptual complexity of method and tools
– Adoption by peers
n Second order barriers
– Use of ICTs in teaching practice
– Teachers’ motivation
Some METIS findings
Barriers for adoption of LD
F. M. Dagnino, Y. Dimitriadis, F. Pozzi, J.I. Asensio-Pérez, B. Rubia-Avi, “Exploring teachers’ needs and the existing barriers to the adoption of
Learning Design methods and tools: a literature survey”, British Journal of Educational Technology, 49(6) 998-1013 (2018)

40. Some METIS findings
Lightweight tools for CSCL patterns
The PyramidApp
D. Manathunga, K., Hernández-Leo (2018), Authoring and enactment of mobile pyramid
based collaborative learning activities, British Journal of Educational Technology, 49(2),262–275, doi:10.1111/bjet.12588

41. Collaboration tools/resources
n Building the foundation or context for collaboration
n Building interventions within contexts
n Orchestrating student and teacher activities
n Analyzing data
41
Rosé, C., Dimitriadis, Y. (2021). Tools and Resources for Setting Up Collaborative Spaces. In: Cress, U., Rosé, C., Wise, A.F.,
Oshima, J. (eds) International Handbook of Computer-Supported Collaborative Learning. Computer-Supported Collaborative
Learning Series, vol 19. Springer, Cham. https://doi.org/10.1007/978-3-030-65291-3_24.

42. The collaboration management cycle
42
Soller, A., Martínez-Monés, A., Jermann, P., Muehlenbrock, M. (2005) From Mirroring to Guiding: A Review of the State of the Art
Technology for Supporting Collaborative Learning International Journal of Artificial Intelligence in Education (ijAIED). 15:261-290

43. Distributed scaffolding
43
• Puntambekar, S. Distributed Scaffolding: Scaffolding Students in Classroom Environments. Educ Psychol Rev (2021).
https://doi.org/10.1007/s10648-021-09636-3
• https://www.imec-int.com/en/research-portfolio/steams: Supporting TEAMS in ambient learning spaces
Across
1. Tools and social scaffolds
2. Levels (individual, group, and whole class)
3. Time and Contexts

44. Collaboration analytics (I)
n Collaboration Analytics
– Collocated, face-to-face groups, and platform-mediated
groups (synchronous and asynchronous)
– Help the team, educators, researchers or other stakeholders
gain insights into the activity, and inform action that
improves collaboration processes and products
n Collaboration analytics require the integration of
– (i) theoretically sound concepts
– (ii) well designed tasks that generate useful data traces to
inform analytics
– (iii) human-centered design that empowers stakeholders to
shape the analytics and produces effective user interfaces
44
Martinez-Maldonado, R., Gašević, D., Echeverria, V., Fernandez Nieto, G., Swiecki, Z., & Buckingham Shum, S. (2021). What Do You Mean
by Collaboration Analytics? A Conceptual Model. Journal of Learning Analytics, 8(1), 126-153. https://doi.org/10.18608/jla.2021.7227.

45. Collaboration analytics (II)
45
Martinez-Maldonado, R., Gašević, D., Echeverria, V., Fernandez Nieto, G., Swiecki, Z., & Buckingham Shum, S. (2021). What Do You Mean by
Collaboration Analytics? A Conceptual Model. Journal of Learning Analytics, 8(1), 126-153. https://doi.org/10.18608/jla.2021.7227.

46. Human-Centered Learning Analytics
46
Human-Centered Design should involve:
Inclusion via stakeholder participation in the design process
+
Empathic experiences (particularly when making design
decisions)
Giacomin, J. (2014). What is human centred design? The Design Journal,
17(4), 606–623. https://doi.org/10.2752/175630614X140561854801.

47. Levels of human-centeredness
47
Smuha N.A. (2023). “Pitfalls and pathways for trustworthy Artificial Intelligence in education” in The Ethics of Artificial
Intelligence in Education Practices, Challenges, and Debates, W. Holmes, K. Porayska-Pomsta (Eds). Taylor and
Francis.
n Human in command
– Oversee when and how to use AI/ITS
n Human on the loop
– Participate in design and operation
n Human in the loop
– Get involved in every lifecycle phase

48. Human-Centeredness in MMLA-AIED
48
Kukurova, M. (2022). “Multimodal Learning Analytics in Real-world Practice: A Bridge Too Far?”, Webinar at Spanish
Network of Learning Analytics (SNOLA), May 2022. https://snola.es/2022/05/03/webinar-multimodal-learning-
analytics-in-real-world-practice-a-bridge-too-far-mutlu-cukurova/

49. Human-centered and trustworthy AI
49
• Delgado Kloos, C., et al. (2022), H2O Learn - Hybrid and Human-Oriented Learning: Trustworthy and Human-
Centered Learning Analytics (TaHCLA) for Hybrid Education. IEEE Global Engineering Education
Conference, EDUCON 2022,
• HLEG-AI (High-Level Expert Group on Artificial Intelligence) (2019), “Ethics Guidelines for Trustworthy AI:
Requirements of Trustworthy AI,” Available: https://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines/1

50. Orchestration and LA (OrLA)
50

51. LA-based orchestration tools for CSCL
Cristina Villa-Torrano
Examining the relationship between groups’ engagement within
SSRL Docs and performance
51
n Orchestration tools build upon learning analytics
n Information that could be adapted to the specific situation
n Detrimental effects if not tailored to a teacher’s needs
n Teachers in the advising condition more often detected the
problematic group, needed less effort to do so, and were
more confident of their decisions
Anouschka van Leeuwen, Nikol Rummel. 2020. Comparing Teachers’ Us of Mirroring and Advising Dashboards. In Proceedings of the 10th
International Conference on Learning Analytics & Knowledge (LAK’20). ACM, New York, NY, USA, X pages.

52. Collaborative Dialogic Feedback (I)
Framework
52
Erkan Er, Yannis Dimitriadis & Dragan Gašević (2020): A collaborative learning approach to dialogic peer feedback: a theoretical
framework, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2020.1786497

53. Dialogic Feedback (II)
Design principles
n Connect self-evaluation with peer evaluations
n Provide opportunities to resolve the discrepancies in
students’ perspectives about the quality of the work
n Provide mechanisms to (collectively) plan the feedback
before its provision
n Enable dialogue around the feedback to support its uptake
n Enable students to set goals and create an action plan with
peers based on the feedback
n Enable dialogue with peers while students are revising
their work
53
Erkan Er, Yannis Dimitriadis & Dragan Gašević (2020): A collaborative learning approach to dialogic peer feedback: a theoretical
framework, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2020.1786497

54. Dialogic feedback (III)
The Synergy system
54
Erkan Er, Yannis Dimitriadis & Dragan Gašević (2020): Collaborative peer feedback and learning analytics: theory-oriented design for
supporting class-wide interventions, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2020.1764490

55. Socially Shared Regulated Learning (I)
55
Cristina Villa-Torrano
Examining the relationship between groups’ engagement within
SSRL Docs and performance
Socially-Shared Regulation of Learning
Cognitive Metacognitive
Motivation Emotions

56. Socially Shared Regulated Learning (II)
To extract meaningful SSRL indicators
from trace data to support groups during
collaboration
Cristina Villa-Torrano
How have we incorporated SSRL phases in the
design?
Task
Understanding
Setting Goals &
Planning,
Adaptation
Task Enactment
56

57. Socially Shared Regulated Learning (III)
Cristina Villa-Torrano
Examining the relationship between groups’ engagement within
SSRL Docs and performance
Which data sources do we have to extract meaningful
information?
57

58. Detection/adaptation of roles
58
J.A. Marcos, A. Martínez, Y. Dimitriadis, “DESPRO: A method based on roles to provide collaboration analysis support adapted to the participants
in CSCL situations”, Computers & Education, 82 235-253 (2015)

59. Group Formation in MOOCs
n Support for CSCL in MOOCs (guide and tool)
n Design guide allowed teachers
– Be aware of the contextual factors to consider when forming
the collaborative groups
– Be informed about some configuration parameters of the
activity and the group formation
n Strategy with higher degree of homogeneity
– Grouped students with similar levels of engagement
– Achieved best results in terms of group performance, group
interactions and student satisfaction
59
Luisa Sanz-Martínez, Erkan Er, Alejandra Martínez-Monés, Yannis Dimitriadis & Miguel L. Bote-Lorenzo (2019) Creating
collaborative groups in a MOOC: a homogeneous engagement grouping approach, Behaviour & Information Technology, 38:11,
1107-1121, DOI: 10.1080/0144929X.2019.1571109

60. Prediction/interventions in MOOCs
n Predictive models
– Accurately classify learners according to their expected
engagement levels in an upcoming peer-review activity
– Pedagogical utilities (e.g., improving peer reviews and
collaborative learning activities)
n In situ learning and transfer across courses
– Proxy label to train a model within the same course
– Using labels obtained from past course data
n Actionable predictions
– While the course continues, vs. post-hoc approaches
60
Erkan Er, Eduardo Gómez-Sánchez, Miguel L. Bote-Lorenzo, Yannis Dimitriadis & Juan I. Asensio-Pérez (2020) Generating actionable predictions
regarding MOOC learners’ engagement in peer reviews, Behaviour & Information Technology, 39:12, 1356-1373, DOI: 10.1080/0144929X.2019.1669222

61. Challenges and future directions
n Human-centered design approaches are necessary to
give voice to stakeholders and promote agency
n Support to AI/LA-supported CSCL requires balanced
and joint use of learning theories, data science, and
design approaches
n Mindsets of learners and teachers serve as promoters
and obstacles of successful CSCL
n Distributed scaffolding across tools, persons, and
contexts is actual, promising and challenging
61

62. Take home messages
n Collaboration has multiple relevant facets that require
careful design to be successful
n Teachers as designers and orchestrators of CSCL can
and should be supported by tools
n TEL-CSCL ecosystems are increasingly complex and
AI/LA companions pose severe threats to agency
62