Involving stakeholders in Learning analytics design is a hard task that requires a clear strategy that otherwise creates a problem with low adoption, disengagement with the tools and unclear expectations. Including teachers, learners, developers and other stakeholders as collaborators in design (Co-design) bring promising benefits in democratizing, aligning and acknowledging stakeholders’ expectations.

1. Engaging stakeholders in
the learning analytics
design process
Carlos Gerardo Prieto Alvarez
PhD candidate in Learning Analytics
1
Supervisors:
Simon Buckingham Shum
Roberto Martinez-Maldonado

2. 2

3. Design process
Learning Analytics (LA) concept and
design
3
Common LA interfaces
Researcher

4. Stakeholders are excluded in the LA
design process
4
Assumptions
(Drachsler and Greller
2012 ; McGregor
2016)
Misrepresentation of
learners interest
(Beattie 2014, Sharon
Slade and Prinsloo
2013)
Powerlessness
(Randy Bass, Mindy
McWilliams et al. 2015)
Disengagement
(Mitra 2004)
Concerned
(Sharon Slade and Prinsloo
2015)
(Bart Rienties 2014)
Alienation
Impact of adoption
(Dyckhoff et al. 2012)
Teachers
(Mitra 2004 ; Beattie
am Woolner 2007;
Scott 2014)
Neglecting student
involvement
Students

5. c
Stakeholders collaboration in design
5Based on Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. CoDesign, 25(2).
Expert Mindset
“users” seen as
subjects (reactive
informers)
Participatory Mindset
“users” seen as
partners (active
co-creators)
Design-Led
Research-Led
c
Co-Design
Co-design research field
The field sits in a place where
contemporary PD invites
stakeholders as active
participants while
researchers/practitioners
manage the design process.

6. Gap in learning analytics design
6
DATA
Mining techniques
Analysis methods
Analytics design
EDUCATION
Learning design
Learner experience
Epistemic theory

7. Gap in learning analytics design
7
DATA
Mining techniques
Analysis methods
Analytics design
EDUCATION
Learning design
Learner experience
Epistemic theory
STAKEHOLDERS
IN TECH DESIGN
Methods
Tools and Techniques
Collaborative process
GAP
Lack of active stakeholders’
collaboration in the design
process of learning analytics

8. Gaps in research for co-designing LA
8
Unclear definition of
the distinctive challenges of
co-design learning analytics.
Lack of understanding on
how to support designers
and researchers in co-
designing with stakeholders
for learning analytics.
The role of the
practitioner/researcher when
engaging with stakeholders
for co-design in learning
analytics.

9. Research questions
9
How do co-design techniques assist in the integration
of diverse stakeholders in the LA design process?RQ1
What is the role of the Co-design
practitioner/researcher in the LA design process?RQ2
What are the obstacles/challenges to engaging
stakeholders in the LA design process?RQ3

10. 10
Design-based research + DT iterations

11. 11
Design-based research + DT iterations
Inside each iteration

12. 12
Analysis method: Thematic Analysis
ST1: “So if I... All I do is UTS
online. So they’re collecting
data from me when I do
that? When I log into my
course? How do they...?
What data are they
collecting?”
Transcript Assign Code Pre-set Themes
Surveillance
Data literacy

13. 13
Analysis method: Thematic Analysis

14. 14
Analysis method: Critical incidents
Design Vignette
Tool interaction
Facilitator action
Conversation Participant action
Facilitator Action Conversation Tool Interaction
[Narrative]
Exemplify
Would you use
social media data
or blog posts?
DS: Blog posts Points at one
card
[Narrative]
Inquire
What AM would
you use?
DS: Text analysis Moves one card

15. 15
Analysis method: Knowledge Art
Framework
Knowledge art framework
(Selvin & Shum, 2014)
Facilitator
action

16. 16
Case study 1: Feedback in Nursing training
Context
Design an automated learning analytics tool that can provide immediate
feedback to nurses in the classroom while they engage in healthcare
simulations. Sessions with students involve multiple teams working at the
same time and the main observed issue is that students don’t receive
enough feedback after practice.
Participants (12 Sessions 3 Iterations)
15 Learners (L)
2 Teachers (T)
1 Learning Designer (LD)
1 Course Director (CD)
2 Researchers (R)
1 Co-design practitioner (CP)

17. 17
Automated feedback tool design

18. 18
Case study 2: Tracking graduate attributes for data science
students.
Context
This study was conducted as part of an authentic process to create a
learning analytics tool for Graduate Attribute development with students
from MDSI (Masters in Data Science and Innovation). The current tools used
by students provided a limited but well-received help for new and senior
students on the program. However, there was a need for new tools starting
for the current learning analytics component of a blogging platform called
CIC Around offered by the University of Technology Sydney.
Participants: 9 sessions 2 Iterations
16 Learners (L)
2 Teachers (T)
1 Learning Designer (LD)
2 Data Scientists (DS)
2 Course Director (CD)
1 Researcher (R)
1 Co-design practitioner (CP)
2 Developers (Dev)

19. 19
Analytics tool to support graduate attributes development in
CICAround

20. 20
Case study 3: Co-designing OnTask rules to deliver feedback on
scale
Context
OnTask feedback is sent through personalized emails to hundreds of students.
For teachers and academics to deliver so many emails, they must understand
how to set up rules defining conditions in the student data which will trigger
the inclusion of a feedback message in the email. A rule requires for teachers
to select the intended audience for the email, pick a source of data available
from the LMS, assign values and conditions to make sure the message is being
sent to the right students.
Participants (1Session 1 Iteration)
1 Teacher (T)
2 Learning Designers (LD)
1 Faculty Admin (FA)
1 Ed Tech Designer (ED)
1 Co-design practitioner (CP)

21. 21
Generating rules for OnTask to deliver personalized feedback.

22. Thesis Contributions
22
Analysis of the design
challenges for co-design in
LA leading to
recommendations
Analysis of the role of the co-
design practitioner leading to
guidelines
Toolkit for practitioners to
adopt/adapt co-design
techniques for LA co-design

23. 23
Toolkit for practitioners to adopt or adapt co-
design techniques for LA co-design

24. 24
Adopted techniques for co-design in LA.

25. 25
Focus Group
Fabulation
Adopted co-design techniques for LA
design
Collaborative Sketching Collaborative Persona

26. 26
Adapted techniques for co-design in LA.

27. 27
Learner/Data Journeys

28. Typical User Journey Mapping
28

29. Adaptation: Learner/Data Journey
29
Published in: Carlos G. Prieto-Alvarez, Martinez-Maldonado, R., & Shum, S. B. (2018). Mapping Learner/Data Journeys: Evolution of a Visual Co-Design Tool.
Paper presented at the OzCHI'18, Melbourne, Australia.

30. Example of a Learner/Data Journey
30

31. 31
Paper-based Learner/Data Journey
Effectiveness
Opportunity to have a voice
Identify pain points
Visualize opportunities to deliver a first design

32. 32
Interactive Learner/Data Journey
Effectiveness
Synthesises and cleans up multiple student maps
Prompts instant self-reflection
Helps participants understand the role of learning analytics tools.
Produce feedback useful for the co-design practitioner

33. 33
Effectiveness
Teachers
Learners
Interactive Learner/Data Journey
Synthesises and cleans up multiple student maps
Prompts instant self-reflection
Helps participants understand the role of learning analytics tools.
Produce feedback useful for the co-design practitioner

34. 34
LA-DECK (Learning Analytics Design Cards)

35. Card-based design games
35
Product
Design
Card Game

36. LA-DECK Cards
36
Toolkit available http://ladeck.utscic.edu.au

37. Example of participants using the LA-DECK
37Carlos G. Prieto-Alvarez, Roberto Martinez-Maldonado, & Simon Buckingham Shum (2020). LA-DECK: A card-based learning analytics co-design tool.
10th International Conference on Learning Analytics and Knowledge (LAK20, March 2020), Frankfurt, Germany.

38. Summary of movements LA-DECK
38
Learning Objective
User Interface
Analytics Type
Data Source
Analytics Methods
Privacy
Testing Site
Carlos G. Prieto-Alvarez, Roberto Martinez-Maldonado, & Simon Buckingham Shum (2020). LA-DECK: A card-based learning analytics co-design tool.
10th International Conference on Learning Analytics and Knowledge (LAK20, March 2020), Frankfurt, Germany.

39. LA-DECK effectiveness
39
Common learning analytics language
Expressiveness of the deck.
Facilitating creative combinations of information and ideas
The cards serve as hints for participants to structure their ideas using in a
language that others could understand.
Categories and cards complete the main components in a LA tool.
The visual representation of the process helps participants navigate options.
Some suits generated more discussion than others, like Privacy and User
Interface. In other cases, ideas converge, and the co-design process continues.

40. LA-DECK effectiveness
40
Common learning analytics language
Expressiveness of the deck.
Facilitating creative combinations of information and ideas
The cards serve as hints for participants to structure their ideas using in a
language that others could understand.
Categories and cards complete the main components in a LA tool.
The visual representation of the process helps participants navigate options.
Some suits generated more discussion than others, like Privacy and User
Interface. In other cases, ideas converge, and the co-design process continues.

41. LA-DECK effectiveness
41
Common learning analytics language
Expressiveness of the deck.
Facilitating creative combinations of information and ideas
The cards serve as hints for participants to structure their ideas using in a
language that others could understand.
Categories and cards provide broad coverage of key components in a LA tool.
The visual representation of the process reminds non-experts of options.
Some suits generated more discussion than others, like Privacy and User
Interface. In other cases, ideas converge, and the co-design process continues.

42. 42
LA-DECK
OnTask Edition

43. Translating technical concepts into cards
43
Visual programming examples

44. 44
LA-DECK
OnTask Cards

45. 45
Using the cards
as expected

46. 46
Failure to
complete the
task

47. Why participants failed to complete the
task?
47
Common learning analytics language
The concepts are too complex for participants to understand pushing them to create their own representations
through post-its.
Expressiveness of the deck.
Technical language used on cards was mostly familiar for IT people but not educators
Agreement/disagreement.
Participants found the object too complex making them disregard the cards and have short conversations
without using the representations.

48. Co-design techniques recommendations
48
The adoption and adaption of co-design tools benefits from
stakeholders’ diversity1
Select the proper combination of tools by understanding
their intended use and limitations2
Tools & techniques can be used to teach participants how
to become better co-design partners.3

49. Co-design techniques recommendations
49
Provide diverse communications channels for participants to
communicate before/after sessions.4
Make design results available for participants.5
The co-design process is not only a software development
task but also a learning opportunity for participants.6

50. Download the co-design toolkit
50
Toolkit available
https://masterprieto.com/playbook.html

51. 51
Analysis of the design challenges for co-
design in LA leading to recommendations

52. 52
Challenges of co-design for LA in practice
SURVEILLANCE Are you following me?
POWER RELATIONSHIPS Students against teachers.
LEARNING DESIGN Let’s start with the
curriculum!!
TEACHING & LEARNING
EXPERIENCE
I’m not an expert but….
DATA & ALGORITHM
LITERACY
What is this “analytics” about?

53. Challenges emerging in LA Co-design
53
SURVEILLANCE
There may be tension between what teachers want to see in students’ data, what
students are willing to share, and the purposes for which data are being used.

54. 54
Students differences in concerns over repercussions when
exposing data about mistakes.
Facilitator: What do you think about sharing this data
with other students?
ST3: I think it’s good as long as it’s not used against us
so we’re not ranked badly or something like that
because that’s…
ST2: I never thought of that.
ST3: It’s quite personal information.
ST2: Information should be shared. If information is
retained, it’s pointless.
Practitioner
Action
Inquire

55. Challenges emerging in LA Co-design
55
LEARNING DESIGN
During a Co-design session it may be hard to distinguish between designing the learning
analytics tools and designing the learning activity/assessment.

56. 56
The course director argues about assessing writing skills through
the current learning design as very subjective.
Facilitator: Did you manage the user interface in CICAround
as a teacher in your case what would you like to see either
to help you map the writing skills.
CD: So is this like feedback to the student like in AWA or is it
for the academics and… The thing is, when you say develop
writing skills, the quality of writing is highly subjective.
CD (Action – points at the data source cards and learning
context card [Blogging activity])
CD: And the other thing there… Yes and the other thing is
how are you going to judge it. Are you going to judge it one
student against the other?, or are you going to judge it in
terms of this student over this time has gotten better, and
what does that mean and what would that look like?.
Practitioner
Action
Inquire

57. Challenges emerging in LA Co-design
57
DATA & ALGORITHM
LITERACY
Learners and teachers may provide valuable insights, or have strong views about, data and
algorithms but most of the time ignore the inner workings of these algorithms and
representations of data.

58. 58
Students’ inexperience with data leads to a misunderstanding of
how information is being analysed.
ST1: It is just having an algorithm and that is say, this, this,
tick, tick, tick, this feedback and then this, this, this, this
feedback.
Facilitator: Do you find it sensible to be tracked during the
simulations?
ST2: I didn't even realize they collected data. How did they
collect data from us?
ST1: So if I... All I do is UTS online. So they’re collecting data
from me when I do that? When I log into my course? How
do they...? What data are they collecting?
ST2: Oh, so your teachers know you’ve done stuff.
ST1: You have done stuff.
ST2: Oh my God, I do everything...
Practitioner
Action
Inquire

59. Challenges emerging in LA Co-design
59
TEACHING & LEARNING
EXPERIENCE
Asking students to co-design LA tools may be challenging for facilitators considering that
learners’ remarks are mostly empirical.

60. 60
Students Session 3
Facilitator: We can provide you feedback, too; your patterns, conversations,
something. Would you prefer to be… Would you return to your table, or [unclear]
the mannequin?
EST3: The mannequin.
EST2: yes
EST2: I think I’d prefer to get feedbacks and remediation when there’s a problem
that arises rather than toward the end when the whole thing has become more
blurred.
Teacher Session 5
TE: I think it’d be really useful in the debrief. I don’t think it’s going to be good
during the sim, because I think they’ll get distracted by it, they’ll be looking at
what they’re doing. I think if any student says, I want that, in the middle of a sim, I
think it’s going to be detrimental; it’s going to…
There is a tension between students’ preference and teachers’
pedagogy expertise in terms of automated feedback provision.
Practitioner
Action
Inquire

61. Challenges emerging in LA Co-design
61
POWER RELATIONSHIPS
Respect the contributions that every stakeholder brings while addressing the potential
tensions or conflicting perspectives that may appear amongst stakeholders holding
asymmetric power relationships.

62. 62
Students inexperience with data leads to a misunderstanding of
how information is being analysed.
Facilitator: Do you think stakeholders’ position and ability to influence the design process had an
impact in your project?
Practitioner/Researcher: So, because of [Course Director] came in… students were more engaged into
the… Into this research. Then we could have access to students. And [Course Director] said, yay, it will
be good, to all the students. It will be good if you had some time like next week to do some
interviews with [Practitioner/Researcher]and [Supervisor], because blah, blah, blah. It was because
of [Course Director] comments to students that we could get students. They also suggested to add
question that were not part of my research questionnaire.
Practitioner/Researcher: I was completely lost actually. I think that I won't be, I wouldn't be able to
finish my PhD at this point.
Practitioner
Action
Inquire

63. 63
Challenges
Surveillance
Teaching and
learning
expertise
Learning design
Data/Algorithm
Literacy
Power
relationships
Learning inexperience leads
to unfeasible learning
design requests
Low data literacy limits
privacy negotiations
Some challenges are connected and lead to other discussions

64. 64
Analysis of the role of the co-design practitioner
/researcher

65. 65
Identifying roles and making decisions
“Knowledge Art” framework
Decisions
Decisions in co-design for LA.
Identify the roles

66. 66
New glossary of actions
Represent
InterveneGuide Adapt
InquireExemplify

67. 67
The co-design practitioner
Represent
Responsible of planning, managing
and implementing co-design
methods and techniques into the
learning analytics design process.

68. 68
Co-design practitioner as a researcher
Represent
Adapt
Exemplify
Researcher
Contributes to learning analytics and co-design research
Brings in other researcher skills to assist
Documents the co-design process

69. 69
Co-design practitioner as a facilitator
Represent
Adapt
Exemplify
Facilitator
Inquire stakeholders into reflection
Maintains the participatory landscape clean after constant
deliberation
Improvises and avoids the anchoring effect
Balances personal research interests with the needs of
stakeholders, and quality of the design process

70. 70
Decision process in co-design for LA
Process

71. 71
Relevance for the LA field.
The co-design practitioner for LA must act not only as a facilitator
to guide the co-design process but must also balance their
researcher role.
The toolkit and recommendations allow LA designers to bring
collaboration into their projects.
Co-design practitioners for LA can prepare and be aware of the
emerging challenges before starting their collaborative process.

72. @MasterPrieto
72
www.Masterprieto.com
ORCID: 0000-0002-7282-6069
Q & A