This document summarizes research on developing a methodology for generating Learning Analytics Dashboards (LADs) that effectively support sensemaking and decision-making. The methodology combines participatory design with stakeholders and generative design techniques. A tool called LADStudio was created to help designers rapidly prototype LADs based on specifications from co-design processes. An evaluation study found the tool had satisfactory usability and user experience. Future work includes collecting more user proposals and addressing barriers to adopting the LAD design methodology in practice.

1. Generating LADs that
make sense
CSEDU 2023 ✤ 22/04/2023
Madjid SADALLAH* & Jean-Marie GILLIOT
MOTEL, IMT Atlantique / Lab-STICC. Brest, France
*madjid.sadallah@imt-atlantique.fr | https://www.madjidsadallah.net/

2. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Roadmap
Context & Research questions
1
Theoretical framework
2
LADStudio
3
Evaluation study
4
Conclusion
5
2

3. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Why Learning Analytics Dashboards (LADs) Design Matters
• LADs : visualize educational data resulting from LA process
• Well-designed LADs: effective to awareness & decision-making
– sustain the learning process & improve its outcomes
• …but hard to design → Limited LAD adoption
– little or no stakeholder involvement in the design process
 inadequate addressing of their needs and expectations
 insufficient support for their abilities e.g., visual literacy
– poor understanding of the associated decision-making process:
how the sensemaking with LADs occurs
3
Our aim : “A design methodology that addresses these factors for effective LADs”

4. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Research project
4
Guiding Design
Design Space
DEFLAD
Ideation:
Codesign toolkit
PaDLAD
Prototyping:
Generating LADs
that make sense
LADStudio
(Prieto et
al., 2018)

5. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Current stage, in other words
From this: To this:
5

6. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Research Questions
6
How can the decision-making process be reflected on a LAD?
RQ1
How to support designers in translating design specifications into
LADs, with explicit support for the decision-making process?
RQ2

7. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Making sense of LAD sensemaking
“
“Integrate the sensemaking dimension in LAD
design”
• Sensemaking = process of creating meaning from complex data
• SM with LADs = process of interpreting data presented on the
LAD and using it to inform and support decision-making
7
How can the decision-making process be reflected on a LAD?
RQ1

8. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Making sense of LAD sensemaking
Distributed cognition theory (Hutchins 1995)
• Sensemaking develops through perception
and interaction
Data/Frame theory of SM (Klein 2006)
• LAD displays = Data
• Gained insight = frame
8
How can the decision-making process be reflected on a LAD?
RQ1
“Interaction-Sensemaking Loop”
LAD design: adequate support needed for accurate frame construction (e.g.,
effective interaction and appropriate representations)

9. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Making sense of LAD sensemaking
“
DEFLAD : LAD Design Framework, defining a refined design space
• Context of use
• LAD goal, focus and targeted SA level
• Stakeholders, timing and info circulation
• Interaction & visualization
• Data & representation
• Types of Interactions
• Explicit SM features at different levels (★)
9
How can the decision-making process be reflected on a LAD?
RQ1

10. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Generative CoDesign Methodology
A methodology to combine
participatory design & generative design
• Participatory design
– high levels of stakeholder involvement
– stakeholders’ needs, requirements and abilities
• Generative design
– rapidly generate prototypes compliant with stakeholders’ reqs and designer’s
specs
– leveraging the computational power and efficiency
10
Supporting designers in prototyping LADs with explicit SM processes?
RQ2

11. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Generative CoDesign Methodology
Generative codesign
methodology
• Clear and structured
approach
• Encourage stakeholder
involvement
• Comply with end-user needs
• Rapid prototyping
11
Supporting designers in prototyping LADs with explicit SM processes?
RQ2

12. M. Sadallah. Generating LADs that make sense. CSEDU 2023
LADStudio, a tool for generating codesigned LADs
• Prototyping tool that implements the generative codesign
approach: from needs capturing to their realization
– Specifications developed through co-design process
– Built within the proposed design space
– Instrumented with features to support end-user SM and DM processes
– A platform for stakeholder involvement and customization
• Enables designers to rapidly obtain functional prototypes.
12
4 modules
1. Library of components
2. Specification module
3. Generation module
4. Grafana instance

13. M. Sadallah. Generating LADs that make sense. CSEDU 2023
LADStudio : library of components

14. M. Sadallah. Generating LADs that make sense. CSEDU 2023
LADStudio : LAD specification & generation

15. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Qualitative evaluation study
• Evaluate usability and user experience of LADStudio
• 13 Participants: purposive sampling
– Inclusion: experience with EdTools
– Exclusion: unwillingness to participate
• Evaluation procedure
– LADStudio demonstration
– Independent experimentation
– Collective codesign workshop
– Questionnaire and open-ended questions
– Duration: 2 hours
15

16. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Evaluation study: instruments
• Usability evaluation
– System Usability Scale (SUS)
questionnaire
– Reliable usability measure
• User Experience (UX) evaluation
– User Experience Questionnaire (UEQ)
– Valid tool for UX assessment
– 26 items, 6 scales: attractiveness,
efficiency, perspicuity, dependability,
stimulation, novelty
16

17. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Results: usability
• Highest scores
– Q1: … able to use it frequently
– Q3: … easy to use
– Q5: …components well integrated
• Lowest scores
– Q2: … unnecessarily complex
– Q4: …need support
– Q6: …too much inconsistency
17
SUS score: 71.15
System usability rated as OK

18. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Results: UX
• Highest score
– novelty
– efficiency
– stimulation
• Lowest score
– perspicuity
18
UEQ score: 1.99
Rating : Good!

19. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Results: Participants' Feedback
• Challenging at first but effective with practice
• Capable of creating useful and well-designed LADs
• Reasoning mirroring, support and guidance, at design
Improvement areas
• Challenging theoretical concepts
• Need design support ; lack of data/visual literacy
19

20. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Discussion
• Usability & UX: satisfactory
• Encouraging feedback
• Some limitations
– self-selection bias
– small sample size restricts generalizability,
– further investigation needed for LAD quality
20

21. M. Sadallah. Generating LADs that make sense. CSEDU 2023
Conclusion & outlook
• Efficient and involved LAD design methodology
– participatory & generative
• LADStudio: functional prototypes from co-design specs
– Success in innovative proposals and LAD adoption
• Next steps:
– collect user and practitioner LAD proposals
– address design adoption issues, and
– share findings with the learning community.
21

22. Thank you for your
attention!
22
Madjid SADALLAH
madjid.sadallah@imt-atlantique.fr
https://www.madjidsadallah.net/