The big data revolution is an exciting opportunity for universities, which typically have rich and complex digital data on their learners. It has motivated many universities around the world to invest in the development and implementation of learning analytics dashboards (LADs). These dashboards commonly make use of interactive visualisation widgets to assist educators in understanding and making informed decisions about the learning process. A common operation in analytical dashboards is a ‘drill-down’, which in an educational setting allows users to explore the behaviour of sub-populations of learners by progressively adding filters. Nevertheless, drill-down challenges exist, which hamper the most effective use of the data, especially by users without a formal background in data analysis. Accordingly, in this paper, we address this problem by proposing an approach that recommends insightful drill-downs to LAD users. We present results from an application of our proposed approach using an existing LAD. A set of insightful drill-down criteria from a course with 875 students are explored and discussed.

1. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards
Automated Insightful Drill-Down Recommendations for
Learning Analytics Dashboards
Dr Hassan Khosravi
h.khosravi@uq.edu.au
Professor Marta Indulska
m.indulska@uq.edu.au
Shiva Shabaninejad
s.shabaninejad@uq.edu.au
Dr Aneesha Bakharia
a.bakharia1@uq.edu.au
Dr Pedro Isaias
pedroisaias@gmail.com

2. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 22
Overarching Research Question Under Investigation
 Given the existing challenges in algorithmic bias and predictive models, can
we use AI to guide data exploration while still reserving judgement about
interpreting student learning to instructors?

3. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 33
Educational Data and Learning Analytics Dashboards
Source: http://flc.learningspaces.alaska.edu/?p=4425
The increasing use of technology in education
enables universities to collect rich data on learners.
Learning Analytics Dashboards (LADs) have emerged
as a field to help make sense of data on learners

4. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 44
Learning Analytics Dashboards
LADs are defined as a tool that aggregates the data about learner(s), learning process(es)
and/or learning context(s) captured from digital educational systems into one or multiple
visualizations (Schwendimann, 2017).

5. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 55
The Educational Data Revolution
As the volume, velocity, and variety and veracity of learner data
increases, making sense of Learner data becomes more challenging
Source: https://www.ibmbigdatahub.com/infographic/four-vs-big-data

6. Automated Insightful Drill-Down Recommendations for Learning Analytics DashboardsDevelopment and Adoption of an Adaptive Learning System
Manual and Smart Drill Downs
Finding Insights in LADs
Application
Conclusion and Future Work
Automated Insightful Drill-down (Aid) Recommendations

7. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 77
Finding Insights In LADs
Predictive LADs
 Provide automatic decision-making.
 e.g., label students based on their estimate
of success.
Predictive LADs
 Provide automatic decision-making.
 e.g., label students based on their estimate
of success.

8. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 88
Finding Insights In LADs
Exploratory LADs
 Allow users to navigate through multi-
dimensional data sets.
 Relies on user’s judgement for understanding
and interpreting the results.
Exploratory LADs
 Allow users to navigate through multi-
dimensional data sets.
 Relies on user’s judgement for understanding
and interpreting the results.
Curiosity Driven Exploration
 Can be used to answer specific questions
 e.g. ‘how do new students that have failed the
midterm compared to other students’)
Curiosity Driven Exploration
 Can be used to answer specific questions
 e.g. ‘how do new students that have failed the
midterm compared to other students’)
Operations in Online Analytical Processing (OLAP)
Source::https://senturus.com/blog/reporting-multidimensional-olap-data-sources/

9. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 99
Finding Insights In LADs
Exploratory LADs
 Allow users to navigate through multi-
dimensional data sets.
 Relies on user’s judgement for understanding
and interpreting the results.
Exploratory LADs
 Allow users to navigate through multi-
dimensional data sets.
 Relies on user’s judgement for understanding
and interpreting the results.
Operations in Online Analytical Processing (OLAP)
Source::https://senturus.com/blog/reporting-multidimensional-olap-data-sources/
Data Driven Exploration
 Finding insightful features for drill-downs
 e.g. ‘Identify the student attributes of
students that have failed the midterm”.
Data Driven Exploration
 Finding insightful features for drill-downs
 e.g. ‘Identify the student attributes of
students that have failed the midterm”.

10. Automated Insightful Drill-Down Recommendations for Learning Analytics DashboardsDevelopment and Adoption of an Adaptive Learning System 3
Finding Insights in LADs
Application
Conclusion and Future Work
Automated Insightful Drill-down (Aid) Recommendations
Manual and Smart Drill Downs

11. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1111
All Possible Drill-down ActionsAll Possible Drill-down ActionsSample Students DatasetSample Students Dataset
Finding Insightful Features
Student
#
Residential
Status
Video
Engagement
Assessment
Score
S1 Domestic High High
S2 Domestic High High
S3 Domestic High High
S4 Domestic High High
S5 Domestic High High
S6 Domestic High Mid
S7 Domestic Low Low
S8 Domestic Low Low
S9 Domestic Low Low
S10 International High Low
…
Target
Insightful drill-downs

12. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1212
Challenges With Manual Drill Downs
Too many drill-down choices
Lack of insightful results

13. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1313
Challenges With Manual Drill Downs
Drill-down fallacies: incorrect reasoning for a deviation

14. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1414
Smart Drill-down Approaches

15. Automated Insightful Drill-Down Recommendations for Learning Analytics DashboardsDevelopment and Adoption of an Adaptive Learning System 3
Finding Insights in LADs
Application
Conclusion and Future Work
Manual and Smart Drill Downs
Automated Insightful Drill-down (Aid) Recommendations

16. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1616
Automated Insightful Drill-down (AID) Recommendations
 Given the existing challenges in algorithmic bias and predictive models, can
we use AI to guide data exploration while still reserving judgement about
interpreting student learning to instructors?

17. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1717
Aim and Problem Statement
Aim: Automatically recommend insightful drill-downs

18. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1818
Automated Insightful Drill-down (AID) Recommendations

19. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 1919
Approach
 AID process steps:
1. A decision tree classification method to
examine the insightfulness of a set of
promising drill-down paths
2. KL-divergence to rank and select the highest
ranked path

20. Automated Insightful Drill-Down Recommendations for Learning Analytics DashboardsDevelopment and Adoption of an Adaptive Learning System
Finding Insights in LADs
Automated Insightful Drill-down (Aid) Recommendations
Conclusion and Future Work
Manual and Smart Drill Downs
Application

21. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2121
The Course Insights Platform
• Course Insights is a LAD
that provides filterable
and comparative
visualisations
• Course Insights aims to
provide actionable
insights for teachers by
linking data from several
sources

22. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2222
Insights: Manual Drill-downs
Aim: To study how manual drill-downs are used in LADs.
Research Questions
• RQ1 How complex are the manual drill-downs applied by users?
• RQ2. Which manual drill-downs are commonly applied?
Data set: 356 manual drilldown actions performed by 71 teaching staff members
who were involved in teaching courses that used Course Insights at The University of
Queensland in 2019.
Aim: To study how manual drill-downs are used in LADs.
Research Questions
• RQ1 How complex are the manual drill-downs applied by users?
• RQ2. Which manual drill-downs are commonly applied?
Data set: 356 manual drilldown actions performed by 71 teaching staff members
who were involved in teaching courses that used Course Insights at The University of
Queensland in 2019.

23. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2323
Insights: Manual Drill-downs – Results
RQ1 How complex are the manual drill-downs applied by users?
Results: 84% of drill-downs were performed on a single attribute and the average
number of attributes for each drill-down was 1.35 ± 0.55
RQ1 How complex are the manual drill-downs applied by users?
Results: 84% of drill-downs were performed on a single attribute and the average
number of attributes for each drill-down was 1.35 ± 0.55
RQ2 Which manual drill-downs are commonly applied?RQ2 Which manual drill-downs are commonly applied?
Mostly used to
investigate simple
curiosity driven
questions.
Mostly used to
investigate simple
curiosity driven
questions.

24. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2424
Insightful Drill-Downs in Action – Case Study
Location: The University of Queensland (UQ)
Course: Calculus and linear algebra
Number of students: 875
Selected attributes: {Brand New, click-stream,
Gender, Program, Residential Status}
Target Feature: Final exam
Possible drill down actions: 1728

25. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2525
Insightful Drill-Downs in Action – Case Study
Main Findings
1. AID can generate insightful (high significance score) drill-down recommendations
instantly.
2. The average length of a drill-down action is not a good indicator of its insightfulness.
3. plays a very important role in the quality of the generated recommendations.
Main Findings
1. AID can generate insightful (high significance score) drill-down recommendations
instantly.
2. The average length of a drill-down action is not a good indicator of its insightfulness.
3. plays a very important role in the quality of the generated recommendations.

26. Automated Insightful Drill-Down Recommendations for Learning Analytics DashboardsDevelopment and Adoption of an Adaptive Learning System 3
Finding Insights in LADs
Automated Insightful Drill-down (Aid) Recommendations
Application
Manual and Smart Drill Downs
Conclusion and Future Work

27. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2727
Conclusion
Automated Insightful Drill-down (AID): Guiding data
exploration while still reserving judgement to
instructors.
Insights gained from an application of AID in a real life
data set.
Challenges in getting insights from students data in
learning analytics dashboards (LADs)
Filtering students’ data provides a potential solution.
But it is time consuming and error prone.

28. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2828
Future Work
• Extend AID to suggest promising values of
• Partner with academics to investigate:
• criteria for determining insightfulness
• best ways to present recommendations
• Extend AID to suggest promising values of
• Partner with academics to investigate:
• criteria for determining insightfulness
• best ways to present recommendations

29. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards 2929
Future Work
performed
From recommendations to weekly actionable Insights
To be included in the JLA extension submission

30. Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards
Automated Insightful Drill-Down Recommendations for
Learning Analytics Dashboards
Dr Hassan Khosravi
h.khosravi@uq.edu.au
Professor Marta Indulska
m.indulska@uq.edu.au
Shiva Shabaninejad
s.shabaninejad@uq.edu.au
Dr Aneesha Bakharia
a.bakharia1@uq.edu.au
Dr Pedro Isaias
pedroisaias@gmail.com