Presentation at ascilite 2015, Perth, Australia. Full paper: https://www.academia.edu/19123100/An_enhanced_learning_analytics_plugin_for_Moodle_student_engagement_and_personalised_intervention

1. An enhanced learning analytics plugin for Moodle
STUDENT ENGAGEMENT AND PERSONALISED INTERVENTION
Monday 30 November 2015 | ascilite 2015

2. Learning analytics
in Moodle
CONTEXTS AND APPROACHES
Danny Liu

3. Learning analytics in Moodle
WHY?
• Benefits
• Familiarity for staff
• Single point of access
• Student activity already
in database – what data
are most valuable?
• Open source
http://analytics.jiscinvolve.org/wp/files/2015/10/Learning-analytics-framework.jpg
e

4. Learning analytics in Moodle
BUILT-IN REPORTING
Logs
Activity report
Activity completion
Statistics report

5. https://moodle.org/plugins/view/block_analytics_graphs
Learning analytics in Moodle
CURRENTLY AVAILABLE PLUGINS - VISUALISATION
Analytics Graphs
GISMO
https://moodle.org/plugins/block_gismo
Moodle Activity Viewer
MOCLog
Mazza et al. (2012)
Jones & Clark (2014)

6. Learning analytics in Moodle
CURRENTLY AVAILABLE PLUGINS - ANALYSIS
SmartKlass
https://moodle.org/plugins/local_smart_klass
Engagement Analytics
https://moodle.org/plugins/browse.php?list=set&id=20

7. Learning analytics in Moodle
A FRAMEWORK FOR EVALUATION AND IMPROVEMENT
• IRAC framework (Jones, Beer, & Clark, 2013)
Jones, D., Beer, C., & Clark, D. (2013). The IRAC framework: Locating the performance zone for learning analytics. In: Proceedings of ascilite 2013, Sydney.
Only and all relevant information available
Needs to aid the task being undertaken
Providing support for context-appropriate action
Iterative development of other three factors
Information
Representation
Affordances for action
Change

8. Moodle Engagement Analytics
PLUGIN OVERVIEW
• Originally developed by Phillip Dawson, Adam Olley, and Ashley Holman
Moodle
Report
Logins Forums Assessments
Parameters
Traffic
lights
BYO
Moodle Engagement
Analytics Plugin
Indicators
Action

9. Requirements
gathering and
iterative design
ANALYSIS AND DESIGN
Chris Froissard

10. ?
Extend capability and functionality of
MEAP
++

11. 1) IV1-Unit convenor survey around expectations for student early
alert system
2) IV2-Unit convenor survey around use of MEAP
4) User testing with use case (5 video sessions)
3) Prototyping-development of interface
5) Refinement of design based on usage and feedback
Overview of design process

12. What information woud you like to see to help
you decide what to include in your alert?
What would you find most useful about a
student alert system?
Collect information about expectations
1) IV1-Unit convenor survey around expectations for student early
alert system
2) IV2-Unit convenor survey around use of MEAP

13. Prototyping
1) IV1-Unit convenor survey around expectations for student early
alert system
2) IV2-Unit convenor survey around use of MEAP
3) Prototyping-development of interface

14. Prototyping: concepts

15. Prototyping: paper designs

16. Prototyping: mockups

17. User testing
1) IV1-Unit convenor survey around expectations for student early
alert system
2) IV2-Unit convenor survey around use of MEAP
4) User testing with use case (5 video sessions)
3) Prototyping-development of interface

18. User testing: scripts

19. User testing: recordings

20. 1) IV1-Unit convenor survey around expectations for student early
alert system
2) IV2-Unit convenor survey around use of MEAP
4) User testing with use case (5 video sessions)
3) Prototyping-development of interface
5) Refinement of design based on usage and feedback
Refinement of design

21. Refinement of design based on UAT

22. Refinement of design based on
usage in pilot
22

23. Moodle Engagement
Analytics Plugin
DEVELOPMENT, IMPLEMENTATION,
AND EVALUATION
Danny Liu

24. Enhancements
MODIFYING AND ADDING INDICATORS
I
R
A
C
Assessments Gradebook
• Submission
of quizzes,
assignments
• Added
Turnitin

25. Enhancements
ALTERNATIVE REPORT
I
R
A
C
• Less abstracted representation of student engagement data

26. Enhancements
26
AFFORDANCES FOR ACTION
• Customisable,
personalised
messages
directly to
students
• Suggested and
other snippets
(some from
PassNote)
• Save messages
for re-use
I
R
A
C

27. MEAP+
27
ENHANCED PLUGIN FOLLOWING IRAC FRAMEWORK
Moodle
Report
Logins Forums Assessments
Parameters
Traffic
lights
Gradebook
Moodle Engagement
Analytics Plugin
Action

28. Preliminary outcomes
28
IMPACT OF MEAP AT MACQUARIE
• 3400+ personalised emails sent, average ~46% opened
0
50
100
150
200
250
300
350
01Aug
08Aug
15Aug
22Aug
29Aug
05Sep
12Sep
19Sep
26Sep
03Oct
Numberofstudents
0
20
40
60
80
100
AreyouhavingproblemsaccessingiLearn?
ACCG250-Ifyousnoozeyoulose!
AreyouhavingtroubleloggingintoiLearn?
AreyouhavingtroubleloggingintoiLearninACCG100?
MECH201:EngineeringDynamics-iLearnActivity
ENGG150quizfeedback
ECON110iLearnLogin
AHIS140MythintheAncientWorld
AHIS140MythintheAncientWorld
AreyouhavingtroubleloggingintoECON111
AHIS140Session2Participation
AHIS140MythintheAncientWorld
Quiz1
Quiz1Result-Afin253
Ilearn-Afin253
ACST101-Quiz1
ECON110TutorialSubmissions
ECON111-Tutorialscontributetoyourfinalgrade
ECON111-Tutorialscontributetoyourfinalgrade
MECH202Progressthusfar
MECH203Progressthusfar
ISYS114
ACST101-Lastdaytowithdrawwithoutfailchargeis…
ISYS114Performance
ISYS114weeklytutorialsubmissions
ISYS114AttendanceandPerformance
ISYS114WorkshopAttendanceandPerformance
ISYS114Workshopattendance
Tutorialfeedback
Tutorialparticipation
TutorialparticipationforEUL101SocietiesofEurope
ACST101-Mondayisthelastdaytowithdrawwithout…
ACST101-Mondayisthelastdaytowithdrawwithout…
ACST101-Mondayisthelastdaytowithdrawwithout…
Itsnottoolatetogethelp!
ACST101-Mondayisthelastdaytowithdrawwithout…
ECON111-Pleasecheckyourtutorialrecords
ECON111-pleasecheckyourtutorialrecords
Percentageopened
1 hour 1 day 1 weekEmail opened within:

29. Preliminary outcomes
29
IMPACT OF MEAP AT MACQUARIE
• Who: unit
convenors and
student support
staff
• What: census,
updates, reminders
• Why: predominantly
for at-risk students
• How: logins,
assessment
submissions, grades,
attendance
I was surprised someone cared/was actually monitoring, kind
of a weird, I don't know totalitarian/'people are watching you'
feeling? But in this situation I was happy.
I'm glad I was contacted as it motivated me to complete the
work and almost "show" the unit convenor that I could do
it.
He gave me specific advice and encouraged me and it made
me feel much better.
The email basically kicked me into gear and I completed all my
assessments post-email to a high level.
Very useful. I wouldn't have been able to do such a large scale
analysis and identify so many students without MEAP. I
wouldn't have been able to send them such tailored, structured
and consistent messages.

30. Preliminary outcomes
30
THE DARK SIDE
• Issues around
• Message
composition
• Suggestions
• Student contexts
• Being labelled
I was contacted but in a manner that suggest I should drop out of
the course and not waste the convenor's time. She didn't ask
whether I was experiencing any issues outside of university,
but simply that I should transfer course if I can't handle the
workload.
Being labelled as lazy when you're doing your best and
don't have any other choice is quite sad.
Being aware and then being told of my own inadequacies is
confronting and, yes, does make me feel worse about life in
general. It's something I need to be told and is still that extra bit of
motivation.
The email was worded in a way that it the unit [convenor] was
trying to tell me I was doing horrible and should drop out and
didn't refer any help.

31. Future directions
31
NEXT STEPS FOR MEAP+
• Further work
• Measuring impact of
interventions
• Automated determination
of parameters (machine
learning)
• Investigating staff use
• Alternative forms of
representation
• Opening source
• Institutionalisation

32. Acknowledgements
IT TAKES A VILLAGE
• Students
• Unit convenors
• Student support staff – Janise Farrell
• Learning systems – James Hamilton, Jaymie Parker, Michael Mendoza
• Reference group
• Team members – Deborah Richards, Amara Atif
• 2015 Macquarie University Teaching Development Grant

33. Get in touch
@dannydotliu
@jcfroissard
danny.liu@mq.edu.au
chris.froissard@mq.edu.au