The relationship between student engagement, progression, and attainment: NTU's experience of learning analytics

1. The relationship between student engagement, progression and
attainment: NTU’s experience of learning analytics
ABLE Project 2015-1-BE-EPPKA3-PI-FORWARD

2. NTU’s experience of learning analytics
• Dashboard overview
• Developing the NTU Student Dashboard
• Addressing disparities in student attainment
• Plans for future developments

3. What is the NTU Student Dashboard?

4. Learning Analytics
"Analytics is a term used
in business and science
to refer to computational
support for capturing
digital data to help
inform decision-making
… Learning Analytics
appropriates this concept
for education.”
(Buckingham Shum,
2012, p.1)
(Clow, 2012)

5. Why NTU is interested in learning analytics?
• Reduce barriers to being known by a tutor
• Space for tutors to make notes & plan
• Students as agents
• Can see own grades, and compare self to
peers
• Feedback
• Tool for academic use
• Potentially promotes virtuous behaviour
• Alerts tutors if students are high risk for
leaving early
• Students can compare their engagement
with their peers
Attainment
Progression
Belonging
Strategic
information
• Insights into groups at risk
• Potentially design and delivery of
courses

6. What does the Dashboard do?
NTU
Student
Dashboard
Student biographical
info, e.g. enrolment
status
Evidence of student
engagement
• Door swipes
(where appropriate)
• Library books
• NOW use
• Dropbox
submissions
• Attendance data
• Access to e-
books and
journals through
Shibboleth
authentication
Staff
view
Student
view
Compares student
engagement across
the cohort & gives
rating
Can make
comments in
free text box
Raises
alerts!!

7. The NTU Student Dashboard
• The Dashboard monitors students’ engagement with their course
– Door swipes, Library use, log ins to the VLE & submission through NOW Drop
box
• It compares this data to a profile from previous years and assigns
the four ratings:
– High
– Good
– Partial
– Low
– Not fully enrolled (for students who are only eligible to enroll at the start of the
year, or who have withdrawn)
• Students and staff can see exactly the same view
– Staff have a few additional management screens
• Tutors can also make notes in the notes section
• Staff are also sent an alert if there is no engagement for two weeks
during term time

8. What does the student view look like?

10. Opens on click Opens on click
• Scroll down landing page to
find more information.
• Dashboard designed to be
simple at first glance, with the
option to click for extra detail.
• Built in partnership with
Solutionpath.

11. Student profile

12. • This data is drawn
automatically from
the attendance app
• The wheel shows
overall attendance
• It is therefore likely
that there will
appear to be a
discrepancy
between weekly
and overall
attendance,
particularly later in
the year
• As the data is
presented the next
day, it is important
to keep attendance
data up to date

13. Assessment & Feedback View
This view will only show assessments and feedback
submitted through NOW

14. Induction trial
Designed to give
lecturers information for
early tutorials and to help
students reflect back
later in the year

15. What does the staff view look like?
Opens link

16. Tutorial Group View
Links to group
of students in
engagement
category

17. Links to
student’s
dashboard
Able to sort on headings
Search function
• Designed so staff have easy access to student data.
• Allows staff to quickly identify potentially at risk students

18. Development of Dashboard

19. Developmental Cycle
• Pilot Phase (2013-14)
– 4 courses, 40 staff & 500 1st year students
– Willing participants – very positive staff feedback,
limited student awareness
• Phase 1 (2014-15)
– Launched in 8 of the 9 Schools, briefings for all
Schools
– Developed formal governance structure and tied
into academic committee structure
– Lots of problem-solving
– Ethical policy written
• Phase 2 (Sept 2015-February 2016)
– New data sources added
– Launched in all Schools
– Assessment view
• Phase 3 (now)
– Personalisation, embedding into University
systems

20. Staff consultation
• Pilot year survey & interviews (2013-14)
• Staff briefings & online user survey (2014-15)
• Workshops (2015-16, 2016-17)
• iPad pilot surveys (2016-17)
• Feedback to service desk (ongoing)

21. Dashboard Governance Group
Dashboard Operations Group
Informal student
group
Ethics Group University Systems Group
Academic Standards & Quality
Committee
University Governance &
Management
Academics
Students
Educational developers
Student Support
Student Planning
Information Systems

22. Student consultation
• Pilot year focus groups (2013-14)
• Transition surveys (2012-13 – onwards)
– Feb- Mar each year, approx. 500 respondents
• Questionnaires
• Student research 1 – 1 interviews
• Students involved in ongoing development
– Governance group, student placements

23. Student research 2014 - 15
Student focus groups (2014)
• Familiar with being monitored
• Transparency - “I want to be able to see what is collected on me so I know
what they can see about me”
Student Transition survey (February - March 2015)
• Is there anything NTU could do to encourage you to log into the Dashboard
more frequently?
• Have you done anything differently as a result of using the Dashboard?
– Changing behaviours including those that aren’t monitored
• How could NTU improve the Dashboard?

24. What does the Student Dashboard provide?

25. Picture of Overall Engagement
• Engagement of year (mode) for different cohorts

26. Relationship between average engagement & progression
1st year students only Sept 2014 –June 2015

27. Relationship between engagement and final
degree attainment

28. Addressing disparities in student attainment

29. How are we using the Dashboard to address
disparities in attainment?
• Two change agents
• Developing learning analytics is challenging
• Institutional change based on learning analytics may be a different
order of magnitude
– We already have data on students at risk, attendance, non-submission of
coursework
– Yet it is often extremely difficult to change student outcomes
Students Staff

30. Staff as Change Agents
• Personal tutor
– Induction activity
– Primary contact for alerts
– Ongoing support
– iPad trial
• Referrals to support services
– Notes and referrals function
• Staff communication
– Briefings and drop-in sessions
– Newsletter term 1 – low engagement
– Newsletter term 2 – alerts

31. Newsletter 1 - Relationship between engagement
at key times of year and progression

32. Newsletter 2 - No engagement alerts
• Any one alert is an indicator of risk
• Students with multiple alerts had lower incidence of progression

33. Students as change agents
• Dashboard as resource to inform students about their engagement
• Dashboard focuses positively on engagement rather than risk
• Challenges around Success for All
• Focus on communication and development of tool in-line with
student views
• https://youtu.be/CThv-Dcwb54

34. Feedback from the Student Transition Survey
• The transition survey was conducted with first year students (Feb/March 2016)
• 91% reported using the Dashboard at least once (90% in 2015)
Have you logged into the NTU student Dashboard?
When using the Dashboard, how often have you explored the following?
Base: 515 (2016), 469 (2015)
4%
5%
14%
15%
13%
24%
5%
6%
26%
24%
35%
36%
12%
13%
33%
34%
34%
36%
79%
77%
28%
28%
19%
5%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Spoke to someone providing specialist help (for
example student support services/ library) as a result
of looking at information on the Dashboard
Spoke to your tutor as a result of looking at
information on the Dashboard
Changed your behaviour to raise or maintain your
engagement score (for example made sure that you
swiped to go into a building)
Compared your engagement score with other students
on your course
Increased the amount of time you spend studying
Checked your own engagement score
Very Often Often Sometimes Never

35. Relationship between number of Dashboard
log-ins and engagement classification

36. Relationship between Dashboard log-ins and
progression

37. Plans for future development

38. IT infrastructure, management processes & quality
New data sources for the learning analytics model
Further Dashboard developments: NECs, design, use of notes etc.
Communications
Integrating Dashboard into institutional working practices
Stuff we haven’t considered yet

39. Thank you for listening.
Any questions?