Using NTU's Student Dashboard to open new insights into successful students and successful staff interventions

1. Using NTU’s Student Dashboard to open new insights into successful
students and successful staff interventions
Rebecca Edwards & Ed Foster

2. Presentation overview
• Introduction to STELA and ABLE projects
• Understanding how to improve student engagement
• Using engagement data to measure the impact of interventions
• Next steps – pilot studies at NTU

3. STELA Project
• The main goal of the STELA project is to enhance a successful
transition from secondary to higher education by means of
learning analytics.
• The STELA project…
 Involves designing and building student and staff facing analytics dashboards
 Aims to develop dashboards that go beyond identifying as-risk students;
allowing actionable feedback for all students on a large scale
• NTU is ‘case study zero’ following involvement in “What works?”
Student Retention and Success and based on experience with the
NTU Student Dashboard
STELA Project: 562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD

4. ABLE Project
• The main goal of the ABLE Project team is to research strategies
and practices for using learning analytics to support students
during their first year at university. Our work focuses on both
developing the technological aspects of learning analytics and, more
importantly, on how it can be used to actually support students.
• The ABLE project…
 Involves developing the technological aspects of learning analytics
 Focuses on how learning analytics can be used to support students
• NTU is project lead
ABLE Project: 2015-1-BE-EPPKA3-PI-FORWARD

5. Understanding how to improve student
engagement
• NTU has three years’ worth of data to demonstrate that
engagement is an important factor in student success
• Challenges around being able to improve a student’s engagement,
and if this will change a student’s trajectory
• The NTU Student Dashboard offers two advantages:
Tool to
provide data
Tool to
analyse
impact

6. Logging into the NTU Student Dashboard
• Student Transition Survey (STS) results show that many students
report changing behaviour based on seeing data in the Dashboard
0% 20% 40% 60% 80% 100%
Spoke to someone providing specialist help (for example
student support services/ library) as a result of looking at
information on the Dashboard
Spoke to your tutor
Increased the amount of time you spend studying
Changed your behaviour to raise or maintain your
engagement score (for example made sure that you swiped
to go into a building)
Checked your attendance
% Of students who responded 'yes- very often, often or sometimes'
Student actions after logging in to the Dashboard (n=753,
Feb/Mar 2017)

7. Logging into the NTU Student Dashboard
• Design and framing of Dashboard very important
• Recent, small scale research on student perceptions of the
Dashboard found that seeing the data results in positive reactions

8. Who are we asking about the Dashboard?
• STS and student interviews were offered to all students, but who
answered?
• Results should not be dismissed, but it is important lens
NB: 85 % of students who answered the STS gave us permission to link their answers to their end of
year engagement

9. Relationships between log-ins and success
• Positive correlation between the number of log-ins to the Dashboard
and student progression and attainment
• Question: Is the Dashboard serving as a tool to improve outcomes
or is logging into the Dashboard an activity successful students do?

10. What type of students are logging into the
Dashboard
• Broad usage across university with >90 % of students logging in
once or more
• But, it is the most engaged students who log-in most frequently,
and hence who are likely to benefit most

11. The role of staff in changing student outcomes
• Need to accept that the Dashboard will not inspire and motivate all
students  importance of staff as change agents
• Staff are encouraged to help those students who will not proactively
seek help themselves – this is not a trivial task
• We need more data on ‘what works?’ to change students outcomes;
if it works, how it works, when it works
• The NTU Student Dashboard has the facility to add notes about staff
and student interactions, with the aim of combining the quantitative
data with the qualitative aspect of ‘what works

12. Notes in the NTU Student Dashboard
• Free text box allows staff flexible use but creates challenges around
broad brush analysis of the impact of notes
• First steps: case studies based on notes inputted in a systematic
manner

13. Library Learning and Teaching Team
• The team offers guidance covering all aspects of academic skills.
• Students can book 30 minute one-to-one sessions with team
members between 9 am and 5 pm using an online booking system
• In 2015-16, the eight team members inputted notes into the NTU
Student Dashboard during/shortly after one-to-one sessions with
815 students
More information about the team can be found at http://www4.ntu.ac.uk/library/learning_teaching/teaching_support/index.html

14. Who is using the 1:1 library service?
• 75 % of students had one note from a member of the library
learning and teaching team. One student had 21 notes.
• 83 % of notes for UG students (PGR students not in Dashboard)

15. Who is using the 1:1 library service?
• A higher proportion of fourth year students use the 1:1 service than
other years and usage varies across Schools

16. Who is using the 1:1 library service?
• Many of the Success for All groups are using the 1:1 service
marginally more than their counterparts, with the exception of male
students

17. Who is using the 1:1 library service?
• Students with the lowest (1-120) and highest (301 +) entry tariffs
were over represented in the data for users of the 1:1 service

18. Why use the Dashboard to measure impact?
• Daily engagement ratings allow the analysis of short term impact of
a specific intervention.
• Increasing a student’s engagement with the course is a success
factor in its own right.
• The data is automatically generated, it does not require additional
work to gather

19. How can the Dashboard be used to measure
impact?
• Daily engagement data allows two types of analysis:
1. Change in engagement over timeframe
2. Rate of change in engagement over timeframe
Date of
intervention
7 days
before
14 days
before
7 days after 14 days
after
• Important context for analysing impact of intervention:
1. Behaviour before intervention
2. Behaviour of others on same course
Time

20. Impact of Library Learning and Teaching
Team on student engagement
• After the note was inputted the proportion of students with
engagement higher than the course average increased

21. Impact of library learning and teaching team
on student engagement
• After the note was inputted the proportion of students whose rate of
increase in engagement was higher than course average increased

22. Timing of the appointments with the Library
Learning and Teaching Team
• Engagement increases in the time leading up to the appointment
with the library team for both the attending students and the course
average
• Engagement drops away less quickly for students who attended an
appointment with the library team

23. Timing of the appointments with the Library
Learning and Teaching Team
• Students appear to be using library as part of the process of engaging
more with the University, rather than the library appointment being
the trigger for increased engagement

24. Progression and attainment differences for
users of the 1:1 library service
• Progression rates for students who visited the library team was 8.6 %
higher than those who didn’t
• Average GPA was 2.6 points higher for progressing students who
visited the library team than for those who didn’t

25. Attainment differences for users of the 1:1
library service
• 65.2 % of students who used the service had a GPA equivalent of a
2:1 or first compared to 54.1 % for students who didn’t
• So 11.1 % more students who got a 2:1 or first for those who had an
appointment (20.5 % higher)

26. Engagement of students before meeting the
Library Learning and Teaching Team
• Students who visited the library team were more engaged with the
University than their course average before attending their one-to-
one appointment (as well as being more engaged afterwards)
One week before appointmentTwo weeks before appointment

27. Summary
• Students had higher engagement after visiting the Library Learning
and Teaching Team
• Students who used the 1:1 library service had better outcomes in
terms of progression and attainment than those who didn’t
What can we say about students who attended a 1:1 session with
the Library Learning and Teaching Team?
What can’t we say?
• Why did these student go?
• What worked well about these appointments, and what worked less
well?
• What impact did these appointments have on how the students felt?
• Did attending these appointment improve student outcomes or is
using this type of service an activity successful students do?

28. Summary
• Only 1.6 % of male (FT, UG) students in years 1-4 attended a one-
to-one session with the Library Learning and Teaching Team
compared with 3.9 % of female students
• On the day of the session with the Library Learning and Teaching
team only 8 % of students had an engagement ratings lower than the
course average
• At NTU we have sophisticated support mechanisms, but if they rely on
students taking the first step, then they appear to only work for the
most engaged students.
What is it important to say about students who attended a 1:1
session with the Library Learning and Teaching Team?

29. Next steps – pilot studies at NTU
• Referrals to services through the NTU Student Dashboard
– Signposting to support has been in Dashboard from start
– In 2015/16 the ability to make referrals to certain services was piloted
– In 2016/17 the referrals function will be expanded
• Further case studies of staff notes
– Notes functionality within Dashboard is being adapted to enable more analysis
of the impact of staff interventions on student outcomes
– Series of case studies based on courses using notes in a systematic way
• Students receiving alerts directly
– Previously “no engagement” alerts have been sent to staff only, in 2016/17 we
will trial sending alerts to students
– Automated email associated with alert will depend on the number of alerts a
student has generated

30. Thank you for listening.
Any questions?
www.ABLEproject.eu www.stela-project.eu
@ABLEproject_eu @STELA_Project