1. Module Evaluations, Institutional
Responses: Using Data to Focus Improvements in
the Student Experience
Dr. Peter Bird Learning Technologies
Dr. Rachel Forsyth Centre for Learning and
Teaching
Newcastle | Feb 12th 2015

2. Session Format
• Institutional Context
• The EQAL transformation program
• Use of course and module data to measure
impact
• Group Exercise to develop further impact
measures
• Plenary Discussion
• Key Lessons

3. Context | An interesting place to work
• Large, popular, multi-campus university
– 36,000+ students | 1,000+ courses | consolidating from 7 sites
• Pursuing an ambitious transformation initiative
– New Buildings, New Curriculum, New Admin Processes & Systems,
New Learning & Mobile Tech, New Quality Processes…

4. Problems impacting students and
staff
• Lack of standardization of course units
• Course units with too many assessments
• Timetabling issues
• Quality of Feedback
• Disconnect between library resources and course
needs
• Robust but complex effort intensive QA
processes

5. Coordinating our response
Improve satisfaction, retention & success
Great
Online Experience
(Seamless, personalised)
Great
Learning Spaces
Great
Teaching
Simplify
Unwieldy
Curriculum
£350M

6. • A coordinated strike for step-change improvement
EQAL
New Curriculum
• designing new units, …
New Admin Systems & Processes
• personal timetabling, …
New Virtual Learning Environment
• Moodle & myMMU web/mobile, Talis Aspire…
New QA & QE Processes
• facilitating curriculum transformation
In the current climate
Diminishing unit of resource
Everything depends on everything else
We are large and risk averse

7. MISSION:

8. 868
L3+L4
modules

9. 9
Wrapping the institution around the learner
Student ID
Timetable
This work is licensed under a Attribution-
NonCommercial-ShareAlike 2.0 licence
Sync to
personal
device

10. Summary | we cleared a
path to the information our
learners needed…

11. Engaging our
students
Providing
business
intelligence
Automating
our
processes
Are we in a better place now?

12. … we can now begin to
interpret some digital
footprints

13. Using the data
2. Identify
relevant data
sources
3.Summarise
individual
data sources
4 Join on
common
identifiers
5 Prep the
data for
analysis
6.Analyse &
visualise
1.Appreciate
the issue
Refining
understanding of a
problem space

14. Understanding student satisfaction
2. Identify
relevant data
sources
3.Summarise
individual
data sources
4 Join on
common
identifiers
5 Prep the
data for
analysis
6.Analyse &
visualise
1.Appreciate
the issue
Refining
understanding of
student satisfaction
Student
demographics
Entry profiles
Internal survey
responses
Join on
student ID
Random
Forests
analysis
Satisfaction
predictors
↗ Confidence
↘ Organisation
→Teaching
Sum entry
points &
qual type
Handle missing
values, collapse
categories…

15. Internal student survey
2011/12

16. Things students found useful…
“…At the beginning we talked through the assessment criteria in
tutorials - decoding it. This was very helpful! I now know what is
expected of me…” (MMU Internal Survey. Spring, best things, 2012).
“Clear teaching and materials are given out at the start of the unit and
assignment available early on so can be started on quickly.” (MMU Internal
Survey. Spring, best things, 2012).

17. Good reasons for clear briefs…
“[Tutor name] has been off and we have been given a substitute lecturer who
although is very keen and clearly very intelligent …no body on the course had
no idea about the … project so we have had no help at all since week 1 and
subsequently our marks could be low - very disappointing since we pay for this!”
Things to improve, Winter 2011
“Again there was much inconsistency with marking between the three tutors
teaching this course and while I had an excellent tutor in *Teacher name A*
friends who were in other groups (with *Teacher name B* or *Teacher name C*)
were being given entirely different instructions for the same assignments and
even appeared to be being marked to different criteria.” Things to improve,
Spring 2012

18. Suggestions for improvement
“Set of notes provided plus slides uploaded onto moodle. Clear explanation
of what is expected of us for each assignment and assessment and from
what we will have an exam at the end.” Winter, improve 2011
“…Last term I went to one of them with my work so far and they said it was
fine and meeting the assessment criteria. However I only got 58%. how is
this mark fine?! and why didn’t they guide me in the right direction when I
went for help?...” Spring, improve 2012

19. Time to look at some data….

20. Are we in a better place now?

21. Getting better but further to go…

22. Change management lessons
1. Needs Hands-on management to unblock the big issues
2. Communication/Coordination between linked sub-projects vital
3. Participative inclusive approach
4. Make ‘big asks’ as small as possible
5. Set-up strong links into faculties
6. Use data responsibly not as a big stick
7. Work with staff in workshops to interpret data, identify the root
problem(s) and jointly develop improvement plans.
http://tinyurl.com/jiscmmusrcCurriculum Design
Assessment & Feedback http://tinyurl.com/jiscmmutraffic