This document discusses using programme surveys to inform educational enhancement. It addresses the purposes and challenges of programme surveys, using survey results as quality indicators, and engaging academic staff. Programme surveys provide partial representations of student experience and should be triangulated with other data sources. Survey results alone do not indicate quality but can help identify areas for improvement when discussed openly among staff, students, and administrators. Engaging all stakeholders is key to effectively utilizing programme surveys for educational enhancement.

1. Planning for
enhancement
using programme
surveys
Dr Paul Bennett
p.g.bennett@durham.ac.uk
Informed by work at the
Higher Education Academy

2. ∂
Outline
1. Purpose and challenges of programme surveys
• Purposes and types of programme survey
• Challenges of student surveys
• Challenges of‘programme-ness’
1. Driving enhancements to education
• Survey results as quality indicators
• Engaging academic staff
are the main strengths and limitations of using the NSS for informing
changes to teaching?

3. ∂
Purposes of programme surveys
NSS Review
• Providing information to prospective students
• Providing information to HE providers
(for quality assurance and enhancement)
• Ensuring public accountability
(especially informing government and funders)
• Informing current students strengths and
limitations of using the NSS for informing changes to
teaching?

4. ∂
Purposes of programme surveys
QAA UK Quality Code (B5 – Student Engagement)
• Questionnaires as one of a wide range of
mechanisms by which students can be involved in
quality systems
• Survey results themselves part of the information
base for student-staff discussions

5. ∂
Types of student surveys
• Satisfaction
• Experience
• Climate (more common at institution level)
• Engagement
• Diagnostic (more common at module level)
Questions may attempt to extract: opinions, attitudes, factual
information, perceptions, motivations, expectations,
behaviours, development, understanding…

6. ∂
Love? Hate?
Efficient Overused
Democratic Consumerist
Digestible Shallow
Comparable Misleading
Reliable Invalid
Motivating Demotivating
SURVEYS
STUDENT EXTRACT
Challenges of student surveys

7. ∂
Challenges of ‘programme-ness’
• Defining the correct population
• Long time period with unknown reference points
• Multiple experiences, modules, lecturers
• End of programme – tend to be summative
• Game playing / desirability bias
• Generic questions
• Masks variability of experience between students

8. ∂
Discussion
1. Which programme surveys do you use and for
what purposes?
2. What are the main challenges in meeting those
purposes?
3. How can we best deal with ‘programme-ness’?

9. ∂
Survey results as quality indicators
• As with any research method, surveys provide a partial
representation of …
• Do programme surveys purport to indicate ‘quality’?
Graham Gibbs (2010) Dimensions of Quality
• Measures of quality should relate to educational gain
• Measures of effective practice (student engagement,
intellectual challenge, deep learning) are good predictors of
educational gain
• Such measures are found in the National Survey of Student
Engagement (NSSE) but not in the UK’s NSS

10. ∂
Survey results as quality indicators
Implications
•Survey results should be triangulated with other sources of
information
•Don’t just mirror the NSS in internal programme surveys or
module evaluations
•Engagement surveys being trialled in UK context (HEA ‘UKES’
pilot promising, new NSS pilot in progress)
•BUT the NSS will remain a priority in University planning and
quality enhancement – engaging staff vital

11. ∂
Engaging academic staff
• Staff sceptical of survey metrics:
- Reductionist
- Data problems, aggregation, programme-ness…
- Response rates
- Question focus and language
- Top down approach with a ‘deficit’ focus –
‘a stick to beat us with’
• Why are the results as they are?

12. ∂
Engaging academic staff
• Opportunities to unpack and openly discuss results
• Focus on positive areas and sharing best practice
• Triangulation with other information and expertise
• Further research and investigation to ‘drill down’
• Staff – student engagement

13. ∂
Progress in use of the NSS for enhancement
- ‘Traffic-light’ performance monitoring
- Use in staff meetings and action-plans
- ‘You said, we did’
- Further research and investigation
- Staff-student-manager dialogue (formal/informal)
- Staff-student events
- Students as partners (e.g. in action planning)

14. ∂
Discussion
1. How do survey results inform planning for
enhancement at your institution?
2. What are the challenges in ‘drilling down’ into
results to find out what the issues are?
3. How best can academic staff be engaged in
discussion of survey results?

15. ∂
• Programme surveys are useful
research method giving a partial
representation of experience
• Recognising strengths and
limitations is key to planning
effective enhancement
• Needs to be triangulated with
other information – e.g. module
evaluation, qualitative feedback
• Importance of staff and student
engagement
Summary

16. ‘Making it Count’
- using the NSS for
enhancement
www.heacademy.ac.uk/nss
16

17. Planning for
enhancement
using programme
surveys
Dr Paul Bennett
p.g.bennett@durham.ac.uk
Informed by work at the
Higher Education Academy