Working with students to create authentic and timely course evaluation of teaching excellence – a case study using a structured qualitative approach

1. Working with students to create authentic and
timely course evaluation of teaching excellence
– a case study using a structured qualitative
approach
Dr Dawn Morley, Lecturer in Higher Education, University of Surrey
d.morley@surrey.ac.uk
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
1

2. Evaluating the Magnolia Homestay, Hoi An
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
2

3. 5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
3

4. What were the criteria of this evaluation?
1. It focused on the service users’ experience
2. It provide timely evaluation that monitored the quality of the experience
3. The quantitative measurement was useful for an overall quality
measurement
4. The qualitative comments provided personalised and deep evaluation.
This was based on uncategorised and authentic opinion led by the
service user.
5. Evaluation comments are approved before they are available on the trip
advisor site. This process seemed to be quick – on average 2-5 days.
6. We enjoyed adding our own evaluation! It was an opportunity for us to
reflect on and discuss our experience.
7. The providers had a “right to reply”
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
4

5. Discussion activity
• What criteria should an
evaluation of teaching and
learning in HE include?
• Do the Magnolia Homestay
criteria apply to HE?
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
5

6. What were the criteria of this evaluation?
1. It focused on the service users’ experience
2. It provide timely evaluation that monitored the quality of the experience
3. The quantitative measurement was useful for an overall quality
measurement
4. The qualitative comments provided personalised and deep evaluation.
This was based on uncategorised and authentic opinion led by the
service user.
5. Evaluation comments are approved before they are available on the trip
advisor site. This process seemed to be quick – on average 2-5 days.
6. We enjoyed adding our own evaluation! It was an opportunity for us to
reflect on and discuss our experience.
7. The providers had a “right to reply”
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
6

7. Why evaluate this course?
• New curriculum that uses flexible pedagogies for example,
case based learning.
• Students entering a new role and profession alongside
established traditional health care professions.
• 21 students in the first cohort.
• Building functional links between the Department of HE and
the faculty in the third space (Whitchurch 2008) where DHE
staff act as external researchers and then academic
developers.
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
7

8. The Delphi evaluation process (1 hour)
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
8
7
7
7

9. The report writing
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017 9
Coded report written in
three colours of the group
responses
DHE facilitators check
validity of their own groups
14th February
Coding removed and report
finalised
17th February
Preview by course leader Permission to distribute to
students
19th February
Report downloaded to
google docs
Students contacted to add
their comments
anonymously.
22nd- 1st March
Validated final report
returned to course leader
1st March
16 days in total

10. Did the data have validity and resonance?
“you have definitely captured the perspectives of my group … in terms
of process, I actually think it worked really well. I would have welcomed
the opportunity to digitally record the session so I wasn’t scribbling so
much but I understand this isn’t a focus group as such”
(DHE group facilitator, 16/02/17)
“the 2 way iterative dialogue as a means of developing the programme
is an excellent design and has resulted in some really in-depth and
interesting evaluative comments from the students”
(DHE report reviewer, 07/03/17)
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
10

11. Key findings from students and faculty
• The strengths and weaknesses identified a range of teaching and learning
dilemmas (eg management of e portfolio with practice partners,
consistency of academic level between external lecturers, anxiety of
national exam). Known issues were made more explicit to faculty and
innovative solutions were able to be considered eg students articulating
their new role to placement partners.
• The opportunities and threats offered a broader picture of students’
wider anxieties of the course (eg funding).
• A monthly informal meeting between students and faculty has been put in
place with increased faculty ownership for evaluation (Richardson 2005).
• The report was seen as measured, clear and with a high degree of
representativeness and neutrality by faculty.
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
11

12. Areas for future clarification
• Discussion how development points are identified and actioned between
faculty and DHE staff. The next phase will assess the implementation of the
first and how the feedback loop has been completed for the students
(Watson 2003).
• Evaluation could be developed as part of an official reporting mechanism
with key findings summarised with an overall conclusion.
• Opportunities and threats presents an opportunity to begin to engage
students in professional discussion about the wider context of their future
work and leaning gain.
“Evaluation can be reframed as learning using a reflective approach that
benefits both learners and teachers” (Ryan 2015, p.1156)
• The Delphi evaluation technique could be best placed to evaluate the first
year of new curricula5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
12

13. References
Richardson, J.T.E., 2005. Instruments for obtaining student feedback: a
review of the literature. Assessment and Evaluation in Higher
Education, 30 (4), 387-415.
Ryan, M., 2015. Framing student evaluations of university learning and
teaching: discursive strategies and textual outcomes. Assessment and
Evaluation in Higher Education, 40 (8), 1142-1158.
Watson, S., 2003. Closing the feedback loop: Ensuring effective action
from student feedback. Tertiary Education and Management, 9 (2),
145-157.
Whitchurch, C., 2008. Shifting Identities and Blurring Boundaries: the
Emergence of Third Space Professionals in UK Higher Education. Higher
Education Quarterly, 62 (4), 377-396.
5/10/2017
SEDA Spring teaching and learning and assessment conference
2017
13