Using thematic analysis of student free text to enhance assessment and feedback strategies

1. ‘Translating’ the NSS: Using thematic analysis
of student free text to enhance assessment
and feedback strategies
Dr Deborah Rafalin
City, University of London
Chair of Academic Misconduct, SASS
Learning Development Fellow, LEaD
Senior Lecturer in Psychology
SEDA Spring Teaching, Learning and Assessment
Conference 2019

2. Today
 Present a method of Thematic Analysis that can be used to
explore themes in free text qualitative data eg the NSS free
text, SSLC minutes, external examiners reports, focus groups
 Explore ideas about how emergent themes from analysis of
free text student feedback might lead to ideas about
assessment and feedback development and specific strategies
 Share ideas, challenges and good practice
All presented data has been anonymised

3. UG Assessment and Feedback Project
 Across all schools of the University
 SASS
 Law
 Cass
 Health
 SMCSE
 Several identified programmes within each school
 Collaboratively developing individual projects focussing on
assessment and feedback with each programme team
 Aim to improve student progression and enhance student
satisfaction through development of assessment and feedback
 TEF and NSS
Dr Deborah Rafalin

4. Theme NSS Question
The teaching on my course 1. Staff are good at explaining things
2. Staff have made the subject interesting
3. The course is intellectually stimulating
4. My course has challenged me to achieve
my best work
Assessment and feedback
8. The criteria used in marking have been
clear in advance
9. Marking and assessment has been fair
10. Feedback on my work has been timely
11. I have received helpful comments on my
work
Academic Support
12. I have been able to contact staff when I
needed to
13. I have received sufficient advice and
guidance in relation to my course
14. Good advice was available when I
needed to make study choices on my course
TEF related NSS questions

5. Challenges
 Large numbers and resource challenges
 Diverse student population
 Educationally
 Motivationally
 Socio-economically
 Challenges with student engagement
 Diverse teams – different strengths and challenges
 Scrutiny and expectation
NSS scores indicate some student dissatisfaction with
Assessment and Feedback but what do these scores actually
mean?
Dr Deborah Rafalin

6. Thematic Analysis (Braun and Clarke, 2006)*
• A simple qualitative research method which pinpoints patterns/themes in
data to illuminate an area under investigation.
• ‘A theme captures something important about the data in relation to the
research question, and represents some level of patterned response or
meaning within the data set’ (Braun and Clarke; 2006, p11)
• There may be a number of instances of the theme across the data set, but
more instances do not necessarily mean the theme itself is more crucial
• The ‘keyness’ of a theme is not necessarily dependent on quantifiable
measures – but whether it captures something important in relation to the
area under investigation
• Not the experiences of every student, just particular students. However,
overlapping comments could suggest that the issue is more widespread
and worthy of highlighting to help identify where practice could be improved
• Not generalizable – but illuminating and possibly transferable
*(Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101)

7. The Six Phases of Thematic Analysis
(Braun and Clarke, 2006)
• Phase 1: familiarising yourself with your data
• Phase 2: generating initial codes
• Phase 3: searching for themes
• Phase 4: reviewing themes
• Phase 5: defining and naming themes
• Phase 6: producing the report

8. Phase One: familiarising yourself with the data

9. The Six phases of Thematic Analysis
(Braun and Clarke, 2006)
• Phase 1: familiarising yourself with your data
• Phase 2: generating initial codes
• The process of coding is the organising of data into meaningful groups
• Work systematically through the entire data set, giving full and equal attention
to each data item, and identify interesting aspects in the data items that may
form the basis of repeated patterns (themes) across the data set
• All actual data extracts are coded, and then collated together within each code
• Remember that you can code individual extracts of data in as many different
‘themes’ as they fit into - so an extract may be uncoded, coded once, or coded
many times, as relevant
• Note that no data set is without contradiction, so tensions and inconsistencies
can exist
• Phase 3: searching for themes
• Sort the different codes into potential themes.
• Essentially, you are starting to analyse your codes, and consider how different
codes may combine to form an overarching theme.
• Phase 4: reviewing themes
• Phase 5: defining and naming themes
• Phase 6: producing the report

10. Phase Two: generating initial codes

11. The Six phases of Thematic Analysis
(Braun and Clarke, 2006)
• Phase 1: familiarising yourself with your data
• Phase 2: generating initial codes
• The process of coding is the organising of data into meaningful groups
• Work systematically through the entire data set, giving full and equal attention
to each data item, and identify interesting aspects in the data items that may
form the basis of repeated patterns (themes) across the data set
• All actual data extracts are coded, and then collated together within each code
• Remember that you can code individual extracts of data in as many different
‘themes’ as they fit into - so an extract may be uncoded, coded once, or coded
many times, as relevant
• Note that no data set is without contradiction, so tensions and inconsistencies
can exist
• Phase 3: searching for themes
• Sort the different codes into potential themes.
• Essentially, you are starting to analyse your codes, and consider how different
codes may combine to form an overarching theme.
• Phase 4: reviewing themes
• Phase 5: defining and naming themes
• Phase 6: producing the report

12. Phase Three: Emergent Themes and Sub-themes
• Exam Guidance
• Sub-theme I: More specificity needed?
• Sub-theme II: Not aligned with what is actually in the exam
• Alignment between Exam and Teaching
• What are the marking criteria?
• Feedback
• Sub-theme 1: Quantity
• Sub-theme II: Quality
• Sub-theme III: For Aspiration – how can I do better?
• Formative Assessment
• Breadth of Assessment Type
• Issues of Trust
• Sub-theme I: Inconsistency
• Sub-theme II: Fairness

13. What initial
codes/themes can
be seen within the
given sample of
programme data?
Ideas exchange:
Thoughts and
questions?
Can these emergent
codes/themes suggest
areas of assessment and
feedback strategy that
might benefit from
development?

14. Thank you