TESTA workshop with three research methods

1. Getting to grips with the TESTA
methods
How do I do a
TESTA audit? What is the
AEQ?
What does the
data tell me
about the
programme?
What can I
learn from a
focus group?

2. The Audit: Caveats
1. Audit is not everything
2. Official discourse
3. Planned curriculum
4. Some better data, some weaker, some gaps

3. Activity One: mock audit
Programme
Team
Meeting
Assessment
Experience
Questionnaire
(AEQ)
TESTA
Programme
Audit
Student
Focus Groups

4. • Some context
• Number of summative
• Number of formative
• Varieties of assessment
• Proportion of exams
• Written feedback
• Speed of return of
feedback
Summary of audit data

5. TESTA definitions
Summative:
graded assessment which counts towards the degree
Formative:
Does not count: ungraded, required task with
feedback

6. Mock Audit

7. Summary of first year
TESTA audit

8. Make sense of audit data
1) What is striking about the data?
2) What surprises or puzzles you?
3) What would you want to know
more about?
Each table look at:
• 1 x overview data
OR
• 1 x discipline
OR
• 1 x university type.
• Briefly discuss in relation
to questions

9. Typical A&F patterns
73 programmes in 14 unis (Jessop and Tomas 2017)
Characteristic Low Medium High
Volume of summative
assessment
Below 33 40-48 More than 48
Volume of formative only Below 1 5-19 More than 19
% of tasks by examinations Below 11% 22-31% More than
31%
Variety of assessment
methods
Below 8 11-15 More than 15
Written feedback in words Less than
3,800
6,000-7,600 More than
7,600

10. Assessment loads: how much is too much?
(Tomas and Jessop 2018)
Summative
Exams

11. The scope of information that you
need to know for that module is
huge…so you’re having to revise
everything - at the same time, you
want to write an in-depth answer
(Student, TESTA data).
Heavy workloads lead to surface
learning
(Lizzio, Wilson and Simons 2002)

12. Characteristic Medium
(Research-
Intensive)
Medium
(Teaching
Intensive)
Tests of
difference
Summative
Formative
Proportion of
Examinations
Variety assessment
methods

13. Characteristic Medium
(Research-
Intensive)
Medium
(Teaching
Intensive)
Mann-
Whitney U
test results
Summative 41-79
(Median 50)
34-41
(Median 35)
RI*

14. Characteristic Medium
(Research-
Intensive)
Medium
(Teaching
Intensive)
Mann-
Whitney U
test results
Summative 41-79
(Median 50)
34-41
(Median 35)
RI*
Formative 1-26
(Median 3)
3-17
(Median 7)
n.s.

15. Characteristic Medium
(Research-
Intensive)
Medium
(Teaching
Intensive)
Mann-
Whitney U
test results
Summative 41-79
(Median 50)
34-41
(Median 35)
RI*
Formative 1-26
(Median 3)
3-17
(Median 7)
n.s.
Proportion of
Examinations
27-42%
(Median 30%)
5-19%
(Median 10%)
RI*

16. Characteristic Medium
(Research-
Intensive)
Medium
(Teaching
Intensive)
Mann-
Whitney U
test results
Summative 41-79
(Median 50)
34-41
(Median 35)
RI*
Formative 1-26
(Median 3)
3-17
(Median 7)
n.s.
Proportion of
Examinations
27-42%
(Median 30%)
5-19%
(Median 10%)
RI*
Variety assessment
methods
8-10
(Median 8)
12-14
(Median 15)
TI*

17. Research Tools 2:
The AEQ

18. Why a questionnaire about assessment?
• Weaker NSS scores
• Weak NSS?
• Quick and big data
• Quant/qualitative

19. AEQ 3.3 (2003)
• Designed to measure ‘conditions under which
assessment supports learning’
• Based on theory and evidence + selected CEQ scales
• Robust enough factor structure and scale coherence –
measures what it’s meant to be measuring?

20. AEQ 4.0 (2018)
• Fill in the AEQ 4.0 from the vantage point of
being a student in one of your classes
• Paper or online?
• https://educ.sphinxonline.net/v4/s/ha2fbs

21. Comparing Audit and AEQ data
from one programme
In pairs or groups, explore programme
audit and AEQ data from one programme.
Does anything stack up? Are there loose
ends, questions, contradictions?

22. Research Tools 3:
Focus Groups

23. Have a go at triangulating data
• Read through audit, AEQ and focus group data from one
programme
• Quick abstract/bullet points of what seems to be going on
• Discuss with your group/flipchart
a) What are the stand out themes?
b) What jigsaw pieces fit together?
c) What unresolved issues remain?

24. Main pointers for focus group
• Questions are broad themes
• Easy to complicated
• Sit in a circle
• It’s the discussion that matters
• Go with the flow
• But steer when off topic, direct, pass the ball
• Troubleshooting
• Ethics
How to run a focus group>resources>research toolkits
http://www.testa.ac.uk/index.php

25. Getting students to attend…
• Get the support of lecturers, programme team
• Explore using student researchers
• Use vouchers
• Food
• Between 3 and 8 students for one hour
• Ethics and confidentiality

26. Role play (5 minutes)

27. What the data looks like:
…and the intelligent transcript
texttoMP3
https://transcribe.wreally.com/

28. Coding 101
CODES
• Marker variation
• Written criteria
• Peer feedback
• Subjectivity
• Formative feedback
• Exemplars
• Multi-stage assessment
• Marking exercises
CATEGORY
Internalising standards

29. TESTA is like fly fishing