Patrick Raven and Matt Wingfield - Using Controlled Digital Frameworks to Assess Practical Synoptic Tasks | VocTEL 2015

1. PATRICK CRAVEN – DIRECTOR ASSESSMENT POLICY & RESEARCH
VocTEL 2015
Using Controlled Digital Frameworks
to Assess Practical Synoptic Tasks
Presentation – 17th June
MATT WINGFIELD – CHIEF BUSINESS DEVELOPMENT OFFICER

3. ASSESSMENT: AN INEXACT SCIENCE
RELIABILITYTrust & Regulation Agenda Rigour and Quality Assurance
VALIDITY
Fit for Purpose and Authentic
Industry &
Assessment
Agenda
VIABILITY
Robust and Sustainable
Service &
Business
Agenda

4. THE SCOPE FOR INNOVATION
Convergent
Assessment
Divergent
Assessment
Multiple Choice,
Hot-Spot,
Drag & Drop,
Interactive ...
OBJECTIVE SUBJECTIVE
ASSESSMENT TYPES
E-ASSESSMENTCOMPLEXITY
Single Word/Value
Response
Explanatory
Phrase/Sentence
Formulae
Explanatory
Paragraph
Technical
Factual
Essays
Creative
Essays
Skills/Functions
Tests
Technical
Performance
Creative
Professional
Performance
AUTOMATED ASSESSMENT
HUMAN MARKING

5. THE EVOLUTION OF E-ASSESSMENT
• E-assessment is no longer just on-screen MCQ:
• Adaptive Testing
• Simulations
• Evidence-based assessments
• All are robust, scalable, proven and auditable
• There are great examples out there that...
• Support self-reflective learning
• Evidence the learning process
• Assess the application of knowledge & understanding

6. THE WISDOM OF ROUTINE

7. THE THEORY OF EVOLUTION
In the struggle for survival, the
fittest win out at the expense
of their rivals because they
succeed in adapting themselves
best to their environment.
(Charles Darwin)

8. 8
WHAT NEXT …..?

9. • A leading provider of evidence-based assessment tools to
support personalised assessment for learning;
• Used by institutions around the world to track and assess
learner progression and to provide feedback;
• Used by professional/accreditation bodies since 2004 to
support formal assessment of knowledge & skill application.
DIGITAL ASSESS

10. SO, WHAT DOES THIS LOOK LIKE?

11. STRUCTURED SYNOPTIC ASSESSMENT

12. ILLUSTRATE UNDERSTANDING OF SKILLS

13. CAPTURE PLANNING & PREPARATION

14. CAPTURE PERFORMANCE

15. CAPTURE REVIEW & ANALYSIS

16. CAPTURE THE LEARNER JOURNEY

17. • People are good at:
• Relative Judgements;
• Ranking work.
• People are less good at:
• Absolute Judgements, e.g. Scoring.
• In addition, when assessing student work we:
• are influenced by past experience;
• work independently – magnifying the effect of
subjectivity.
THE ASSESSMENT CHALLENGE

18. Comparative Judgement:
Perhaps we should take a different approach?
• Measuring without subjectivity –
adaptive scaling without needing
scores;
• Presents a pair of items and asks
the observer to make a value
judgement – which is ‘better’?
• Repeat this through all possible
combinations, and record the
results to create a rank order;
• The Law of Comparative
Judgement (Thurstone - 1927).
THE ASSESSMENT CHALLENGE

19. We have created a web-based ADAPTIVE comparative judgement solution for
assessing student work...presents successive pairs of student work to
assessors and asks them to say which is best at meeting assessment criteria:
ACJ – Adaptive Comparative Judgement:
HOW IT WORKS

20. • The system dynamically organises the work into a scaled rank
order as the students complete their judgements.
• Which can produce a grade/score if required.
HOW IT WORKS

21. • The benefits of structured synoptic e-assessment:
• Enables quicker work-readiness through use of authentic tasks;
• Provides a legitimate place for technology in assessment;
• Facilitates richer, stronger formative support of learning
through improved visibility of process;
• Provides ‘forward indicators of readiness’:
 The right skills;
 The right attitude;
 In the right time;
• Provides greater transparency:
• extending learning to address weaknesses.
• of collaboration and teamwork
• of progress, decision making, critique and explanation
WHY DOES THIS MATTER?

22. The eAssessment Assocation
Join the eAA for FREE @ www.e-assessment.com

23. THANK YOU FOR LISTENING
Patrick Craven
Patrick.Craven@cityandguilds.com
www.cityandguilds.com
Matt Wingfield
Matt.Wingfield@digitalassess.com
www.digitalassess.com