Designing useful evaluations - An online workshop for the Jisc AF programme_Inspire Research Ltd

JISC Assessment & Feedback Programme Online workshop 7 th November 20 11 Designing useful evaluations Dr Rachel A Harris

Aims of the session
Look at the evaluation cycle.
Consider project ‘logic’.
Review a nticipated outcomes & impact.
Appraise which project activities to evaluate .
Identify evaluation questions.
Review how the evaluation will be undertaken.
Find out what other projects are doing!
Consider the baseline
Other evaluation issues?

Curriculum Delivery Programme & the role of Evaluation
The JISC & Becta funded programme invited projects to “ transform how they deliver and support learning across a curriculum area through the effective use of technology ”.
Emphasis on gathering and using evidence
Identifying what works
Adding value by generating evidence
How did projects determine if delivery was ‘transformed’ and the technology was ‘effective’?
Project outputs: http:// jiscdesignstudio . pbworks .com/

Initial evaluation stages
Envision potential impact /achievements:
potential indicators of impact/achievement,
credible evidence of impact , from different stakeholders ’ perspective s .
Baseline:
first set of evaluation data
review the current context
identify existing practice
determine stakeholders’ attitudes
clarify challenges (and identify barriers)

A multitude of approaches
Action research ( Atelier-D, Duckling, KUBE )
Independent internal/external evaluator ( Cascade )
Appreciative inquiry ( ESCAPE )
Balanced scorecard ( COWL )
Formative evaluation ( MAC )
Micro & macro data sources ( Springboard TV )
Rich qualitative methods ( ISCC )

Action Research McNiff and Whitehead’s (2006) ‘ A ction-reflection cycle’ Observe Reflect Act Evaluate Modify Move in new directions Baseline Pilot Revisit Review

CIPP Evaluation (Cascade)

Assessment & Feedback Programme
The programme focuses on “ large-scale changes in assessment and feedback practice, supported by technology “ with the aim of enhancing the learning and teaching process and delivering efficiencies and quality improvements ”.
Pedagogic enhancements
Workload efficiencies
Building the business case

Designing your evaluation (I)
What do you want to happen as a result of your project’s assessment and feedback innovation ? ( Anticipated Outputs and Outcomes, & Anticipated Impact )
Who and/or what might this innovation impact on? ( Stakeholders & think about l evels)
How will you know the intended outcomes have been achieved ? Or What indicators could you use to demonstrate what ha s been achieved? ( Measures )

What do you want to know? & What indicators could you use?

Report back

Big picture evaluation questions
Process evaluation – administrative and pedagogical design , and implementation .
Outcome evaluation – value of outcomes .
Learning – barriers and enablers, surprises, causal explanations .
Overarching questions about value/worth .
Forward or outward focused evaluation questions – reusability, sustainability .

Questions ‘cheat sheet’
How well was the project designed & implemented?
How valuable were the outcomes to participants (students, lecturing staff, administrators)? To the institution, the community, the economy?
What were the barriers and enablers?
What else was learned?
Was the project worth implementing?
To what extent are the project’s content, design or implementation likely to be valuable elsewhere?

Share your evaluation questions
Some examples:
What value do students and tutors gain from using spoken e-feedback?
How has providing spoken rather than written feedback impacted on tutor workload?
What value did teachers and students gain from collaborating long-term on assessment and feedback?
What impact has providing ‘ balance of assessment ’ data had on programme teams?

Designing your evaluation (II)
How will you find out whether the intended outcomes have been achieved ? (What, How and When evidenced)
How will you use this information?
Who else might this information be of use to?
What other questions should be considered?

Existing sources
What existing sources of evidence are you planning to use?
Previous course evaluations
Departmental timesheets
Data from previous studies
Centrally held institutional data
NSS data
No plans to use existing data
Others?

How will you find out?

The Baseline
Where are you starting from?
Qualitative versus quantitative evidence
Hitting a moving target
Previous projects’ baselines: http:// jiscdesignstudio . pbworks .com/w/page/46422956/Example%20baseline%20reports

What happens next?
Reflect on your thinking from today.
Review the feedback on your draft plans.
Consider how you might want to refine your evaluation plans.
Use your evaluation questions to inform baseline activities.
Programme mapping!

Any questions?

Programme Evaluation Resources https://programmesupport.pbworks.com/w/page/42511148/Evaluation%20Resources Dr Rachel A Harris twitter : raharris email: rachel @inspire-research.co. uk web: www.inspire-research.co.uk

http://www.scoop.it	196
http://blog.inspire-research.co.uk	136
http://webcache.googleusercontent.com	1

Designing useful evaluations - An online workshop for the Jisc AF programme_Inspire Research Ltd

by Rachel Harris on Nov 09, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 333

Statistics

Designing useful evaluations - An online workshop for the Jisc AF programme_Inspire Research Ltd — Presentation Transcript