A formative evaluation of the CETL programme Peter Knight and Julie Morrison, Institute of Educational Technology, The Open University. On behalf of the full Lancaster-OU formative evaluation team http://kn.open.ac.uk/public/index.cfm?wpid=7076

Peter Knight and Julie Morrison, Institute of Educational Technology, The Open University.

On behalf of the full Lancaster-OU formative evaluation team

http://kn.open.ac.uk/public/index.cfm?wpid=7076

Our evaluation approach Exploring theories and practices of change embedded in the CETL programme We want our work to be evaluation research Utilization-focused We want our evaluation research to be useful. Appreciative We want to tell stories about what works well.

Exploring theories and practices of change embedded in the CETL programme

We want our work to be evaluation research

Utilization-focused

We want our evaluation research to be useful.

Appreciative

We want to tell stories about what works well.

CETL programme theory of change CETL programme has an embedded theory of change that is based on the guiding principles of reward and recognition. It also assumes that: Excellent teaching produces excellent learning Recognising individual and institutional excellence in teaching and learning promotes excellence across the sector The relatively ‘light’ steer on specific designs for excellence These propositions shape our evaluation

CETL programme has an embedded theory of change that is based on the guiding principles of reward and recognition.

It also assumes that:

Excellent teaching produces excellent learning

Recognising individual and institutional excellence in teaching and learning promotes excellence across the sector

The relatively ‘light’ steer on specific designs for excellence

These propositions shape our evaluation

The broad CETL strategy The CETL programme is a complex national policy instrument, with a diverse range of mechanisms, to bring about: Enhanced learning outcomes; Change in cultures and practices of teaching and learning that constitute inspirational embodiments; New institutional systems and approaches; Dissemination of clusters or beacons of good practice; Collaborative mechanisms; The use and development of pedagogic research; Strategic coherence in institutional developments.

The CETL programme is a complex national policy instrument, with a diverse range of mechanisms, to bring about:

Enhanced learning outcomes;

Change in cultures and practices of teaching and learning that constitute inspirational embodiments;

New institutional systems and approaches;

Dissemination of clusters or beacons of good practice;

Collaborative mechanisms;

The use and development of pedagogic research;

Strategic coherence in institutional developments.

Evaluation implications This is a formative, utilization-focused process. We identify promising and effective practices ― ‘appreciative inquiry’. We are interested in the tools used to build bridges from ‘excellence’ to ‘competence’ and thence to ‘new excellence’. We clarify foci, explore theories of change and discuss strategic implications for the future. We look for unanticipated or unintended effects. We also look for multiplier effects (e.g. matched funding). We explore the effects of HEFCE’s capital investment in CETLs. We are interested in connective tissue.

This is a formative, utilization-focused process.

We identify promising and effective practices ― ‘appreciative inquiry’.

We are interested in the tools used to build bridges from ‘excellence’ to ‘competence’ and thence to ‘new excellence’.

We clarify foci, explore theories of change and discuss strategic implications for the future.

We look for unanticipated or unintended effects.

We also look for multiplier effects (e.g. matched funding).

We explore the effects of HEFCE’s capital investment in CETLs.

We are interested in connective tissue.

Data sources Key informant interviews Case study visits (including all Northern Ireland CETLs) Survey of CETL directors’ with follow-up interviews Environmental scanning Web-based array of visual data with evocative vignettes and exemplars Local evaluators’ outputs which we will collaboratively collate

Key informant interviews

Case study visits (including all Northern Ireland CETLs)

Survey of CETL directors’ with follow-up interviews

Environmental scanning

Web-based array of visual data with evocative vignettes and exemplars

Local evaluators’ outputs which we will collaboratively collate

Working with the local evaluators Challenges: Variation and diversity, competing commissioning loyalties, administrative burden, unstable data and the problem of validity. Resources: Empathy, helped by experience of previous systems and approaches. Expertise in evaluation, HE, pedagogy and e-learning Approach: Collegial

Challenges:

Variation and diversity, competing commissioning loyalties, administrative burden, unstable data and the problem of validity.

Resources:

Empathy, helped by experience of previous systems and approaches.

Expertise in evaluation, HE, pedagogy and e-learning

Approach:

Collegial

Possible report structure 1.0 Executive Summary: One page maximum 2.0 Introduction: 2.1 Structure and audience; 2.2 Connection with other reports; 2.3 Purpose; 2.4 Overview of the evidential base 3.0 Brief description of the aims and scope of the CETL: 3.1 Purposes; 3.2 Specific goals; 3.3 Activities; 3.4 Roles and participants 4.0 Evaluation framework and approach: 4.1 RUFDATA or equivalent statement about overall approach; 4.2 Reflections on the evaluation process 5.0 Findings addressing key evaluation foci from RUFDATA or equivalent: 5.1 Student experience; 5.2 Connections with external partners; 5.4 Internal strategic impact; 5.5 Effects on teachers; 5.6 Effects on learning designs 6.0 Lessons learned and future adjustments: 6.1 Overview of new knowledge resources from students; 6.2 Emerging teaching practices; 6.3 Implications for University systems and practices; 6.4 Any sector wide multiplier effects; 6.5 Adjustments and future plans; 6.6 Reflections on the idea of a CETL as a change strategy Figures, appendices, data sources and references

1.0 Executive Summary: One page maximum

2.0 Introduction: 2.1 Structure and audience; 2.2 Connection with other reports;

2.3 Purpose; 2.4 Overview of the evidential base

3.0 Brief description of the aims and scope of the CETL: 3.1 Purposes; 3.2 Specific goals; 3.3 Activities; 3.4 Roles and participants

4.0 Evaluation framework and approach: 4.1 RUFDATA or equivalent statement about overall approach; 4.2 Reflections on the evaluation process

5.0 Findings addressing key evaluation foci from RUFDATA or equivalent: 5.1 Student experience; 5.2 Connections with external partners; 5.4 Internal strategic impact; 5.5 Effects on teachers; 5.6 Effects on learning designs

6.0 Lessons learned and future adjustments: 6.1 Overview of new knowledge resources from students; 6.2 Emerging teaching practices; 6.3 Implications for University systems and practices; 6.4 Any sector wide multiplier effects; 6.5 Adjustments and future plans; 6.6 Reflections on the idea of a CETL as a change strategy

Figures, appendices, data sources and references

Overall challenges Staying creative Insisting that we are taking a systems perspectives Making the evaluation participative Making it safe for colleagues to share ideas and experiences offers Creating open discussion about programme evaluation Working appreciatively with diversity and creativity Supporting and benefiting from local evaluations Doing so much in so short a time Exploring the CETL programme’s embedded theory of change

Staying creative

Insisting that we are taking a systems perspectives

Making the evaluation participative

Making it safe for colleagues to share ideas and experiences offers

Creating open discussion about programme evaluation

Working appreciatively with diversity and creativity

Supporting and benefiting from local evaluations

Doing so much in so short a time

Exploring the CETL programme’s embedded theory of change

Please contact us [email_address] [email_address] http://web.mac.com/knightpeter/iWeb/Site/Welcome.html http://iet.open.ac.uk/pp/peter.knight/research.cfm

[email_address]

[email_address]

http://web.mac.com/knightpeter/iWeb/Site/Welcome.html

http://iet.open.ac.uk/pp/peter.knight/research.cfm

The academic team World-class in educational, programme and policy evaluation Track record of evaluative research of the change process Research-based expertise in policy support and development in Higher Education A body of published work on Higher Education policy, change, teaching and learning Professor Paul Trowler Lancaster University Professor Murray Saunders Lancaster University Dr Paul Ashwin Lancaster University Dr Sadie Williams Lancaster University Professor Peter Knight The OU Dr Joan Machell Lancaster University