The national evaluation of the CETL programme 2005-2010: first formative phase CSET in the Department of Educational Research Lancaster University, The Institute of Educational Technology The Open University

The national evaluation of the CETL programme 2005-2010: first formative phase

The theory of change embedded in the CETL programme The following propositions flow from an embedded theory of change that is based on the guiding principles of reward and recognition Excellent teaching produces excellent learning Recognising individual and institutional excellence in teaching and learning promotes excellence across the sector The relatively ‘light’ steer on specific designs for excellence These propositions form the central core of concerns for the evaluation

The following propositions flow from an embedded theory of change that is based on the guiding principles of reward and recognition

Excellent teaching produces excellent learning

Recognising individual and institutional excellence in teaching and learning promotes excellence across the sector

The relatively ‘light’ steer on specific designs for excellence

These propositions form the central core of concerns for the evaluation

Overall understanding of the CETL strategy The CETL programme is a complex national ‘policy instrument’ . It incorporates a diverse range of ‘mechanisms’ . They are embodied in the specific designs of individual CETL activity. They are intended to bring about: Enhanced learning outcomes, Change in cultures and practices of teaching and learning that constitute inspirational embodiments, New institutional systems and approaches, Dissemination of clusters or beacons of good practice, Collaborative mechanisms, The use and development of pedagogic research, Strategic coherence in institutional developments.

The CETL programme is a complex national ‘policy instrument’ .

It incorporates a diverse range of ‘mechanisms’ .

They are embodied in the specific designs of individual CETL activity.

They are intended to bring about:

Enhanced learning outcomes,

Change in cultures and practices of teaching and learning that constitute inspirational embodiments,

New institutional systems and approaches,

Dissemination of clusters or beacons of good practice,

Collaborative mechanisms,

The use and development of pedagogic research,

Strategic coherence in institutional developments.

Implications for the evaluation We will focus on enclave effects, bridgeheading strategies, the development of new embedded practices and systemic change This is a formative, utilization-focused process. We are interested in bridging tools that are used. We offer a strategic commentary on provisional stabilities. This involves clarifying foci, exploring theories of change, plenty of interim reporting and discussing strategic implications for the future. We look for unanticipated or unintended effects. We also look for multiplier effects in resource procurement for T&L developments (e.g. matched funding). We will explore the effects of HEFCE’s capital investment in CETLs. We identify promising and effective practices -- ‘appreciative inquiry’. We are interested in ‘connective tissue..

We will focus on enclave effects, bridgeheading strategies, the development of new embedded practices and systemic change

This is a formative, utilization-focused process.

We are interested in bridging tools that are used.

We offer a strategic commentary on provisional stabilities. This involves clarifying foci, exploring theories of change, plenty of interim reporting and discussing strategic implications for the future.

We look for unanticipated or unintended effects.

We also look for multiplier effects in resource procurement for T&L developments (e.g. matched funding).

We will explore the effects of HEFCE’s capital investment in CETLs.

We identify promising and effective practices -- ‘appreciative inquiry’.

We are interested in ‘connective tissue..

Data sources Key informant interviews Case study visits (including all Northern Ireland CETLs) Survey of CETL directors’ with follow-up interviews Environmental scanning Web-based array of visual data with evocative vignettes and exemplars Local evaluators’ outputs which we will collaboratively collate

Key informant interviews

Case study visits (including all Northern Ireland CETLs)

Survey of CETL directors’ with follow-up interviews

Environmental scanning

Web-based array of visual data with evocative vignettes and exemplars

Local evaluators’ outputs which we will collaboratively collate

Working with the local evaluators Challenges: Variation and diversity, competing commissioning loyalties, administrative burden, unstable data and the problem of validity. Resources.: Empathy, helped by experience of previous systems and approaches. Expertise in evaluation, HE, pedagogy and e-learning Approach: Participative design, agreed reporting themes, meta–review matrix

Challenges:

Variation and diversity, competing commissioning loyalties, administrative burden, unstable data and the problem of validity.

Resources.:

Empathy, helped by experience of previous systems and approaches.

Expertise in evaluation, HE, pedagogy and e-learning

Approach:

Participative design, agreed reporting themes, meta–review matrix

Overall challenges Creating valid indicators of the embedded theory of change Working with multiple and diverse CETL designs Enabling the effective use of local evaluations Doing so much in so short a time

Creating valid indicators of the embedded theory of change

Working with multiple and diverse CETL designs

Enabling the effective use of local evaluations

Doing so much in so short a time

The Evaluation Team World-class in educational, programme and policy evaluation Track record of evaluative research of the change process Research-based expertise in policy support and development in Higher Education A body of published work on Higher Education policy, change, teaching and learning Professor Paul Trowler Lancaster University Professor Murray Saunders Lancaster University Dr Paul Ashwin Lancaster University Dr Sadie Williams Lancaster University Professor Peter Knight The OU Dr Joan Machell Lancaster University