Overview of CETL programme evaluation


Published on

Outlines the formative evaluation of the Centres for Excellence in Teaching and Learning programme

Published in: Economy & Finance, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Overview of CETL programme evaluation

  1. 1. A formative evaluation of the CETL programme <ul><li>Peter Knight and Julie Morrison, Institute of Educational Technology, The Open University. </li></ul><ul><li>On behalf of the full Lancaster-OU formative evaluation team </li></ul><ul><li>http://kn.open.ac.uk/public/index.cfm?wpid=7076 </li></ul>
  2. 3. Our evaluation approach <ul><li>Exploring theories and practices of change embedded in the CETL programme </li></ul><ul><ul><li>We want our work to be evaluation research </li></ul></ul><ul><li>Utilization-focused </li></ul><ul><ul><li>We want our evaluation research to be useful. </li></ul></ul><ul><li>Appreciative </li></ul><ul><ul><li>We want to tell stories about what works well. </li></ul></ul>
  3. 4. CETL programme theory of change <ul><li>CETL programme has an embedded theory of change that is based on the guiding principles of reward and recognition. </li></ul><ul><li>It also assumes that: </li></ul><ul><ul><li>Excellent teaching produces excellent learning </li></ul></ul><ul><ul><li>Recognising individual and institutional excellence in teaching and learning promotes excellence across the sector </li></ul></ul><ul><ul><li>The relatively ‘light’ steer on specific designs for excellence </li></ul></ul><ul><li>These propositions shape our evaluation </li></ul>
  4. 5. The broad CETL strategy <ul><li>The CETL programme is a complex national policy instrument, with a diverse range of mechanisms, to bring about: </li></ul><ul><ul><li>Enhanced learning outcomes; </li></ul></ul><ul><ul><li>Change in cultures and practices of teaching and learning that constitute inspirational embodiments; </li></ul></ul><ul><ul><li>New institutional systems and approaches; </li></ul></ul><ul><ul><li>Dissemination of clusters or beacons of good practice; </li></ul></ul><ul><ul><li>Collaborative mechanisms; </li></ul></ul><ul><ul><li>The use and development of pedagogic research; </li></ul></ul><ul><ul><li>Strategic coherence in institutional developments. </li></ul></ul>
  5. 6. Evaluation implications <ul><li>This is a formative, utilization-focused process. </li></ul><ul><li>We identify promising and effective practices ― ‘appreciative inquiry’. </li></ul><ul><li>We are interested in the tools used to build bridges from ‘excellence’ to ‘competence’ and thence to ‘new excellence’. </li></ul><ul><li>We clarify foci, explore theories of change and discuss strategic implications for the future. </li></ul><ul><li>We look for unanticipated or unintended effects. </li></ul><ul><li>We also look for multiplier effects (e.g. matched funding). </li></ul><ul><li>We explore the effects of HEFCE’s capital investment in CETLs. </li></ul><ul><li>We are interested in connective tissue. </li></ul>
  6. 8. Data sources <ul><li>Key informant interviews </li></ul><ul><li>Case study visits (including all Northern Ireland CETLs) </li></ul><ul><li>Survey of CETL directors’ with follow-up interviews </li></ul><ul><li>Environmental scanning </li></ul><ul><li>Web-based array of visual data with evocative vignettes and exemplars </li></ul><ul><li>Local evaluators’ outputs which we will collaboratively collate </li></ul>
  7. 9. Working with the local evaluators <ul><li>Challenges: </li></ul><ul><ul><li>Variation and diversity, competing commissioning loyalties, administrative burden, unstable data and the problem of validity. </li></ul></ul><ul><li>Resources: </li></ul><ul><ul><li>Empathy, helped by experience of previous systems and approaches. </li></ul></ul><ul><ul><li>Expertise in evaluation, HE, pedagogy and e-learning </li></ul></ul><ul><li>Approach: </li></ul><ul><ul><li>Collegial </li></ul></ul>
  8. 10. Possible report structure <ul><li>1.0 Executive Summary: One page maximum </li></ul><ul><li>2.0 Introduction: 2.1 Structure and audience; 2.2 Connection with other reports; </li></ul><ul><li>2.3 Purpose; 2.4 Overview of the evidential base </li></ul><ul><li>3.0 Brief description of the aims and scope of the CETL: 3.1 Purposes; 3.2 Specific goals; 3.3 Activities; 3.4 Roles and participants </li></ul><ul><li>4.0 Evaluation framework and approach: 4.1 RUFDATA or equivalent statement about overall approach; 4.2 Reflections on the evaluation process </li></ul><ul><li>5.0 Findings addressing key evaluation foci from RUFDATA or equivalent: 5.1 Student experience; 5.2 Connections with external partners; 5.4 Internal strategic impact; 5.5 Effects on teachers; 5.6 Effects on learning designs </li></ul><ul><li>6.0 Lessons learned and future adjustments: 6.1 Overview of new knowledge resources from students; 6.2 Emerging teaching practices; 6.3 Implications for University systems and practices; 6.4 Any sector wide multiplier effects; 6.5 Adjustments and future plans; 6.6 Reflections on the idea of a CETL as a change strategy </li></ul><ul><li>Figures, appendices, data sources and references </li></ul>
  9. 12. Overall challenges <ul><li>Staying creative </li></ul><ul><li>Insisting that we are taking a systems perspectives </li></ul><ul><li>Making the evaluation participative </li></ul><ul><ul><li>Making it safe for colleagues to share ideas and experiences offers </li></ul></ul><ul><ul><li>Creating open discussion about programme evaluation </li></ul></ul><ul><li>Working appreciatively with diversity and creativity </li></ul><ul><li>Supporting and benefiting from local evaluations </li></ul><ul><li>Doing so much in so short a time </li></ul><ul><li>Exploring the CETL programme’s embedded theory of change </li></ul>
  10. 14. Please contact us <ul><li>[email_address] </li></ul><ul><li>[email_address] </li></ul><ul><li>http://web.mac.com/knightpeter/iWeb/Site/Welcome.html </li></ul><ul><li>http://iet.open.ac.uk/pp/peter.knight/research.cfm </li></ul>
  11. 15. The academic team World-class in educational, programme and policy evaluation Track record of evaluative research of the change process Research-based expertise in policy support and development in Higher Education A body of published work on Higher Education policy, change, teaching and learning Professor Paul Trowler Lancaster University Professor Murray Saunders Lancaster University Dr Paul Ashwin Lancaster University Dr Sadie Williams Lancaster University Professor Peter Knight The OU Dr Joan Machell Lancaster University