Evaluating the Effectiveness of Academic Leadership Programmes - a Way Forward
•
1 like•856 views
Evaluating the Effectiveness of Academic Leadership Programmes - a Way Forward Presented by Linda McKellar (formerly McLain) at the TEMC conference, Melbourne, October 2010
More Related Content
What's hot
What's hot (20)
Similar to Evaluating the Effectiveness of Academic Leadership Programmes - a Way Forward
Similar to Evaluating the Effectiveness of Academic Leadership Programmes - a Way Forward (20)
Recently uploaded
Recently uploaded (20)
Evaluating the Effectiveness of Academic Leadership Programmes - a Way Forward
- 1. Evaluating the Effectiveness of Academic Leadership Programmes A Way Forward Linda McLain McKellar TEMC Melbourne October 2010
- 2. Session Overview The need for evaluation The University of Auckland overview A case study at the U of A Findings / Conclusions Summary
- 3. 2E or not 2E?
- 4. 1. Articulate Strategic Imperatives Source: Cacioppe 1998
- 6. 38 550
- 7. 6 785
- 8. 60 TOP
- 9. 7
- 10. Heads Up Future New Current
- 11. Leadership Programmes for Women
- 14. Before
- 15. After “Excellence is not a singular act, but a habit. You are what you repeatedly do” Aristotle
- 16. A snapshot 1 Review
- 17. A snapshot 2 Mapping
- 18. A snapshot 3 Best Practice
- 19. Kirkpatrick’s 4- Level Evaluation Model Increasing organisational performance Increasing effort and complexity
- 20. Life is trying things to see if they workRay Bradbury
- 21. So what?
- 23. Continuous improvement Stakeholder feedback Improve Incorp improvements Plan Review Programme Design Programme Evaluation Do Programme Delivery
- 25. clear framework
- 27. in a large organisation
- 30. References
- 31. Kirkpatrick 4-Levels Model as used at UoA
- 33. McLain PDRI Model (2010)
- 34. UoA Leadership Programs detailed evaluation informationAvailable for download from tslc.com.au/mckellar Slides by:
Editor's Notes
- IntroductionLeadership development programmes carry with them the inherent difficulty of measuring success. The question of how and what to evaluate is complex and multi faceted.Some university programmes have no formal evaluation of leadership programmes except the usual ‘happy sheet’ which included (sometimes) a link back to the objectives of the learning opportunity and also asked about the level of catering and venue. These last two issues are still important aspects because these ‘hygiene issues’ must be considered and one must be comfortable to be able to learn.While evaluations are often carried out as a necessary ‘add on’, it is important to consider the question of why programmes should be evaluated in the first place; what is to be gained by evaluating a leadership programme?
- Leadership Use examples of leadership definitions/research, etc. from AmandaSinclair, Brad Jackson and Henry Mintzberg and perhaps some othersWeave analogy about good leadership means higher staff retention whichmeans a more profit or...............EvaluationDo we need it? | Why do we do it? Why is there such an emphasis on leadership & management programs?________________________________________________________________________________________________________According to the Centre for Creative Leadership (2009), there are many reasons for evaluating leadership development programmes, including:the ability to demonstrate the benefit of the programmeto use the evaluation as a mechanism to improve the initiativesto demonstrate the impact on the specific goalsTo promote learning-centred reflection. For example, if participants in the leadership development programme are contacted sometime after the programme has been completed and are asked how it has affected their work (and hopefully improved it).to link with accepted contextually and important leadership capabilities within the organisation, andto be used as a catalyst to discuss what works and whyGOING TO NEED TO GIVE SPECIFIC EXAMPLES OF EACH OF THESE
- An integrated approach to planning leadership developmentIn short, evaluation enables organisations to track behavioural change, measure the value of these programmes and their contribution to the organisation’s goals and objectives. “…… when evaluating development programmes and whole systems of programmes, the real goal is to find a causal link between initiative objectives and behavior change or development” (Allen, 2009).From Scott, G. (2008),Learning Leaders in times of change – Academic Leadership Capabilities for Australian Higher Education.The process shown in this Figure,illustrates an integrated model and approach for the design and assessment of effective leadership development programmes. Evaluation should be developed at Steps four and five so there has been a link developed between the programme’s objectives and aims and evaluation can be used as a source of measure of success. Evaluation is integral to the entire process of programme development.
- In my practice, I have designed the evaluation methodology and instrument concurrently with the development of the academic leadership programme. This evaluation has been linked clearly to the objectives of the programme which themselves link clearly to the strategic objectives of the university. This seeks to enhance organisational learning and to demonstrate the link between leadership development and evaluation in achieving organisational goalsIn order for us to navigate this story, we must first look at the complexity of the University of Auckland in which this Case Study is set.
- University of Auckland38 550 students
- University of Auckland6 785 staff
- University of AucklandOne of the top 60 ranked research universities globally
- there are a zillion leaders at UofA and we have 7 leadership programmes (compared to ECU, we had 180 identified leaders down to HoS & HEW 10 level)
- Links to further information on these leadership programs are available on a website for you to download and a link is available at the end of presentation
- Links to further information on these leadership programs are available on a website for you to download and a link is available at the end of presentation
- Links to further information on these leadership programs are available on a website for you to download and a link is available at the end of presentation
- This program is currently in it’s first yearLinks to further information on these leadership programs are available on a website for you to download and a link is available at the end of presentation
- These leaderships programmes are multi, layered, multi-faceted & complex (see handout available online)We didn't know if behaviour change was happening and we didn't know if learning was taking place in the old approach prior to the Leadership Programme Review____________________________________________________________________________________________________Some techniques previously used in non-systematic waysWorkshop feedback sheetsFacilitator impactAssessment of programme aims / achievementOverall programme satisfactionReflective journal writing____________________________________________________________________________________________________Fundamentally:Ad hocNon-systematicScattergunNo modelNo formal evaluation programme closure (happy sheets). How are these suggestions ever used / not used in PDRI
- Talk about the Foundation or grounding for this work and other modelsof leadership programmes and leadership capability frameworks.I commissioned an overall leadership programme review at UofA and thisled into the question of how we evaluate our leadership programmesIn order to know if leadership programmes are meeting their objectives in helping the organisation to achieve its goals, there needs to be a structured framework for evaluating the impact.
- We went through a 3 step processRun Animation with mouse clicks and explain steps BRIEFLYReview Mapping PROGRAMME | AUDIENCE | PURPOSE | DURATION | OTHER | EVALUATION TECHNIQUESBest Practice - looked at Kirkpatrick, deSimone et al, Vaill, Mintzberg and othersOne way to do this is to use Kirkpatrick’s 4 Level Evaluation Model, which outlines the hierarchy of change, in a useful framework for leadership programme evaluation. Presently, the leadership programmes at the University of Auckland are evaluated at Level 1 and sporadically and unsystematically at Levels 2 and 3, which I will explain on the next slide.
- We went through a 3 step processRun Animation with mouse clicks and explain steps BRIEFLYReview Mapping PROGRAMME | AUDIENCE | PURPOSE | DURATION | OTHER | EVALUATION TECHNIQUESBest Practice - looked at Kirkpatrick, deSimone et al, Vaill, Mintzberg and othersOne way to do this is to use Kirkpatrick’s 4 Level Evaluation Model, which outlines the hierarchy of change, in a useful framework for leadership programme evaluation. Presently, the leadership programmes at the University of Auckland are evaluated at Level 1 and sporadically and unsystematically at Levels 2 and 3, which I will explain on the next slide.
- We went through a 3 step processRun Animation with mouse clicks and explain steps BRIEFLYReview Mapping PROGRAMME | AUDIENCE | PURPOSE | DURATION | OTHER | EVALUATION TECHNIQUESBest Practice - looked at Kirkpatrick, deSimone et al, Vaill, Mintzberg and othersOne way to do this is to use Kirkpatrick’s 4 Level Evaluation Model, which outlines the hierarchy of change, in a useful framework for leadership programme evaluation. Presently, the leadership programmes at the University of Auckland are evaluated at Level 1 and sporadically and unsystematically at Levels 2 and 3, which I will explain on the next slide.
- Kirkpatrick Level 1 Participant ReactionMethods – End-of-programme surveyCommon Metrics - Participant satisfaction | Achievement of programme outcomes | Intent to use content | Ratings of facilitator knowledge and skillMeasuring Training Effectiveness - The ‘course feedback’ sheets handed out at the completion of a workshop. Commonly we are asked questions around appropriateness of the course content, the competency of the facilitator and so on.Kirkpatrick Level 2 Attainment of Knowledge (Learning)Methods – End-of-programme survey | Participant impact surveyCommon Metrics - % who report successfully learning the models and content presented during the programme | % of participants who report using specific programme content and finding it valuableMeasuring Training Effectiveness - While an increase in knowledge will not guarantee changes in behaviour in the workplace, often it is the foundation of any future changes. By measuring knowledge attainment, we are able to ascertain the ability of an individual to take the next step and apply the learningKirkpatrick Level 3 Behaviour ChangeMethods – 360 assessment |Participant impact survey |InterviewsCommon Metrics - % who report that the programme outcomes were achieved | Behaviour change in targeted competencies (amount or degree of change) |% who receive high levels of supervisory support | % who experience common challenges in changing behaviour in targeted areasMeasuring Training Effectiveness - This level of evaluation is seen as the most powerful indicators of learning effectiveness. Has the participant changed their behaviours? Are they now performing to a higher level? Just how much more competent are the becoming? Often the course participant and the line manager are best placed to evaluate changes in behaviour.Kirkpatrick Level 4 Business Results Methods – 360 assessment | Participant impact survey | Interviews | Business metrics analysisCommon Metrics - Amount or % of change reported for key competencies | The leading edge in the field of leadership development evaluation is the linkage of programme evaluation data with independent business metrics. This includes not only gathering participant perceptions of business impact within the work group, but also analysing business metrics over time in relation to the programme evaluation data.Measuring Training Effectiveness - “Is the organisation more profitable as a result of the learning program?”(DeSimone, Werner and Harris 2002, p232). This level is seen as the most attractive to those organisations whose sole focus is on ‘the bottom line’
- The “So What” of evaluationAn alternate view which takes account of the many variables to evaluating, is that a ‘snapshot’ is not easily taken because people are changing all the time and the demands on them and the effect of these demands influence how they act and react; how they learn; how they transfer these learning’s and therefore, the bottom line of how we measure the results or the ‘return on investment’. The ‘five-way bottom line’ (Vaill, 1989) supports the premise that the organisation is a five-dimensional, intertwined stream whose energy comes from the individual and joint actions of people as they work out their sense of what is important, eg. organisational goals. This means that measuring any of Kirkpatrick’s levels of impact is difficult and we can only do our best to access and report on and then improve each level.To purport to be able to evaluate leadership programmes against tangible measurements is very difficult. We do the best we can, realising that this is a complex issue with many dimensions and relationships, actions and complexities.
- Leadership Program Review Findings / ConclusionsIn 2009, a review of all of the HR/SODU leadership programmes was commissioned. The purpose was to identify and celebrate what we were doing well; to identify and to improve areas; and, to identify any gaps in the programme content and delivery mode(s) and the evaluation framework.The results found that the content of the programmes was relevant and in line with the findings of the Learning Leaders research (see figure 1) and the evaluation processes in place for the programmes ensured they were tailored towards their participants and the goals of the organisation. As successful as they are, there are opportunities where HR/SODU can further improve upon the excellent standard of leadership development at the University of Auckland. The implementation of learning outcomes is perhaps the only area where other Universities may hold an advantage.The Leadership Programme Review identified the following general issues for improvement:Evaluations are at Level 1 of Kirkpatrick’s level of impact and should be at Level 3There is only ad hoc and anecdotal evidence that the programmes are helping the University of Auckland achieve its strategic goalsThe evaluations are weakly linked to objectives and learning outcomesThere is a need for a clearer systematic process for the evaluations outcomes from previous programmes to be reviewed and incorporated into the programme improvements or notThere is no long term impact measureThere are no KPI’sAn efficient framework could include a variety of options for evaluation which would be dependant upon the programme’s aims and objectives and which would clearly link back to these:Tracking inputs (Level 1)Measuring satisfaction (Level 1)Assessing knowledge/skill acquisition (Level 2)Monitoring behaviour and performance changes (Level 3)Determining results (Level 4)Tracking goal achievement (Level 3 and 4)There is a need to determine which level or combination of levels would be appropriate for which leadership development programme and/or learning opportunity. For example, it may not be necessary to evaluate at Level 4 at all. For some programmes, Level 3 is the most critical level for evaluating the movement to, and alignment with, the organisation’s goals and objectives. Evaluating at Level 1 however may be sufficient for some learning opportunities.
- Conclusion – A WAY FORWARDThe way forward is to develop an evaluation framework which outlines systematic use at specific levels and times during leadership development programmes. It needs to be easy and clear.This could be done in a variety of ways, but most importantly, it needs to be clear to not only the programme developers, but to the participants of the programme that there are a number of identified outcomes that will be evaluated and reported upon. The past participants could even be involved in the evaluation development.Specifically, the way forward could include the development of a ‘feedback loop’ that would help communicate the outcomes of the programme to the wider university community. These methods could include at the ‘ground level’ “An example of good leadership column” in the University News monthly publication; information on the web which could include a similar story; blogs/Twitter or Facebook space or other ideas to let the university community know that the programmes are worthwhile and that these are measurable University results. The feedback loop could include (see graphic)