Rethinking Evaluation


Published on

Rethinking Evaluation: its value and limits in corporate learning. From Learning Technologies 2008 conference

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Rethinking Evaluation

  1. 1. Rethinking evaluation: its value and limits in learning David Wilson [email_address] Europe’s leading Corporate Learning Analysts
  2. 2. ROI Evaluation Value Measurement
  3. 3. ROI Evaluation Value Measurement Learning
  4. 4. ROI Evaluation Value Measurement Learning (ALL)
  5. 5. Research – L&D <ul><li>Evaluation of learning impact is systemically weak </li></ul><ul><li>It is dominated by Kirkpatrick model with increasing pressure to talk about ROI </li></ul><ul><li>As a process, it typically focuses on capturing feedback from the immediate learning event </li></ul><ul><ul><li>Reactionnaires / happy sheets </li></ul></ul><ul><li>Use of technology to automate process is variable but generally very low </li></ul><ul><ul><li>Aggregation and reporting of happy sheet info </li></ul></ul><ul><ul><li>Some use of e-survey and e-assessment tools </li></ul></ul>
  6. 6. Research – Business <ul><li>At Exec level </li></ul><ul><ul><li>There is a disconnect between organisational performance and investment in L&D </li></ul></ul><ul><ul><ul><li>Largely an act of faith </li></ul></ul></ul><ul><ul><ul><li>Some tangential evidence from benchmarking </li></ul></ul></ul><ul><ul><ul><li>L&D provides little or no direct evidence to correct this </li></ul></ul></ul><ul><ul><li>Negatively impacts budgets and commitment of management time and resources </li></ul></ul><ul><li>At a line manager level </li></ul><ul><ul><li>Ensuring staff learning is low on their priority list and personal time </li></ul></ul><ul><ul><li>Disinterested in aggregated happy sheet information </li></ul></ul>
  7. 7. The Cynics View <ul><li>The “business” invests in L&D because it: </li></ul><ul><ul><li>(A) knows it adds value to the business i.e. has evidence </li></ul></ul><ul><ul><li>(B) thinks it to be a good idea, i.e. has belief </li></ul></ul><ul><ul><li>(C) is forced to … i.e. non-discretionary </li></ul></ul><ul><li>But, there is: </li></ul><ul><ul><li>Little (A), Lots of (B), and (C) is overtaking both </li></ul></ul><ul><li>And unfortunately, also a </li></ul><ul><ul><li>(D) wants to shift the focus of blame for non-performance </li></ul></ul>
  8. 8. So What Should We Do About it? ROI Evaluation Value Measurement
  9. 9. ROI Evaluation Value Measurement Outcomes Processes
  10. 10. Learning Impact <ul><li>Make the concept of learning impact central to all your processes </li></ul><ul><ul><li>Analysis, Design, Delivery, Review </li></ul></ul><ul><li>A good example …. </li></ul>
  11. 11. Orientation B&Q University “The way we work” To gain an understanding of the business context To explain the role of the university and how we can support To explain the University process and agree expectations To agree the budget and resources available To agree next steps within process Value of Learning To complete the agreed activity to measure the value of the learning solution Orientation Design the solution To design the solution using university design methodology To agree the sustainable training solution Learning Context Why Learn? To blend or not to blend? Define the desired business outcomes To define the desired business outcomes To gain a view of the current performance To analyse the learning context To agree the way we will measure the value of learning To establish the wider business / change context within which the learning will fit To agree next steps with process Planning & Set up To agree the resources required To plan the development of the team to deliver To define and agree the administration process To plan the logistics of delivery To establish reporting requirements To plan the trial / pilot and how feedback will be captured To test out the delivery of the solution To capture the feedback and learning from the trial / pilot To agree the improvements required and plan for this to take place Trial / Pilot Review To complete the PIR activity (for projects) To ensure the sustainable training is set up To establish how the learning is being put into action within the business Delivery and Action To deliver the solution as agreed Identify the learning needs To discover the learning needs of all job roles To agree the next steps in the process
  12. 12. Learning Impact <ul><li>Impact focuses on Outcomes </li></ul><ul><ul><li>Capability, Behaviour, Performance </li></ul></ul><ul><li>What other factors influence these Outcomes? </li></ul><ul><ul><li>Is this a learning problem at all? </li></ul></ul><ul><li>How does the solution design alleviate or leverage these other influences? </li></ul><ul><li>What can we measure? </li></ul>
  13. 13. Rethinking the Models <ul><li>Kirkpatrick is a bad reference model </li></ul><ul><ul><li>Implied causality </li></ul></ul><ul><ul><li>Legitimises focus on reaction (Level 1) </li></ul></ul><ul><ul><li>Ignores input measures </li></ul></ul><ul><ul><ul><li>Did they attend? </li></ul></ul></ul><ul><ul><ul><li>Time / resource / money invested </li></ul></ul></ul><ul><ul><li>Ignores Influencing Factors </li></ul></ul><ul><li>Refocus on: </li></ul><ul><ul><li>Input measures – time/attendance/cost </li></ul></ul><ul><ul><li>Output measures </li></ul></ul>
  14. 14. Impact of Learning – Our Basic Model Knowledge Application Performance Activity Effectiveness Value How many KNOW it? How many DO it? How it changes PERFORMANCE How effectively they KNOW it? How effectively they DO it? How effectively does it change PERFORMANCE The value of KNOWING The value of DOING The value of the PERFORMANCE change
  15. 15. Impact Framework (Research Version) Learning Process Learning Process Motives/ Expectations Pre- Capability Post- Capability Commitment Post Knowledge/ Skills Pre Knowledge/ Skills Effectiveness Internal Factors Reward Environment Time Support Motivation Goals Application Application External Factors Competition Market Customer Time Budget Goals Performance Performance Learning Process Post Knowledge/ Skills Pre Knowledge/ Skills Value Application Performance Learning Process Post Knowledge/ Skills Pre Knowledge/ Skills Activity Application Performance Analysis Design
  16. 16. IOL Process Outline Analysis Measurement Design Influencing Factors Delivery Review
  17. 17. ROI? <ul><li>ROI is a legitimate discussion for: </li></ul><ul><ul><li>Aggregated investment and activity </li></ul></ul><ul><ul><li>Specific types of learning programmes </li></ul></ul><ul><li>It is not legitimate if: </li></ul><ul><ul><li>No meaningful way of measuring the benefits </li></ul></ul><ul><ul><li>It is not possible to calculate the causality relationship between the learning and the benefits </li></ul></ul><ul><li>Estimation may be possible, but is not the same as measurement </li></ul><ul><ul><li>Sceptical the value of Phillips ROI model although not having a systematic process </li></ul></ul><ul><li>What’s the ROI of measuring and calculating the ROI? </li></ul>
  18. 18. Measurement and Reporting <ul><li>L&D needs a clear strategy and processes for measurement and reporting </li></ul><ul><li>What is already measured? </li></ul><ul><ul><li>Input measures from LMS </li></ul></ul><ul><ul><li>Output measures from business </li></ul></ul><ul><li>What can be meaningfully measured in addition </li></ul><ul><ul><li>Surveying behaviour and performance change </li></ul></ul><ul><ul><li>Knowledge and competency assessments </li></ul></ul><ul><li>Reporting </li></ul><ul><ul><li>Value Added / Dashboards </li></ul></ul><ul><ul><li>Key programmes and outcomes </li></ul></ul>
  19. 19. Measuring and Reporting Impact Defined by project Often Impact (L4) Learning (L2) Behaviour (L3) & Learning (L2) Level of Assessment Aggregated (Benchmarked) Addressed specific learning needs Individual On Demand Directly (Outcomes) Organisationally (Human Capital) Valuation Meet specific project learning goals Performing role with requisite knowledge/skills Success The Business Project Organisation Driver Project-Driven Core Competency Valuing the Learning Portfolio
  20. 20. Questions? [email_address] Research Knowledge Base: http://