An Inch or a Mile?Proven Practices in Measuring Learning ImpactNovember, 2010 <br />KedarKulkarni<br />
ESI & KA Strategic Partnership<br />© ESI International, Inc. 2010.<br />
ESI<br />© ESI International, Inc. 2010.<br />US Offices<br />Arlington/Washington D.C.<br />Chicago<br />New York<br />Sa...
KnowledgeAdvisors & Metrics that Matter®<br />© ESI International, Inc. 2010.<br />KnowledgeAdvisors is ahuman capital ana...
Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Lea...
Current Market Trends<br />More focus on demonstrating business outcomes<br />Open to collaborating with partners and supp...
Current Market Trends—The Undisputed Truth<br />© ESI International, Inc. 2010.<br />Kirkpatrick Four Levels	Continuous Le...
Level 2:  Learning
Level 3:  Transfer
Level 4:  Results (ROI)</li></ul>Effective Assistance (Coaching/Counseling) by Manager<br />Manager<br />(Sender)<br />Man...
Scrap Learning/Manager Support<br />© ESI International, Inc. 2010.<br />Typical Training without Strong Manager Support<b...
Scrap Learning is a Real Problem<br />© ESI International, Inc. 2010.<br />60%of learning is not applied to the job….that’...
  When is learning applied to the job?
  What are the barriers to impact?
  What can be done to reduce scrap?</li></ul>Source: KnowledgeAdvisors <br />
Current Market Trends— Measuring Learning to Impact Project Performance<br />© ESI International, Inc. 2010.<br />
A New Paradigm<br />© ESI International, Inc. 2010.<br />Build on Existing Data Collection Points<br />ROI<br />VOI<br />
Determine where you need to go<br />But no matter where you want to go, your must….<br />© ESI International<br />
Add Value—<br />Understand the value <br />Identify value metrics <br />Establish a baseline for each metric—pragmatically...
Top Five Challenges<br />Lack of resources<br />No executive support<br />Lack of qualified people to determine and track ...
Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Lea...
Showing Business Impact<br />© ESI International, Inc. 2010.<br />Requires actual results data and advanced analysis techn...
Smile Sheets  Smart Sheets<br />© ESI International, Inc. 2010.<br />Level 1<br />REACTION<br />Level 2<br />Learning<br ...
Making Decisions With Impact Data<br />© ESI International, Inc. 2010.<br />Cut or improve programs that aren’t providing ...
Human Capital Approach<br />Uses improved job performance as the overall measure of business impact<br />Easy to use, prac...
Human Capital Approach (continued)<br />Uses principles of estimation, isolation, and adjustment<br />Assumption is that h...
Business Results Evaluation Approach<br />Use specific business results as a measure of business impact<br />Uses an advan...
Business Results Evaluation Process<br />© ESI International, Inc. 2010.<br />
Case Example: Advanced Business Results<br />© ESI International, Inc. 2010.<br />Background:  ERP company performing cust...
Business Impact Template<br />© ESI International, Inc. 2010.<br />Monetize Results <br />Estimate the Impact of Outside F...
Use Business Impact Templates<br />© ESI International, Inc. 2010.<br />Background:  Large CPG based in Canada,sales train...
Use Actual Result Correlations<br />This is a method where actual results can be input or imported at an aggregate, period...
Causal Modeling<br />High level statistical analysis using Structural Equation Modeling<br />Determines the key factors th...
Performance Optimization<br />© ESI International, Inc. 2010.<br />
Actionable Metrics<br />© ESI International, Inc. 2010.<br />Goals:  Actual to Goal <br />Benchmarks:  <br />Internal or E...
IMAGINE…<br />© ESI International, Inc. 2010.<br />Online course end evaluations:<br />Level 1 & 2<br />Predictive Level 3...
Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Lea...
Informal Learning Measurement<br />© ESI International, Inc. 2010.<br /># of respondents<br />% of respondents supporting ...
Quick Poll evaluation at point of use (ex. Amazon-like star rating)
Upcoming SlideShare
Loading in...5
×

Kedar kulkarni ev10 nov09-pmi_india_aninchora_mile

480

Published on

Published in: Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
480
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • We are a human capital analytics firm providing learning and talent measurement solutions.Our on-demand evaluation software, Metrics that Matter, was developed to automate the measurement approaches recommended by Kirkpatrick, Phillips, Brinkerhoff, and Bersin. It offers deep analytic capabilities to help organizations demonstrate the value of training and optimize their workforce.
  • Some of you might be wondering how can I leverage some of the strategies that Jeff talked about to improve project performance. At the end of the day, it is all about project performance.  Allow me to show you how you can link a learning investment in PM to an impact on project performance: CLICKI am sure many of you will relate to this environment—it has the triple constraint and three additional areas that impact project performance.  You can define the business results in many ways, For example, CLICK The desired business results can be defined into these three areas (decreasing costs, increasing quality and increasing productivity). Many others, but for our purposes today let’s focus on these three which are commonly associated with project performance. This chart represents the outcomes of post event and follow up surveys of the learners. Post event is immediately after the class and follow up is 60 days after the class. This data represents 650 learners, for a cross functional PM program over a period of 5 months. Let’s look at the first business result Post event, 24% of learners said that felt they will be able to decrease costs in the future. When we asked them to isolate what percentage of this predicted decrease in costs was only due to the training they just received, the average was 49%. 60 days later we asked them again. 21% of them said that they will be able to decrease costs. When we asked them to isolate what percentage of this decrease in costs was only due to the training they took 60 days ago, the average was 55%. This is a good story that is also supported by a lot of qualitative data with examples on enablers and challenges they faced applying the best practices they leaned in the training. Similarly, the results for the other two areas are shown in this table. A picture starts to emerge that tells you a couple of things:Are you on the right path? Is this making a difference?Is there a need for immediate adjustments or course correction (for example if 2% can apply, we have a problem)You can start to calculate a business impact using some of the approaches that Jeff talked about. This may not necessarily be the 100% proof, but is great evidence that shows better project performance.  As Govt Contractors, I hope you all can see how important it is to implement a scalable and repeatable measurement program to help you assess the impact of PM learning to the outcomes of your projects. At ESI, we believe it is crucial and we have made the investment to make sure that all our programs have this approach built into them. Thank you all for your time today. I will turn it back to Janine.  
  • Discuss Pros/Cons of each
  • Kedar kulkarni ev10 nov09-pmi_india_aninchora_mile

    1. 1. An Inch or a Mile?Proven Practices in Measuring Learning ImpactNovember, 2010 <br />KedarKulkarni<br />
    2. 2. ESI & KA Strategic Partnership<br />© ESI International, Inc. 2010.<br />
    3. 3. ESI<br />© ESI International, Inc. 2010.<br />US Offices<br />Arlington/Washington D.C.<br />Chicago<br />New York<br />San Francisco<br />ESI Partners<br />Japan Poland <br />South Korea <br />Global Offices<br />Australia Canada China Germany Hong Kong India New Zealand Singapore South Africa Spain Sweden Taiwan United Arab Emirates United Kingdom <br />
    4. 4. KnowledgeAdvisors & Metrics that Matter®<br />© ESI International, Inc. 2010.<br />KnowledgeAdvisors is ahuman capital analytics firm.<br />We are the world’s largest provider of learning and talent measurement solutions. <br />Experts in metrics that matter<br />
    5. 5. Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Learning Impact<br />A Note on Informal Learning and Measurement Sustainability<br />Project Management Training—Imagine the Impact to Your Projects (Case study)<br />© ESI International, Inc. 2010.<br />
    6. 6. Current Market Trends<br />More focus on demonstrating business outcomes<br />Open to collaborating with partners and suppliers to build and execute measurement plans<br />Focus on transparency and project reporting<br />More focus on how to establish “Current” State <br />Examples: Knowledge / Competencies / Organizational / Portfolio / Project Assessment Tool<br />© ESI International, Inc. 2010.<br />
    7. 7. Current Market Trends—The Undisputed Truth<br />© ESI International, Inc. 2010.<br />Kirkpatrick Four Levels Continuous Learning Process<br /><ul><li>Level 1: Reaction
    8. 8. Level 2: Learning
    9. 9. Level 3: Transfer
    10. 10. Level 4: Results (ROI)</li></ul>Effective Assistance (Coaching/Counseling) by Manager<br />Manager<br />(Sender)<br />Manager<br />(Developer)<br />Assess<br />Learn<br />Reinforce/Validate<br />Pre-course<br />preparation<br />and assessment<br />Intensive learning<br />through the course<br />(Classroom/e-Training)<br />Post-course<br />implementation<br />and assessment<br />Learning and Development by Him/Herself<br />Person<br />(Trainee)<br />Person<br />(Developed)<br />
    11. 11. Scrap Learning/Manager Support<br />© ESI International, Inc. 2010.<br />Typical Training without Strong Manager Support<br />COSTS<br />? %<br />DID NOT TRY IT AT ALL<br />? %<br />TRIED IT TO SOME EXTENT BUT<br />GAVE UP <br />? %<br />TRIED IT AND GOT POSITIVE RESULTS<br />UNREALIZED VALUE<br />RETURN<br />The preferred method of optimizing training<br />COSTS<br />? %<br />DID NOT TRY IT AT ALL<br />? %<br />TRIED IT TO SOME EXTENT BUT<br />GAVE UP <br />85%+<br />TRIED IT AND GOT POSITIVE RESULTS<br />RETURN<br />Source: Robert Brinkerhoff<br />
    12. 12. Scrap Learning is a Real Problem<br />© ESI International, Inc. 2010.<br />60%of learning is not applied to the job….that’s scrap, but as managers YOU can make a difference!<br /><ul><li> How much learning is applied to the job?
    13. 13. When is learning applied to the job?
    14. 14. What are the barriers to impact?
    15. 15. What can be done to reduce scrap?</li></ul>Source: KnowledgeAdvisors <br />
    16. 16. Current Market Trends— Measuring Learning to Impact Project Performance<br />© ESI International, Inc. 2010.<br />
    17. 17. A New Paradigm<br />© ESI International, Inc. 2010.<br />Build on Existing Data Collection Points<br />ROI<br />VOI<br />
    18. 18. Determine where you need to go<br />But no matter where you want to go, your must….<br />© ESI International<br />
    19. 19. Add Value—<br />Understand the value <br />Identify value metrics <br />Establish a baseline for each metric—pragmatically<br />Identify leading indicators for each metric<br />Measure continuously and manage your stakeholders<br />© ESI International, Inc. 2010.<br />Influence vs. Interest <br />
    20. 20. Top Five Challenges<br />Lack of resources<br />No executive support<br />Lack of qualified people to determine and track metrics<br />Afraid of the results<br />The need has not been there <br />© ESI International, Inc. 2010.<br />www.esi-intl.com/valuesurvey<br />
    21. 21. Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Learning Impact<br />A Note on Informal Learning and Measurement Sustainability<br />Project Management Training—Imagine the Impact to Your Projects (Case study)<br />© ESI International, Inc. 2010.<br />
    22. 22. Showing Business Impact<br />© ESI International, Inc. 2010.<br />Requires actual results data and advanced analysis techniques<br />High Investment<br />Once you determine Business Impact what analysis is appropriate, select a method:<br />Causal Modeling<br />Actual Results Correlations<br />Typically employs surveys<br />Business Impact Templates<br />Advanced Business Results Evaluation<br />Human Capital Approach<br />Smart Sheets<br />Low Investment<br />Complex<br />Simple<br />
    23. 23. Smile Sheets  Smart Sheets<br />© ESI International, Inc. 2010.<br />Level 1<br />REACTION<br />Level 2<br />Learning<br />Level 3<br />Job Impact<br />Level 4<br />Results<br />Level 4<br />RESULTS<br />MANAGEMENT SUPPORT<br />Level 5<br />ROI/ROE<br />Evidence of Success<br />
    24. 24. Making Decisions With Impact Data<br />© ESI International, Inc. 2010.<br />Cut or improve programs that aren’t providing impact to the business<br />“We need our measurement tools and process now more than ever; if we don’t have access to this data, how will we be able to make intelligent decisions on what to cut?”<br />- KnowledgeAdvisors Client<br />
    25. 25. Human Capital Approach<br />Uses improved job performance as the overall measure of business impact<br />Easy to use, practical, scalable, replicable<br />Provides a financial ROI relative to the improvement in human capital performance<br />© ESI International, Inc. 2010.<br />
    26. 26. Human Capital Approach (continued)<br />Uses principles of estimation, isolation, and adjustment<br />Assumption is that human capital’s value (i.e. salary) is like any other asset that has expected value that can appreciate or depreciate over time<br />© ESI International, Inc. 2010.<br />
    27. 27. Business Results Evaluation Approach<br />Use specific business results as a measure of business impact<br />Uses an advanced survey to estimate improvement in results<br />Leverage technology to make practical and scalable<br />Provides a financial ROI relative to the change in the business result that prompted the training<br />Uses principles of estimation, isolation, and adjustment specific to the business result<br />© ESI International, Inc. 2010.<br />
    28. 28. Business Results Evaluation Process<br />© ESI International, Inc. 2010.<br />
    29. 29. Case Example: Advanced Business Results<br />© ESI International, Inc. 2010.<br />Background: ERP company performing customer education<br />Business Issue: Show value of training to customers in terms of increased productivity and quality<br />Measurement Solution: Collected 12,770 end of program evaluations within 6 months asking participants to forecast impact<br />Analytics Results: 18.2% estimated increase in productivity from the training (adjusted down for self-reported bias)<br />
    30. 30. Business Impact Template<br />© ESI International, Inc. 2010.<br />Monetize Results <br />Estimate the Impact of Outside Factors<br />Compare to Cost of Training<br />
    31. 31. Use Business Impact Templates<br />© ESI International, Inc. 2010.<br />Background: Large CPG based in Canada,sales training<br />Business Issue: Visible sales program, requested to link between it and sales results<br />Analytics Solution: Used process to evaluate sales change and impact training had on sales<br />Analytics Results: 7% increase in sales, 20% of the increase was due to training, this was a $74,000 increase per sales person when the training was $5000 per person<br />
    32. 32. Use Actual Result Correlations<br />This is a method where actual results can be input or imported at an aggregate, periodic frequency. It is then correlated in time periods against training evaluation data for impact indicators.<br />The example below is from sales data and sale training.<br />© ESI International, Inc. 2010.<br />
    33. 33. Causal Modeling<br />High level statistical analysis using Structural Equation Modeling<br />Determines the key factors that influence business impact<br />Causation not correlation<br />The Causal Model can be used to predict future results and determine investments<br />© ESI International, Inc. 2010.<br />WorthwhileInvestment<br />0.483<br />0.625<br />Courseware Quality<br />IndividualLearning<br />0.337<br />0.556<br />0.083<br />InstructorEffectiveness<br />
    34. 34. Performance Optimization<br />© ESI International, Inc. 2010.<br />
    35. 35. Actionable Metrics<br />© ESI International, Inc. 2010.<br />Goals: Actual to Goal <br />Benchmarks: <br />Internal or External<br />Dashboards: Concise Data<br />Trends: Time based analysis<br />
    36. 36. IMAGINE…<br />© ESI International, Inc. 2010.<br />Online course end evaluations:<br />Level 1 & 2<br />Predictive Level 3<br />Predictive Business Impact<br />STANDARD REPORT<br />STANDARD REPORT<br />Online follow-up <br />60-90 days—with participant and with their manager about b & c above<br />
    37. 37. Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Learning Impact<br />A Note on Informal Learning and Measurement Sustainability<br />Project Management Training—Imagine the Impact to Your Projects (Case study)<br />© ESI International, Inc. 2010.<br />
    38. 38. Informal Learning Measurement<br />© ESI International, Inc. 2010.<br /># of respondents<br />% of respondents supporting each type of informal learning<br />Source: KnowledgeAdvisors Informal Learning Study, 2009<br /><ul><li>Measure activity trends (ex. utilization of performance support tool)
    39. 39. Quick Poll evaluation at point of use (ex. Amazon-like star rating)
    40. 40. Comprehensive periodic evaluation to review quality and impact
    41. 41. In research, 100% who measured had higher future budget allocations</li></li></ul><li>Measurement Sustainability<br />© ESI International, Inc. 2010.<br />1<br />2<br /><ul><li>Roles and accountabilities for metrics are defined with consistent expectations on what should be measured.
    42. 42. Strong leadership for driving measurement with governance to support it. </li></ul>8<br /><ul><li>Culture supports and drives need for standards and common talent development processes and measurement</li></ul>3<br />Data Utilization<br />3<br /><ul><li>Broad base of skills in analytics and measurement and strong capabilities in end-to-end measurement methods</li></ul>7<br /><ul><li>Strong desire to use the data for decision making
    43. 43. Maturity in how data is reported and used to improve end to end talent processes</li></ul>Processes<br />Technology<br />6<br /><ul><li>Standardization for a consistent and single point of analytics with integration to and from feeder systems</li></ul>4<br /><ul><li>Established and consistent processes, technologies and communications for measurement</li></ul>Standards, Tools<br /><ul><li>Standards have been created for data collection
    44. 44. Data integrity exists </li></ul>5<br />
    45. 45. Agenda<br />Current market trends<br />Top five challenges for measuring impact<br />Six Proven Practices in Measuring Learning Impact<br />A Note on Informal Learning and Measurement Sustainability<br />Project Management Training—Imagine the Impact to Your Projects (Case study)<br />© ESI International, Inc. 2010.<br />
    46. 46. Project Management – Bringing Value to the Business<br />Effective and profitable management of complex projects<br />Global consistency and capability<br />Effective use of resources <br />Higher customer satisfaction<br />Increased employee satisfaction <br />Significant competitive advantage<br />© ESI International, Inc. 2010.<br />
    47. 47. What is Your Definition of Value<br />© ESI International, Inc. 2010.<br /><ul><li>Increased customer & staff satisfaction
    48. 48. Increased quality
    49. 49. Increased productivity
    50. 50. Increased revenue
    51. 51. Decreased costs / cycle times
    52. 52. Decrease risk</li></ul>Maximize return:<br /><ul><li>Improve business results;grow revenue and earnings, cash flow, and reduced cost of operations</li></ul>Increase agility:<br /><ul><li>Enable the business organization and operations to adapt to changing business needs</li></ul>Improve performance:<br /><ul><li>Improve business operations performance end-to-end across the enterprise
    53. 53. Increase customer and employee satisfaction</li></ul>Minimize risk:<br /><ul><li>Ensure continuity of internal business operations, while minimizing exposure to risk factors</li></li></ul><li>How does this impact Project Performance?<br />© ESI International, Inc. 2010.<br />
    54. 54. Think CSI<br />© ESI International, Inc. 2010.<br />“In the absence of proof, let evidence be enough.” <br />Donald Kirkpatrick<br />
    55. 55. Questions?Thank you!November, 2010<br />Raed S. Haddad, <br />Senior Vice President<br />ESI International<br />

    ×