Putting Program Evaluation to Work for You

  • 319 views
Uploaded on

April 19, 2012 ASTD-SHRM Joint Meeting, Jackson, MS. …

April 19, 2012 ASTD-SHRM Joint Meeting, Jackson, MS.


John Kmiec, PhD, CRP

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
319
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
20
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Use these bullets to discuss additional reasons and reinforce their reasons
  • Group Activity/Guided Discussion: Use open-ended questions to bring out their issues and why they are issues for them. Tie back to any problems applying the Program Outcome Model, as applicable. Transition into Evaluation Planning as the means to smooth the practice of evaluation.
  • Group Activity/Guided Discussion: Use open-ended questions to bring out their issues and why they are issues for them. Tie back to any problems applying the Program Outcome Model, as applicable. Transition into Evaluation Planning as the means to smooth the practice of evaluation.
  • Group Activity/Guided Discussion: Use open-ended questions to bring out their issues and why they are issues for them. Tie back to any problems applying the Program Outcome Model, as applicable. Transition into Evaluation Planning as the means to smooth the practice of evaluation.
  • Group Activity/Guided Discussion: Use open-ended questions to bring out their issues and why they are issues for them. Tie back to any problems applying the Program Outcome Model, as applicable. Transition into Evaluation Planning as the means to smooth the practice of evaluation.
  • Group Activity/Guided Discussion: Use open-ended questions to bring out their issues and why they are issues for them. Tie back to any problems applying the Program Outcome Model, as applicable. Transition into Evaluation Planning as the means to smooth the practice of evaluation.

Transcript

  • 1. Putting Program Evaluation to Work for You A Learning Intervention for Work Engagement PolyWrighton, USA John Kmiec, PhD, CRP The University of Southern Mississippi john.kmiec@usm.edu 228.365.2559
  • 2. OverviewThe purpose of this discussion is to explore theelements of effective program evaluation  Why do we need evaluation?  What are your biggest problems with evaluation?  How can evaluation planning help?  What are the components of effective evaluation?  Can you give us an example? (PolyWrighton case study)  Where can we find additional resources?
  • 3. Why do we need evaluation?  Obtain feedback from stakeholders  Verify that the methods specified were used  Determine whether the solution worked  Establish if sufficient impact was made  Help make adjustments during the program  Maintain some control over the project  Account for resources and results  Help plan future actions In today’s globally competitive changing market and constant technological advancement, training is a given. Doing training well – getting results from learning investments – is a must, not a choice. Robert Brinkerhoff
  • 4. What is your biggest problem with evaluation?  Too many theories and models  Models are too complex  Lack of understanding of evaluation  Evaluation is considered a post-program activity  Lack of support from key stakeholders  Improper use of evaluation data  Other
  • 5. How can evaluation planning help?  Shows in advance how you will know when you have accomplished your objectives  Reassures stakeholders that their financial commitment is being well spent  Identifies what is going well and what is not
  • 6. How can evaluation planning help? Who… Will be doing the evaluation? What… Is the criteria for evaluation? Is the process for data analysis? Test instruments/questionnaires will be used? Is and validity/reliability of these instruments? When… Will the evaluation be conducted? Will evaluation reports be produced? Where… Will adjustments be made if needed? How… Was the evaluator chosen? Will you determine if objectives were met and specified methods were used? Will the data be gathered? Analyzed? Disseminated?
  • 7. What are the components of effective evaluation?  An evaluation framework  A data collection plan  A data analysis plan
  • 8. What are the components of effective evaluation?  An evaluation framework  Illustrates program objectives  Shows the chain of impact  Provides Structure  Aids communication  Facilitates Planning
  • 9. Phillips ROI Methodology® The Levels of Evaluation Framework 0. Inputs & Indicators: Measures inputs, including indicators of volume and scope 1. Reaction & Planned Action: Measures satisfaction with the program and planned actions 2. Learning: Measures changes in participants’ knowledge, skills, abilities and attitudes 3. Application: Measures changes in behavior 4. Impact: Measures changes in critical outcomes 5. ROI: Compares program benefits to costs
  • 10. Phillips ROI Methodology® Program Evaluation Framework Starts with a proper needs assessment where Needs drive program objectives and Program objectives drive measurement andevaluation, continuous improvement and results!
  • 11. What are the components of effective evaluation?  A data collection plan shows  Broad program objectives (Why do I ask?)  Measures/Data (What do I ask?)  Data Collection Methods/Instruments (How do I ask?)  Data Sources (Who do I ask?)  Timing (When do I ask?)  Responsibilities (Who asks?)
  • 12. What are the components of effective evaluation?  A data analysis plan shows  Methods for isolating effects of the program  Ways for converting results to dollar values  Program cost categories  Intangible benefits  Communication targets
  • 13. Can You Give Us an Example? A Learning Intervention for Work Engagement PolyWrighton, USA Case Study
  • 14. Background• PolyWrighton manufactures high quality, lightweight plastics used to package food, beverage and personal care products• Complex, massive, hazardous machinery used to process tons of toxic, flammable chemicals under high heat and pressure• Rigorous hygienic, chemical and environmental safety standards and specifications leave very little room for error• Raw materials are very expensive and frequently in short supply, making reduction of product waste and rework imperative
  • 15. Background• Constant monitoring and testing, state-of-the-art technology and a highly skilled workforce are critical to quality and safety• Bankruptcy led to purchase by an out-of-state investment group• All but one site was shut down to reduce costs – all operations consolidated into the single most modern facility in Mississippi• One year mandate to improve business performance imposed
  • 16. Opportunities• Product Waste and Rework: $272,850/month ($3.27M/year) – Waste costs $245K for every 1% OR $600/minute generated – Rework costs $35K for every 1% of rework per total product produced• Stressed employees under the gun to cut costs and improve quality and efficiency – perceived threat of losing their high paying jobs• Perceptions of employee satisfaction and work engagement were, at best, mediocre – improvement needed to move forward• Few immediate managers possessed sufficient training, development or experience in leadership, coaching and supervision
  • 17. Opportunities• Immediate managers play a pivotal role in creating motivational work environments that positively impact employee satisfaction• Employee satisfaction is related to work engagement; psychological state characterized by absorption, dedication, vigor• Increased work engagement is associated with improved job performance, business impact and profitability Confidence (I can) and commitment (I will) Performance = Ability + Motivation + Opportunity Training, education What the organization and experience provides to employees Companies with world-class engagement have 3.9 times the Earnings per Share growth rate compared with organizations with lower engagement in their same industry. (Source: http://www.gallup.com)
  • 18. InterventionPurpose: To prepare immediate managers to more effectively create and sustain motivational work environments that positively impact work engagement and organizational performance.
  • 19. InterventionPurpose: To prepare immediate managers to more effectively createand sustain motivational work environments that positively impactwork engagement and organizational performance. 5-Skills of Savvy Managers (Flagello & Dugas, 2009): 1. Self-managing refers to the practice of continuous self- improvement through purposeful self-observation and - monitoring, self-assessment, goal-setting, and conscious action 2. Reflecting is the simple practice of quietly contemplating, thinking, and/or observing…without judgment 3. Acting consciously is the practice of deliberately and intentionally selecting from feasible options decisions that are better aligned with desired outcomes 4. Collaborating is working with the full involvement of people in order to better align efforts, add value and generate results 5. Evolving is a personal, life-long commitment to the deliberate and continuous pursuit of learning, development and professional growth
  • 20. InterventionPurpose: To prepare immediate managers to more effectively create and sustain motivational work environments that positively impact work engagement and organizational performance. • Impact Measures: • Work Engagement: Test/Control Group Arrangement – 32 line employees from Production unit (test) – 31 line employees from Maintenance unit (control) – Utrecht Work Engagement Scale (Schaufeli & Bakker, 2003) • Production business unit controllable Waste and Rework – Waste costs $245K for every 1% of waste per total product produced OR $600/minute generated – Rework costs $35K for every 1% of rework per total product produced
  • 21. EvaluationPhillips ROI Methodology® The Levels of Evaluation Framework 0. Inputs & Indicators: Measures inputs, including indicators of volume and scope 1. Reaction & Planned Action: Measures satisfaction with the program and planned actions 2. Learning: Measures changes in participants’ knowledge, skills, abilities and attitudes 3. Application: Measures changes in behavior 4. Impact: Measures changes in critical outcomes 5. ROI: Compares program benefits to costs
  • 22. NeedsProgram Needs• Product Waste ($245K/1%) and Rework ($35K/1%) costs $272,850/month ($3.27M/year)• Stressed work environment barrier to Work Engagement• Immediate managers unskilled and inexperiencedEvaluation Needs• Effectiveness of learning intervention?• Increase Work Engagement?• Reduce Product Waste and Rework?• Rollout Decision?
  • 23. Program ObjectivesReaction (1)• Relevance, importance and intent to use on the jobLearning (2)• Successful application of 5 self-coaching skills• Can create and sustain motivational environmentOn-the-Job Application (3)• Effectively, continuously apply 5 self-coaching skills• Create and sustain motivational work environmentBusiness Impact (4)• Reduce % Product Waste and Rework• Increase Work EngagementROI (5): 15% Target ROI
  • 24. Data Collection PlanReaction (1)• Reaction questionnaires taken at the end of each of seven sessionsLearning (2)• Pre- and post program skill self-assessment profiles• Facilitator assessment of participant discussions, responses to questions, and completed assignments during the program• Utrecht Work Engagement Scale at program start, middle and endOn-the-Job Application (3)• Skill self-assessment profiles (3-months)• Utrecht Work Engagement Scale (3-months)Business Impact (4)• Percent Product Waste and Rework• Utrecht Work Engagement Scale (6-months)
  • 25. Data/ROI Analysis Plan Monetary Data Items Isolation of Effects ConversionPercent Waste Trend analysis Percentage of waste times $245,000 Participant and management estimatesPercent Rework Trend analysis Percentage of rework times $35,000 Participant and management estimatesWork Engagement Control Group Not converted
  • 26. ResultsReaction (1)• Content relevance (4.16/5.0, 4.0)• Importance (4.07/5.0, 4.0)• Intent to use (4.23/5.0)Learning (2)• Pre program skill self-assessment profile mean (105.0/125)• Post program skill self-assessment profile mean (107.6/125)• Utrecht Work Engagement Scale: Test group higher than control group by the end of the 90-day learning interventionOn-the-Job Application (3)• Skill self-assessment profile mean (112.6/125) at 3-months• Utrecht Work Engagement Scale: Test group higher than control
  • 27. ResultsBusiness Impact (4) $367,500 Projected Trend - $119,350 Post Average = $248,150 per month $248,150 X 12 = $2,977,800 per year less cost for Waste and Rework 0.50 Impact Estimate X 0.85 Confidence X $2,977,800 = $1,265,565ROI (5) Net Program Benefit/Program Costs X 100 ($1,265,565 - $253,761)/$253,761 = 3.99 X 100 = 399%Intangibles• Increased Work Engagement• Employee Satisfaction• Improved Teamwork and Communications• Better Decision-making
  • 28. ResourcesMeasuring ROI in Learning & Development: Case Studies from Global Organizations. Editors: Patricia Pulliam Phillips, Ph.D. and Jack J. Phillips, Ph.D. ASTD Press, Alexandria, VA , 2012.The Savvy Manager: 5 skills that Drive Optimal Performance. Jane R. Flagello, Ph.D. and Sandra B. Dugas, Ph.D. ASTD Press, Alexandria, VA, 2009.The Value of Learning: How Organizations Capture Value and ROI and Translate them into Support, Improvement, and Funds. Patricia Pulliam Phillips, Ph.D., Jack J. Phillips, Ph.D. Pfeiffer, San Francisco, CA, 2007.
  • 29. ResourcesThe ROI Institute is the leading global resource on research, training, and networking for practitioners of the Phillips ROI Methodology™ Web: http://roiinstitute.net/  Email: info@roiinstitute.net Phone: 205-678-8101  Fax: 205-678-8102The Jack and Patti Phillips Workplace Learning and Performance Institute, located on The University of Southern Mississippi Gulf Park Campus, serves as a research and outreach center for best practices in workplace learning and performance. Web: http://www.usm.edu/gulfcoast/ Email: s.robinson@usm.edu Phone: 228-214-3517
  • 30. ResourcesThe University of Southern Mississippi offers the following programs on the USM Gulf Park Campus in executive format, designed for working professionals:  Professional Development Certificate in Training and Development  Master of Science in Workforce Training and Development  Doctor of Philosophy in Human Capital Development Web: http://www.usm.edu/gulfcoast/ Email: s.robinson@usm.edu Phone: 228-214-3517
  • 31. ConclusionThe purpose of this discussion was to explore theelements of effective program evaluation  Why do we need evaluation?  What are your biggest problems with evaluation?  How can evaluation planning help?  What are the components of effective evaluation?  Can you give us an example? (PolyWrighton case study)  Where can we find additional resources?
  • 32. Putting Program Evaluation to Work for You A Learning Intervention for Work Engagement PolyWrighton, USA John Kmiec, PhD, CRP The University of Southern Mississippi john.kmiec@usm.edu 228.365.2559