Presented at ASAE's Spark: The Art and Science of Learning Conference 2.14.18 by Noelle Akins, Senior Consultant, Navigator Management Partners Can you demonstrate the value of your association's learning program to your target audience? Examine the measures that matter and how to use this data to inform future programming. Discuss key data points used to evaluate training effectiveness at five levels. Determine how, where, and when to capture evaluation data for all five levels in real-life scenarios and how to apply that data to improve future learning outcomes for your association.

1. Measures that Matter
How to capture and communicate the
value of your learning programs
Noelle K. Akins, L.P.C., Navigator Management Partners
February 14, 2018

2. Measures that Matter
“Not everything that can be counted counts,
and not everything that counts can be counted.”
– ascribed to Albert Einstein

3. Introductions
This is me skiing in Germany 
• 15+ years managing and evaluating learning programs
• 10+ years consulting with global, non-profit & higher ed orgs
• Capabilities include leadership development, performance
consulting, customer service, and technical training
In chat, tell us…
• Your name, role and location
• The role evaluation plays in your
learning programs today

4. Learning Objectives
• Identify five levels of “matterful” learning evaluation
• Recognize how, when and where to gather data for each level
• Apply evaluation data to improve future programming and
support organizational outcomes

5. Leading and Lagging Indicators

6. 5 Levels of Evaluation
Based on Kirkpatrick’s 4 Levels / ROI added by Phillips
Reaction
Learning
Behavior
Results
ROI
L1
L2
L3
L4
L5

7. Use of the 5 Levels of Evaluation
4
18
37
55
81
92
4
15
35
60
83
89
0 10 20 30 40 50 60 70 80 90 100
None
Level 5: ROI
Level 4: Results
Level 3: Behavior
Level 2: Learning
Level 1: Reaction
2015 2009
Please select all the levels of evaluation you use to any extent in your organization
*From 2016 ATD Research Report Evaluating Learning: Getting to Measures that Matter

8. IgnitePoll: Driving for the Data
Identify your Association’s top two challenges related to current evaluation
efforts:
A. Aligning learning outcomes and organizational outcomes
B. Determining predefined evaluation points during the design phase
C. Evaluating ROI/ROE or business results
D. Measuring on-the-job behavior change
E. Assessing participant learning
F. Other (please share):

9. A Day in the Life of…
Tim and his team are responsible for the delivery of professional
development programs for project managers. Training delivery options
vary, from web-based ethics and compliance training to technical and
skill-based content taught in classrooms by experts.
Training participation levels are high. You hear that participants like the
classes and learners often register for future classes, but you have no
record of training impact.
Your goal: To illustrate the impact of your programs…from
participant perspectives to organizational outcomes

10. L1 Reaction
Lisa manages the web-based ILT ethics training program. How can she get
participants’ reactions on class content and the instructor?
• Online survey
• Real-time participant feedback
• Instructor observation
When: Immediately following the learning activity
Where: Conversation, Survey Monkey, LMS
Why: To get near-time feedback for immediate adjustments

11. L2 Learning
Jose is delivering web-based training on agile project management. How can he
assess learners’ knowledge of the content and their new skills?
• Knowledge checks and quizzes
• Skill-based branching scenarios
• Activities as simulations, demos and/or labs
When: At the end of key content chunks, then a summative assessment
Where: LMS tool, partner activities
Why: To confirm that content and skills have been trained and gained;
To provide an opportunity for self correction and immediate clarification

12. L3 Behavior
Now that learners have completed training for project execution and monitoring,
how can Tim gather info on participants’ on-the-job application?
• Participant self-report on key PM practices (before and after)
• Improved project execution quality: Budget vs Actuals, Timeline vs % complete
• Manager/team feedback and observation
When: 30, 60, 90 days
Where: LMS tool follow up, Survey Monkey
Why: To demonstrate real and sustained behavior change; to reinforce
usage of new skills and knowledge

13. L4 Results
How will Tim and team measure the impact of improved project
management practices at organizational levels?
• KPI comparisons: time, budget, and quality measures
• Speed to market measures
• Client satisfaction rates & renewed contracts
When: Close of all projects, annual reports
Where: Portfolio Management System, ERP system
Why: To illustrate the importance of learning as a critical enabler for the
organization

14. L5 ROI/ROE
How can the impact of professional development be measured at the
ROI/ROE level in our PM scenario?
When: Often a year out or more
Where: Sponsor interviews and financial reports
Why: To demonstrate the value of learning in financial terms
ROI =
(Gain from investment – Cost of investment)
Cost of Investment
Return on Stakeholder ExpectationsROE =

15. Beginning with The End in Mind
33

16. Outcomes Alignment: The Outcome Guide
Business Outcome
here
© 2017 The Learning Organization. www.outcomeprimers.com.
Licensed under a Creative CommonsAttribution-NonCommercial-NoDerivs 4.0 International License

17. Business Outcome
here

18. IgnitePoll: Using Evaluation Data
To what extent do you agree with the following statements?
1. Our learning evaluation efforts help us
meet our organization’s learning goals.
A. Very high extent
B. High extent
C. Moderate extent
D. Low extent
E. Very low extent
2. Our learning evaluation efforts help us
meet our organization’s business goals.
A. Very high extent
B. High extent
C. Moderate extent
D. Low extent
E. Very low extent

19. Using Evaluation Data
36
44
0 5 10 15 20 25 30 35 40 45 50
Our learning evaluation efforts help us meet our
organization's business goals
Our learning evaluation efforts help us meet our
organization's learning goals
From 2016 ATD Research Report Evaluating Learning: Getting to Measures that Matter
Percent of respondents indicating high or very high extent
To what extent do you agree with the following statements?

20. Continuous Improvement and
Course Corrections
• Content
Revisions
• Instructor
Coaching
Participant
Reaction
• Content
Revisions
• Instructor
Coaching
Knowledge and
Skills • Assessment
Revisions
• Manager
Support
OTJ Application
• Training
strategy
refinement
Business
Impact • Alignment with
organizational
outcomes
ROI/ROE

21. Communicating Results
• Activity Highlights
• Participant Testimonials
• Summative Program Reports
with the five levels

22. Project Management Training
infographic from PMAlliance
https://it.toolbox.com/blogs/tomstevens/do-you-
need-training-pm-infographic-081513
Summer Glow Training infographic
from Massage Envy
https://create.piktochart.com/output/7943928-
summer-glow-training-evaluation
Worth a Thousand Words
Share the impact of
your training
through. . .
• Infographics
• Pictures
• Videos

23. Determine What Matters
1. On a scale of 1-10 assess your current
state at each level
(1 = needs lots of improvement; 10= we rock!)
2. Circle the one level you most want to
improve this quarter
3. Write down two actions you can take to
improve that level
A. What obstacles do you anticipate?
B. Who should you collaborate with and in
what ways?
Reaction
Learning
Behavior
Results
ROI
L1
L2
L3
L4
L5

24. Ways to Learn More
• Aligning Business and Learning Outcomes for Measurable Impact. Akins, Noelle and Telban, Kathy
(2017). http://www.nxtbook.com/nxtbooks/trainingindustry/tiq_20170910/index.php?startid=21
• The Outcome Primer Series. Stiehl, Prickel, Sours, Null, Decker and Telban (2017).
Free Outcome Guide and Scoring Guide templates to assess quality of outcome statements:
http://outcomeprimers.com/templates/
• Kirkpatrick’s Four Levels of Training Evaluation. Kirkpatrick, James and Wendy (2016).
https://www.amazon.com/Kirkpatricks-Four-Levels-Training-Evaluation-ebook/dp/B01INZ4RTI/ref=sr_1_fkmr0_1?s=digital-
text&ie=UTF8&qid=1517493556&sr=1-1-fkmr0&keywords=kirkpatric%27s+four+levels+of+training+evaluation
• Evaluating Learning: Getting to Measurements That Matter. ATD Research (2016).
• Real World Training Evaluation: Navigating Common Constraints for Exceptional Results. Phillips,
Jack and Patti (2015).
• Telling training's story: Evaluation made simple, credible, and effective. Brinkerhoff, R. O. (2006).