Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measuring Learning Effectiveness: Beyond Training Happy Sheets & Memory Recall

540 views

Published on

Despite Learning measurement frameworks like Kirkpatrick and Philips ROI that have been around for so many years, defining meaningful measures still continues to be a big challenge for L&D. ‘E’ has become the most ignored in ADDIE.
We talk to an experienced panel about the unsatisfactory current state and what needs to be done to improve the importance, visibility and value of measuring Learning effectiveness in the real world. We focus on how to collect actionable insights about learning effectiveness: not only seeking to answer, did learning achieve the desired or intended result, but also why/why not.

Published in: Business
  • Be the first to comment

Measuring Learning Effectiveness: Beyond Training Happy Sheets & Memory Recall

  1. 1. 1 Thurs, April 14th, 2016 12 -1 pm AEST Ways to participate: • Q&A Box - comment, whinge & opinions • Twitter Backchannel - #lrncafe Knowledge Sharing Better Practices Experienced Panel
  2. 2. Introductions AbbVie ABC AGL AMP ANZ Bank ARCA Australian Institute of management Australian Red Cross Bank Australia Bankwest Blackboard BNZ BT Financial Group Cancer Council Queensland CBA CBH Group Coca-Cola Amatil Concordia University CUB PTY LTD Degreed Downer Energy Australia Ericsson Ernst & Young FACS FedEx Express Ford-Werke GmbH HCF IAG IMC IP Australia Justine Jardine Lynda Moe Transformations Manipal Global Ed Serv McGrathNiccol Melbourne Business School NAB NATA Novartis Presence of IT Primary Health Care Qantas Qantas Credit Union Qatar Airways QBE Queensland Public Services RACP Recoveriescorp Rio Tinto Shine Laywers Siemens Ltd Simplot Skillsoft SMS Management & Tech St John Ambulance WA Suncorp Telstra Toyota TP3 University of Sydney Westpac WVI Attendees From140+ Vanessa Blewitt Learning Effectiveness Lead, Corporate Training and Learning – Nestle Will Thalheimer President – Work-Learning Research, Inc Jeevan Joshi Producer – LearningCafe
  3. 3. Blog Magazine Webinar Discussions UnConference Twitter Linkedin Facebook Coffee Catch Ups Capability Building Workshops Community of Learning Professionals with a focus on implementing ideas Building L&D Capability
  4. 4. Webinar Next Steps Upcoming Webinars Recorded Webinars UnConference Please subscribe
  5. 5. Next Webinar - VR in Learning – Has its time come ? http://bit.ly/lcwebmay16 • State of VR (Consumer & business) • Use cases for business • How to develop a VR Learning solution • Opportunities and challenges Thurs May 12th, 2016 12 to 1 PM Kain Tietzel Founding CEO –Start Plus others Why you should attend • 2016 is supposed to be year of VR • L&D likely to asked about it. • VR Learning is being built. • Stay ahead of curve.
  6. 6. Melbourne UnConference – Aug 18, 2016 Limited to 100. Tickets available. Sydney was sold out http://www.learningcafe.com.au/events Tony Gleeson CEO - AIRAH Julie Catanach Exec Manager Performance Suncorp Group Peter Hall Head L&D QBE - APAC David Le Page Comm Mgr - Melbourne Andrew Wright Worldwide Intranet Challenge
  7. 7. Ideasatwork.com.au Collaborative explorations by Australian organisations facilitated by LearningCafe Starting Soon TinCan/xAPI MOOCs Content Curation for Learning ? (Too soon) Yes
  8. 8. Monthly Observations in Australian L&D* •Significant restructuring in HR/ L&D. • Head count reductions • Getting new skills in the team. •Gap in Business Acumen in L&D, a real worry. •Major LMS & HRMS not keeping pace with flexibility and UX. • Number of point cloud solutions coming into the market . •More focus on hard skills / business Learning. • Less on soft skills. * Opinions based on discussions and interviews with senior L&D professionals. Not survey involved.
  9. 9. Webinar Dashboard Start Up/ Experiment Organisation / Industry Adoption Why Bother ? Remove Hype Maturity Business Value Low Low High High Now Novice Explorer Practitioner Expert Get Answers Make Sense Looking for Answers Expertise Level Next Steps ? Not Much A Lot Vision LearningCafe Expertise Run workshops for L&D teams for • Leading Australian Bank • Leading Insurance Company Consulting on architecture for • Professional services firm Covered in UnConference 2015 + 2016
  10. 10. Questions from you • What are 3 tips in gathering valuable data/statistics? • How do you measure accurately if for e.g. only 100 out of 500 people respond to a survey? When should you make assumptions from the data? • Like quality, learning effectiveness is also a perception in the minds of key stakeholders (and influenced by their expectations). How do we best accommodate this reality in our measurement strategy?“ • Value in blended learning design - at the end, throughout, at the start or combination of?
  11. 11. HELICOPTER VIEW Jeevan Joshi Producer – LearningCafe & Digital Learning Adviser – Fairfax Media
  12. 12. PRIMER Analysis Design Development Implementation Evaluation ADDIE OK Complex Requires more time & effort Difficult Requires time & effort * http://www.gosignmeup.com/kirkpatrick-phillips-model/ Kirkpatrick Model of Evaluation* WE HAVE A PROBLEM
  13. 13. PROBLEMS Too difficult to isolate impact of Learning Evaluation is not standardized enough Not enough budget and skills Business moves fast and loses interest.
  14. 14. MORE PROBLEMS Rise of Informal Learning Speed of Business in Increasing L&D Budget Likely to Decrease Employee engagement is decreasing
  15. 15. TinCan/xAPI@Work www.tincanatwork.com 17 Intranet – Videos/podcasts Discussion Boards (COP)(yammer) Wiki (Aussie /prod info/Wiki/IT) On job activities (Perf Management) Discussions Team meetings (learning component) Other business systems Online Courses on LMS + Webinar Classroom Courses/ Online Coaching On Job Activities (competence demo) Assessments/surveys (e.g. survey monkey) Simulation Performance Support Tool Secondment Social Media YouTube Facebook Website – HBR Any web resource External COP Coffee conversations MOOCs External courses Formal Education (Degrees) Formal Collaborative Program with Unis Memberships/prof bodies Formal mentoring programs Certifications/exams Formality Sources Internal Informal External Formal
  16. 16. xAPI/ TinCan – Solution ? Layer 1 • Modern version of SCORM Layer 2 • Record any learning experience Layer 3 • Frees data from LMS Layer 4 • Correlate job performance and training data http://tincanapi.com/the-layers-of-tin-can/ Happening Aspirational Technical Strategic Low readiness Maturity /capability www.tincanatwork.com
  17. 17. ORGANISATIONAL REALITIES Vanessa Blewitt - Learning Effectiveness Lead, Corporate Training and Learning Nestle
  18. 18. Deciding to Measure Learning Effectiveness What is the value of T&L investments? Executive Board • That was the easy part!
  19. 19. Starting Point: T&L as a “HR Thing”, an Event • < 1 in 10 learners discuss expected outcomes with line manager. • Alignment to business not clear. • Alignment to learner development plan low. • Key measure =  • No on the job follow- up or support. • Action plans un- actioned. • No measures of what happened…or why. • “Black box”. Before During After Look Familiar?
  20. 20. Defining “Value” Guiding Principle: In any transaction, the supplier determines the price, but the purchaser determines the value. In order to measure value, first need to create conditions for value. Tools, Process, Measures Design Delivery What is the value of T&L investments? Collect Nestle Insights Conduct Industry Research Develop Learning Effectiveness Framework
  21. 21. Defining Learning Effectiveness Measures: Design Design With the business for the business Business Level Measures WHAT: What is expected? What does success look like?  How will you recognize it?  Why this learning…target which business objectives & priorities?  What will you reflect on in 6 months time to decide if learning was worthwhile/ added value?  Expected outcomes? … observable / measurable? KPI impacts? Trends? Feedback? What should CHANGE? WORKSHOP Senior Stakeholders & Alumni H O W & W H O INTAKE FORM Senior Stakeholders & Alumni INTERVIEW Reflections & Feedback Alumni @+200 days
  22. 22. Defining Learning Effectiveness Measures: Delivery Delivery For on the job performance change Learner Level Measures WHAT: What is expected? What happened? WHY: Why did it happen?  Learning Alignment • To development plan & objectives  Learning Impact on business • Expected outcomes e.g. KPIs  Learner Development • On the job behaviors – do & say  Learning Application • Action Plan status • Opportunities & Obstacles  Learner Experience • Confidence • Commitment • Net Promoter Score H O W & W H O Surveys Learner with Line Manager Option: HRBP Facilitation @ Before @After Surveys Learner @Nomination @During No survey is called “survey” Learner centric questions
  23. 23. Importance of Why Measures Example of Actionable Insights WHAT Happened? • Proficiency Gap Closure = 50% v Expectation (Target) of 80%. WHY Did it happen? • Action Plan Status = 60% Fully or Mostly Implemented. • Obstacles to apply learning on the job = 70% learners reported higher/competing priorities. WHAT Action to take? • Current Cohort: Targeted business escalation, coaching. • Future: Increase business to business communication re importance of application to achieve business and career success, learn & leverage from strength business areas, add how to address higher/competing priorities to learning experience. Target Set benchmarks & change variances Track Trends Over time. Avoid “knee-jerk” reactions Tell Tailor to stakeholders Apply “So What?!” Filter
  24. 24. Challenges: Organisation & Process, Data, Enablers Organisation • Identifying right stakeholders & getting them to the table. • Defining what success looks like in a way that is tangible & linked to business. • Defining objectives to support learning effectiveness measures (finite, verb, scalable for proficiency levels). Process, Data, Enablers • Data access & quality for analytics. • Sophistication of requirements v enabler capabilities. • Managing surveys & reporting > timeliness, quality, defining use cases. • Scalability.
  25. 25. Challenges: Climate & Culture Climate & Culture • Survey response rates (L&D to business). • Training as reward & recognition. • Fear of “why” measures (e.g. manager support rate). • Learners: Lack accountability for self-development. “Develop me.” “I was told to come”. • Line Managers: T&L = . Develop people not include in bonus. “ I don’t have time”. • HRBPs: Operational focus. “Not My Job”, “Too hard”, “Good training gets applied”, “Why measure?”. • L&D: Event focused. Self-contained. Service oriented. Uncomfortable to challenge business. “Not what I’m asked for”, “No influence beyond event”, “What am I supposed to do with this?” “What if I’m told to “fix it?”” • ALL: OVERWORKED. OVERWHELMED. CHANGE RESISTANT.
  26. 26. Top 10 Key Learnings WHAT “as expected?” !As defined by stakeholders! WHY “why / why not?” Act to achieve / exceed “as expected” !As defined by stakeholders! Effective learning effectiveness measures 1. “What does success look like?” is a hard question to answer! ASK! Don’t ass-u-me. 2. “Fear” of what measures will reveal & accountability for action 3. Required culture change in L&D as much as business 4. Use inspiring facts & stories. Shine a light: “look familiar?” 5. Appetite exists & can be built. Don’t push. Leverage multiple channels to “build the army” of collaborators & early adopters. 6. Engaging stakeholders increases stakeholder engagement & accountability 7. Respectful challenge is well-received, build & helps earn a seat at the table 8. Alignment & behaviour change more valued than ROI 9. Focus on HOW more than WHAT 10. Long process of CHANGE. Be the tortoise. +1 Start small & focussed
  27. 27. STRONG FOUNDATIONS Will Thalheimer President – Work-Learning Research, Inc
  28. 28. Questions from you • What are 3 tips in gathering valuable data/statistics? • How do you measure accurately if for e.g. only 100 out of 500 people respond to a survey? When should you make assumptions from the data? • Like quality, learning effectiveness is also a perception in the minds of key stakeholders (and influenced by their expectations). How do we best accommodate this reality in our measurement strategy?“ • Value in blended learning design - at the end, throughout, at the start or combination of?
  29. 29. www.learningcafe.com.au lrncafe http://bit.ly/lcafefb blogs learning conversations free resources workshops UnConference 2016 Sydney Melbourne Webinar recording, ebooks, L&D frameworks Building Effective Employee Social Networks 33 Ideas@work Collaborations
  30. 30. Next Steps Join Special Interest Community Attend Workshop Content Curation & Dissemination for Learning Attend UnConference Melbourne Brisbane LearningCafe LinkedIn Subgroup Register interest www.learningcafe.com.au Register interest www.learningcafe.com.au Or send us an email - enquiry@learningcafe.com.au
  31. 31. Ways to participate Q&A Box - comments, gripes & opinion Twitter Backchannel - #lrncafe Webinar Feedback – http://bit.ly/lcrate1 Session Recorded
  32. 32. Next Steps Join Special Interest Community Attend Workshops Attend UnConference Melbourne Brisbane LearningCafe LinkedIn Subgroup Register interest www.learningcafe.com.au Register interest www.learningcafe.com.au Or send us an email - enquiry@learningcafe.com.au

×