Recommended
Recommended
More Related Content
Similar to Evaluate the Effectiveness of Your Online Learning & Training Programs
Similar to Evaluate the Effectiveness of Your Online Learning & Training Programs (20)
Recently uploaded
Recently uploaded (20)
Evaluate the Effectiveness of Your Online Learning & Training Programs
- 1. PRESENTER NAME Presenter Job Title Here Evaluating the Effectiveness of Your Online Learning & Training Programs Presented with Paula Yunker of Limestone Learning
- 2. Housekeeping 2 1. Sound & Audio 2. Poll Questions 3. Q&A 4. Presentation Recording & Slides 5. Post-Webinar Satisfaction Survey
- 3. Your Hosts 3 Paula Yunker Managing Partner & Instructional Designer Nimritta Parmar Marketing Coordinator
- 6. Poll: How important do you believe learning evaluation is? 6 • Not at all • Somewhat important • Important • Very Important • I’m not sure
- 7. Poll: To what degree does your organization use evaluation? 7 • Not at all • Occasionally • Most of the time • All of the time • I’m not sure
- 8. Evaluating the Effectiveness of Your Learning & Training Programs
- 9. Why is Evaluation Important? Training should have a purpose: • To inform • To teach • To improve change Evaluation will identify if learning has occurred, and if improvements have been made 9
- 10. Aligning Learning to Organization Goals • For training to be valued by leaders and stakeholders, all training efforts need to focus on the organizational goals • Consider what your stakeholders are expecting, and the impact on business training needs to have to meet expectation 10 Training Business Goals
- 11. Strategies for Aligning Learning 11 • Step 1: Identify the business need • Step 2: Create the learning plan • Step 3: Measure the business results
- 12. 12 Ask your key stakeholders and business partners: • What are the desired results? • What are the goals & outcomes? • What needs to change to meet these goals & business outcomes? Step 1: Identify the Business Need
- 13. 13 • What KPI’s should change? • What needs to be learned to make those changes happen? • What will success look like to you? Step 1: Identify the Business Need (cont.)
- 14. 14 Develop a learning and evaluation plan to meet the learning requirements based on your answers to Step 1: • What needs to be learned to achieve business outcomes? • What required KPI’s are needed? • What learning experiences and practice are needed to become proficient? Step 2: Create the Learning Plan
- 15. 15 • When and how will these learning experiences and practices be best accomplished? • How and when will the learning be measured? • How will learning be applied to the job, and how will that application be reinforced on the job? Step 2: Create the Learning Plan (cont.)
- 16. 16 To identify how learning aligns to business outcomes, and meets stakeholder/business partners needs: • What changes in business results & key performance indicators have been realized from the training program? Step 3: Measure the Business Results
- 17. 17 • For every dollar invested in training, how many dollars does the organization get in return? Is the ROI significant enough to justify training? • What key intangible benefits (soft data) have been gained? Step 3: Measure the Business Results (cont.)
- 18. 18 Where do you need to be with data & analytics, and how do you get there? Take a BIG PICTURE view: Stakeholders want to know if programs have helped make money, save money, or avoid costs. This brings us to … DATA
- 19. Start by collecting data easy to obtain 19 • System logins by department – are there some departments or organization groups that login significantly less than others? • Course completions by department – are there any learners not completing courses? • Pre and post test and quiz results
- 20. Now consider the tougher questions.. 20 How can additional data be helpful? Calculating Success Measures (using sales training as an example): 1. Measure for each sales rep, if possible 2. Isolate variables that effect revenue 3. Corroborate pre-test, test, and retention quizzes 4. If knowledge is retained, proceed. 5. Wait 3 months to compare sales 6. Isolate data for considerations of seasonality, product, etc.
- 21. Let’s step back – Typical Challenges 21 • Lack of buy-in from the top • No ownership of the project • Inability, or unwillingness to measure and track evaluation on the job • Lack of reinforcement on the job
- 22. Overcome the Challenges: Implementation Strategies 22 What’s needed to gather & measure data, and evaluate the results? • Expertise • Resources • Organizational Support
- 23. Implementation Strategies 23 Evaluation tools that support and measure learning on the job • Create resources that reinforce learning (job aids, reference handouts, etc.) • Create follow-up questions • Ask similar questions to managers and supervisors, internal clients, and customers to get a fuller picture
- 24. Implementation Strategies 24 Learning Action Plan • A contract between the learner and manager, that provides structure to assist managers and employees to put learning into practice • Gives learners the opportunity to reflect on their learning experience • Signed by both parties, and provides accountability
- 25. Poll: Does your organization use some form of Learning Action Plan? 25 • No • Yes • I’m not sure
- 26. Overcome your Challenges 26 Before Learning: • Create pre-evaluation that is directly aligned to learning content, and the learning objectives. • Examples: self-assessment surveys, written pre-tests, blog/wiki questions, reading assignments
- 27. Overcome the Challenges: Implementation Strategies 27 During Learning: • Design formative evaluation tools, and place in strategic spots throughout content. • Examples: team exercises, brainstorming, flipchart discussions, learning games, reflective exercises After Learning: • Design an end of course summative evaluation to evaluate how successful each learner is at meeting learning objectives
- 28. Evaluation Best Practices & Tips 28 • Make sure you’re measuring learner satisfaction AND engagement/relevance to the job • Incorporate statements/questions that are learner centric • Consider using a 7, 9, or 11 point rating scale to provide a richer level of feedback • Don’t jump to conclusions!
- 29. Recommended Resources 29 • Evaluation websites • Books/reference material • Kirkpatrick Evaluation LinkedIn group • Training and certification • Industry and best practice research sites • Limestone Learning blog
- 30. Questions? 30
- 31. Follow-up 31 • You will receive the webinar recording & slides • Post-webinar satisfaction survey • Further questions – don’t hesitate to contact us.
- 32. TOLL FREE +1.888.463.8489 EMAIL pyunker@limestonelearning.com WWW LIMESTONELEARNING.COM
- 33. TOLL FREE +1.877.700.1118 EMAIL SALES@LAMBDASOLUTIONS.NET WWW LAMBDASOLUTIONS.NET
Editor's Notes
- I’d like to introduce you to our presenter - Paula Yunker. Paula is the Managing Partner and Instructional Designer at Limestone Learning. She has taken time out of her busy schedule to be here with us today, so thank you Paula! I’m Nimritta, I’m a marketing coordinator here at Lambda Solutions, and I will be co-presenting, as well as managing any questions you might have throughout the presentation.
- About Limestone
- About Lambda A little bit about Lambda Solutions: We are LMS experts and a complete solutions provider – we offer everything from the technology, to training and services. Beyond the technology, our solutions help you see when, and how, eLearning is achieving goals. We’ve developed a reporting and analytics solution to provide deeper insight into your learners, and learning programs. We have 12 years of experience, over 600 customer implementations, and 1 million active user accounts across most industry sectors. These are just some of our amazing industry leading clients
- Measuring the value to an organization is a complex but important component of any training course or program. Training should have a purpose – to inform, to teach, to improve and change. Introducing evaluation and tying it to performance will identify if learning has occurred during and after training and if job performance improvements have been realized. Measuring improvements in job performance will also provide valuable data for measuring cost-benefits to the organization.
- A key requirement for any organization is to ensure training and learning aligns with organizational goals · In best practice companies, all training programs cascade down from the overall strategic goals. · No programs are developed and implemented unless they produce results that are identified as critical to the organization. · There should be explicit alignment between training programs, learning objectives, and business outcomes – the high level organizational initiatives. · For training to be valued by organizational leaders, all training efforts need to focus on the organizational goals. · For each training initiative, begin by considering what your stakeholder and business partner/business unit expectations are and what impact on the business you want the training to have. · Stakeholders and business partners can provide you with valuable information about what business results are needed (“What will success look like to them?”), why they are important, how they align with organizational goals, and what business measurements (if any) currently exist in their departments or business areas. · The information can then be used when planning and designing new training initiatives.
- How do you ensure training aligns with business outcomes? Step 1 is to identify the business need
- To identify the business need, ask your stakeholders and business partners the following questions: o What desired results from an organizational perspective do you want to realize from training? o What are the goals and business outcomes of the organization? o What needs to change within the organization/ business unit(s) to meet those goals and business outcomes?
- What key performance indicators (or KPIs) should change? o Some examples are sales forecasts, customer satisfaction, employee turnover, operational costs, safety performance and inventory. o What needs to be learned to make those changes happen? o What will success look like to you?
- Step 2 is to Create the Learning Plan Once you understand the business need, develop a training and evaluation plan to meet the learning requirements associated with that need based on the answers to the following questions: o What will need to be learned to achieve the business outcomes? o What required knowledge, skills and attitude (KPAs) are needed? o What type of learning experiences and practice are needed to become proficient?
- When and how will these learning experiences and practice be best accomplished? o How and when will the learning be measured? o How will learning be applied to the job and how will that application be reinforced on the job?
- Step 3 is to Measure the Business Results Many organizations don’t complete this third step. · If you start with the company's business outcomes, you should also end with the company's business outcomes and how the programs you design and deliver affect them. · To identify how learning aligns to business outcomes and meets the needs of stakeholders and business partners, you should find the answers to the following questions: o What changes in business results and key performance indicators have been realized from the learning program?
- For every dollar invested in training how many dollars does the organization get in return? Is the ROI significant enough to justify training? o What key intangible benefits (soft data) have been gained?
- This brings us to data. The data you collect and analyze will begin to paint a picture of where you need to improve, highlight data-need gaps, and prove indicators on how improvements can be made. The important thing to remember here is Take a Big Picture View – Your data should provide insight into how company wide learning/training programs performing? Stakeholders want to know if programs have helped make money, save money, or avoid costs. First, Make money – This can be through sales training, which can come in the form of product knowledge courses, or sales tactic training which help sales improve performance via soft skills. Save money – This can be process driven training that helps employees complete tasks faster, or provide strategies to cut out inefficiencies. Eg. Manufacturing company that educates workers on how they can save material and machinery depreciation costs by wasting less raw material, and taking better care of equipment Avoid costs - Falls under compliance training, such as workplace harassment training, or health & safety training. There are hefty regulatory fines that can come down on any organization if they aren’t meeting compliance standards, especially for those in the healthcare sector.
- You can start by collecting the data that is easy to obtain: For example, system logins by department - are there some departments or organization groups that login significantly less than others? - Course completions by department - are there any learners not completing courses? - Pre and post test/quiz results by department This will provide data to see if training is actually being used. If it is, you can drill deeper into the effectiveness of training in various departments.
- Now consider the tougher questions, and how additional data might be helpful. For this deeper data, you will need to work closer with departmental managers to understand how training has impacted the bottom line. How do we calculate success measures? Sales training is one way to understand how training has impacted revenue, so we are going to use that as an example. Measure for each sales rep if possible Isolate variables that also affect revenues Seasonality, increase in lead generation, new talent, product feature changes. Corroborate pre-test, test, and retention quizzes If knowledge is retained, proceed Wait 3 months to compare sales You can isolate data for considerations of seasonality, product, or feature changes, new talent on the workforce, etc. This general structure can be used to determine how training focused on saving money and avoiding costs actually contributes to the bottom line. To do this, you will have to isolate any variables that have any impact on sales, other than training specifically. This not only helps prove effectiveness of the course, but the content of course - “I designed a course to do this, this proves that it is doing that.”
- Lack of buy-in from the top/ commitment from appropriate stakeholders. o Effective evaluation requires commitment and resources to get it established. o The necessity to track data and information beyond the ‘training room’ environment means that employees across an organization must understand the importance and value of measuring different kinds of results. o Without the voice of committed leadership from the top of an organization, as well as the action to support the efforts, it’s extremely difficult to be successful. o Leadership must be ‘sold’ on the benefits the organization will reap from evaluation in order to get their commitment to make it happen. · Ownership o Who’s going to be responsible for creating, administering, measuring and gathering the evaluation data? o The ability to measure and track performance and the value it brings to an organization are both factors that reach beyond the boundaries of a Learning and Development team or team member. o Often there seems to be no clear ownership of the process. · Unable/unwilling to accurately measure and track evaluation on-the-job o Supervisors and managers often lack the time, resources, commitment and knowledge to measure and track employee performance on the job. o However their participation is crucial for a successful evaluation program. · Lack of reinforcement on the job o Robert O. Brinkerhoff, author and an internationally recognized expert in evaluation and training effectiveness, shows that only 15 percent of what is learned during training will be applied on the job if it is not reinforced and monitored. Creating a post-training support structure as part of the initial training design and development process increases the likelihood that resources will produce a measurable increase in performance. o Providing excellent training does not lead to significant transfer of learning to behaviour and subsequent results without a good deal of deliberate and consistent reinforcement back on the job.
- Substantial levels of investment, expertise, resources and organizational support are required to gather and measure data and evaluate results · It’s important to first identify which training course or program has the greatest impact to the organization and how important the need is to demonstrate the value of learning. · Involve stakeholders and business partners in this decision making process. · One very important decision to make is who will be responsible for gathering and analysing data and measuring the impact of learning on the organization. · Hard and soft data gathering should begin before the training event occurs to create a baseline.
- Think of the learning event as an ongoing process rather than as a one-time event. What do the learners need not only during training but once they’re back on the job. · Build your post-training activities during your program design. · The evaluation tools you design should answer the question “Is training working?” rather than “Did training work?” · Involve managers before you begin to design your evaluation · Find out what effective job performance means to them. Explain how they can effectively support their employees back on the job and identify what tools and support the managers will need. · Then write your learning objectives to align with what the learner will be able to do after training…”At the end of the training session, learners will be able to effectively facilitate a project kick-off meeting.” Or “At the end of the training session, the learners will accurately enter a customer record into the client registration system.” · There are a wide variety of level 3 evaluation tools that can be used to support and measure learning on the job. o Create job aids, reference handouts, and informal video clips of processes as reminders – whatever is needed to reinforce learning. o Create follow-up questions that the learners answer. This can be an efficient and effective way to transition from level 2 into level 3 (and potentially level 4) evaluation. o Learners are asked the questions individually or in group interviews or in an online survey with learners approximately three months post training. Make the questions learner centric. o To more effectively measure changes in behaviour on the job, ask similar questions to the learners’ managers, supervisors, internal clients, customers and business partners to get a fuller picture.
- o The other one is a Learning Action Plan which is used prior to training, at the end of training and back on the job after training. o It’s a contract between the learner and their supervisor or manager that details what will be learned, what has been learned, how the learning will be implemented and used on-the-job and what actions the manager can take to support their employee back on-the-job. o Learners are also given the opportunity to reflect on their learning experience. o The Learning Action Plan is signed by both parties and provides accountability. · The learning action plan also provides the structure to assist managers and employees to put learning into practice, immediately reinforcing new knowledge and skills post-training and improving the likelihood of training being successfully transferred on the job. · Learning Action Plans are common in best practice organizations.
- Before learning: · Create pre-evaluation that is directly aligns to learning content and the learning objectives. · Some examples of pre-evaluation are self-assessment surveys (before and after), written pre-tests, blog or wiki questions and reading assignments with questions.
- During learning: · Design a variety of formative evaluation tools and place in strategic spots throughout your content. There should be at least one formative evaluation per lesson since each lesson should have at least one learning objective. · Some examples of formative evaluation tools are team exercises, brainstorming, flipchart discussions, informal surveys, learning games, reflective exercises, reflection questions to answer, etc. After learning: · Design an end of course summative evaluation to evaluate how successful each learner meets the learning objectives. · Summative evaluations are completed individually and typically marked using a checklist or rating scale. · A checklist measures quantity – how many questions the learner correctly answered or what tasks they correctly completed · A rating scale measures quantity AND quality – not only to measure if the learner can achieve the task/activity but how well can they do it. · Again, there should be at least one question for each objective in the course.
- Measure not only learner satisfaction but their level of engagement and relevance to the job. · Focus less on the room comfort and whether they were happy with the muffins provided and more on the learning experience. · The goal is to incorporate statements or questions that focus the learner on higher levels of evaluation and get them thinking about how the new learning will benefit them and the organization after training has occurred. · For example - make all evaluation statements learner-centered. For example, rather than “The facilitator provided debrief activities for participants to demonstrate their learning”, instead use “The debrief activities helped me to effectively practice what I learned”. · For example, the content was relevant to me and to my job or I estimate that I will apply the following percent of the knowledge/skills learned from this training directly to my job. Then provide a % scale from 0% to 100% in increments of 10. · Consider using a 7, 9, or 11 point rating scale to provide a richer level of feedback. Only label each end of the rating scale, rather than labeling each number on the scale (e.g., 1=strongly disagree and 7=strongly agree). · A cautionary note – just because learner ratings were not high on the level 1 evaluation does not mean that they did not learn! · So use the results carefully and don’t jump to conclusions. · That’s why it’s important to gather more data at different evaluation levels to get a more rounded and truer picture of the learning and its impact.
- Paula has put together some learning evaluation resources for everyone that I will send out in the follow up email.