Measuring Training Effectiveness


Published on

arning & Development business units are under siege and struggling with the effort of what appears to be a very confusing, elephantine challenge of measuring the effectiveness of their training interventions. We need clarity. We need a common sense approach. We need to step up our practice of corporate learning consulting. Let’s discuss existing principles to prove the value of Learning & Development (L&D) deliverables in a corporate environment.

Published in: Business, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Measuring Training Effectiveness

  2. 2. MEASURING TRAINING EFFECTIVENESS ~ Share This! ~ Post this to your blog, Twitter™, LinkedIn® or Delicious™ accounts or email this to someone who might enjoy it. Share Remix Attribute Share Alike 11639 E. Wethersfield Road, Scottsdale, AZ 85259 USA Toll-free: 877-614-84402 © 2011 by Michaels & Associates Docntrain, Ltd. dba Michaels & AssociatesPage Copyright holder is licensing this under the Creative Commons License, Attribution-Share Alike 3.0. For more information, check out
  3. 3. MEASURING TRAINING EFFECTIVENESSMEASURING THE EFFECTIVENESS OF TRAINING—T H E B U S I N E S S O F C O R P O R AT E L E A R N I N GLearning & Development business units are under siege and struggling with the effort of whatappears to be a very confusing, elephantine challenge of measuring the effectiveness of theirtraining interventions. We need clarity. We need a common sense approach. We need to stepup our practice of corporate learning consulting. Let’s discuss existing principles to prove thevalue of Learning & Development (L&D) deliverables in a corporate environment.In the training industry, many practitioners insist that regardless of the size or maturity (PaulKearns consultant, author, teacher at of yourorganization, the Kirkpatrick models and/or Phillips methods of evaluation are the onlysound ways to offer validation of the corporate learning product. 3 Page
  4. 4. MEASURING TRAINING EFFECTIVENESS Going Back to Basics One of the simplest rules for measuring effectiveness is to ensure you’ve developed materials that will actually train corporate employees. I often meet training professionals who have left adult learning theory in the college classroom long ago, have forgotten much of it and feel compelled to whip out training on demand that is instructionally flawed. All too often, through the quirkiness of corporate fate, some organizations have moved competent people from operational positions into training positions without benefit of being formally taught adult learning theory or instructional design. Many of these new learning professionals seek the comfort of becoming competent in their positions, but they have gaps in their knowledge that cause them to create instructionally flawed materials. Because of these and other reasons, during a career that has spanned thirty plus years, I’ve seen a lot of training developed that is simply not going to be effective because of faulty instructional integrity. Organizations spend a lot of money on these materials. Developers still put much work and effort towards developing these materials. This kind of implementation strategy creates frustration all around the world of corporate training, for employees, employers and the learning professional. No wonder we are getting pressure to prove training (or learning) effectiveness to the C- Suite! All this pressure creates another compulsion—a need to measure the effectiveness of the training. Measuring instructionally flawed training is an exercise in futility. Application of adult learning principles combined with execution of good instructional design is a key underlayment to how well employees can learn, and without it, no measurement will yield satisfying results.4Page
  5. 5. MEASURING TRAINING EFFECTIVENESSAdditionally, if you are creating e-learning (online or webinar learning), a good understanding ofusability heuristics is imperative (Jakob Nielsen, 10 Heuristics for User Interface Design). Poorexecution in programming or ignorance of usability rules can sabotage all the work of constructing abeautifully instructionally sound module of training. The most important of these rules, and in myexperience the most flagrantly violated, is to give learners control of their learning experiences.Formative versus Summative AssessmentsFormative assessments in training are those assessments (in lay terms) that engage learners inassessing themselves and that provide assessment as part of the learning experience (Cowie, B., &Bell, B. (1999), “A model of formative assessment in science education”, Assessment inEducation, 6: 101-116). Simply stated, formative assessment provides for two-way communication ofthe learning so that the assessments themselves are a part of the learning process.In an ideal corporate training program, each learning module (live, online, webinar, whatever) wouldhave a small segment of learning. In these segments, we’d develop assessments in which reflectiveanswers connect disparate information. For instance, in a live training, we’d follow a chunk of trainingon a software application with an exercise in using the software processes in the training. The traineroversees this exercise and provides mentoring through the exercise. We’d scored the exercise, butwe wouldn’t actually grade it, because it is after all, part of the learning experience. The exercise helpsthe trainer gather feedback on which learners “get it” and are successful in completing the exercise,and helps the learners gain insight on how they are doing. Optimally, the trainer provides (and thelearner seeks) additional sources of knowledge if the assessment results indicate the need foralternate learning sources. 5 Page
  6. 6. MEASURING TRAINING EFFECTIVENESS Formative assessment is the use of adult cognition that truly engages the learner. Engagement translates to a better and more satisfying learning experience. These assessments take place during the learning, and should directly relate to learning objectives that are measurable and achievable and, ideally, reflect corporate and business unit goals and objectives, a nearly perfect map where possible. Use of formative assessments in exercises and/or in e-learning modules (drag and drop of glossary terms, for instance) can improve learner engagement provided the assessments are appropriate, relevant and continue to provide for the natural curiosity of a learner. Summative assessments test the effectiveness of the training, and they judge the competency of the learner after the training intervention takes place. We use summative assessments to provide quantifiable data about what the learner learned. We grade summative assessments. We determine what constitutes a passing grade, and devise paths for those who do not pass. Typically, we use summative assessments as the “final” assessment and the learning management system (LMS) houses the scores. In my opinion, this is where the pressure to measure the effectiveness of training falls apart. Very often, we use summative assessment techniques with training that is not instructionally sound and doesn’t have formative assessments to support the instructional integrity of the training. How do you measure effectiveness of training for corporate metric purposes when the underlayment is unsound?6Page
  7. 7. MEASURING TRAINING EFFECTIVENESSL&D, the Business Unit, the EnterpriseAccording to Paul Kearns (Evaluating the ROI from Learning - Cromwell Press, Trowbridge, Wiltshire UK.), a solid starting point for corporatelearning professionals (he calls them either trainers or learning consultants) is to begin with evaluating where L&D stands as a group on his sixstage Learning Maturity Model (LMM). He also advises evaluating the business unit and the enterprise on this model. Figure 1 shows anexample of an LMM.Figure 1. Example of LMM 7 Page
  8. 8. MEASURING TRAINING EFFECTIVENESS In a single enterprise, many L&D units can exist, and they exist in varying stages of maturity. What do we mean by “maturity”? Think about these scenarios:  When you receive an email stating a department needs a five-hour course on X Operation in 30 days, how do you respond? If you jump and throw together a PowerPoint presentation or something similar, you might be in Stage 1, or a reactive mode.  If you have worked toward making the line managers aware that L&D follows normal business processes equivalent to theirs, and that a good quality product might result from adequate budgeting and time, you might be at Stage 3.  As you attend meetings with business unit executives, you become aware of initiatives that will require segments of employees to learn new skills. If you present a strategy with a budget, an achievable deadline, and a commitment to prove the strategy executed as planned and get the strategy approved, you might be in Stage 4, or a true learning partner.  If the CEO knows about your learning strategies, and in fact has helped to steer information to you so that you can include corporate strategies into your work, you might be at Stage 6. You could also be a learning consultant who is teaching the entire organization how to be a learning corporation. Studies show that learning corporations are more profitable than corporations that are not making an investment in being true learning organizations.8Page
  9. 9. MEASURING TRAINING EFFECTIVENESSSo where on the LMM is your L&D unit?Where is your business unit?Where is your corporation?These questions are important to consider. If you are at Stage 1, itcan be almost impossible to measure the effectiveness of yourtraining because your environment won’t allow you to constructinstructionally sound, engaging training. “Death by PowerPoint”became a cliché for a reason. Level 1 assessments (often referred toas “smile sheets”) for training aren’t truly effective if your learnerswere not engaged in the learning, unless you are looking for anegative response as a way to educate line managers of the businessunit. Level 1 assessments in this situation can be a waste of yourtime—time better invested in moving your L&D unit further into theLMM.As you evaluate your own L&D unit, consider how it works internally,with the other business units and within the enterprise. If L&D is atStage 1, what things can you do incrementally to move forward inmaturity?  Begin with a through Training Needs Analysis (TNA) in all business units the L&D unit supports. We define the TNA as the process of defining on-the-job performance requirements and the gaps between the requirements and what employees are presently doing.  Approach every training request with questions about what the requester is trying to achieve. Turn your questions into an analysis of whatever depth you can manage to help guide the development of the deliverable. 9 Page
  10. 10. MEASURING TRAINING EFFECTIVENESS  Make allies of the line managers and ask them about their objectives, goals and strategies. They can give you current performance metrics for their employees. They are your customers, so pay attention to what they want to achieve, and plan for how you might help them.  Human Resources (HR) departments typically have job descriptions and the competencies required to perform those jobs. Compare them with what the line managers are telling you.  Analyze your information sources. Define the opportunities. Eliminate the non-training issues.  Develop an L&D strategy for managing training development. Begin discussing with your ally-managers how to plan for training, and what you need to be successful so you can make them successful.  Develop a replicable process for creating instructionally sound training. Paul Kearns recommends the Deming Cycle: Plan, Do, Check, Act (PDCA) shown in Figure 2. Figure 2. Plan, Do, Check, Act (PDCA) cycle10Page
  11. 11. MEASURING TRAINING EFFECTIVENESS  I also suggest using an EADDIE form (Evaluate, Analyze, Design, Develop, Implement, Evaluate). I further maintain that either process model is fine, as long as you find it replicable, consistent and defendable within your organization. Keep in mind that line managers should respect that you have a process that works.  Practice saying “no” in a nice way. Example: “I probably can get you a presentation in a few hours, but for training that boosts X metrics, I’m going to need to go through our process to define your objectives and make sure the training meets them. For instance, I’ll ask you what you would like your employees to be able to do at the end of this training. Would you like to meet on Monday to start that process?”The Elephant in the RoomExuberance regarding the Kirkpatrick and/or Phillips methods of assessment bombards thecorporate training world almost daily, it seems. There is an onslaught of well over sixtybooks, countless training sessions and local and national American Society for Trainingand Development (ASTD) presentations. It is marketing at its most impressive.In my opinion, L&D professionals have adopted these assessment methods incountless L&D units across the world, with very little evaluation. In many L&D units,unit leaders have not considered the maturity of the units, how they service the linemanagers (their customers), or whether the learning developers can actually createtraining to support summative assessments. For me, the vast depth of marketing theseinstruments as a means of demonstrating ROI raises simple questions: How can this be sodifficult? What is it about these methods that takes so much print space, countlesspresentations and evangelizing? Since they take so much explanation, can theseassessment methods be flawed? 11Page
  12. 12. MEASURING TRAINING EFFECTIVENESS Again, in my opinion, the answer is another question: What method isn’t flawed? The flaw lies in execution, and the execution relies on learning professionals to step up their game. Things to consider in using these methods are:  What is your definition of ROI? Is it the same as the business unit manager’s, or the Chief Financial Officer’s?  There’s tremendous variability in results that unsuspecting learning professionals will encounter. For an excellent discussion on these variables, see Ron Drew Stone’s CLOMedia article, “ROI is Like a Box of Chocolates”. He very cleverly proves that variability in measuring results can negate those results if you aren’t careful and knowledgeable. He puts accurate measurement truly in the hands of outside consultants (which is no big wonder since he is a consultant). He raises several excellent points.  Humans are variable. Things that affect humans are varied. A simple example of that is sales training: how will you isolate the results of your training from variables such as the natural effects of the economy, a new marketing campaign, a product release slowdown or failure, compensation adjustments and so forth? Any one or all of these factors can distort and invalidate your ROI calculations. A random quantity adjustment such as suggested in Ron Drew Stone’s article is exactly that: random, and not so easy to defend.  Why have a metric assessment separate of those of business unit managers? They are, in fact, your customers. Servicing them in pursuit of corporate goals is your job. Doesn’t it make sense to figure out how the business unit managers are measured and to align training goals and assessments accordingly? Knowing these things, it has seemed to me for some years now that it might be better simply to prove that the training we construct has value (Proving Learning Value or PLV).12Page
  13. 13. MEASURING TRAINING EFFECTIVENESSProving Learning Value (PLV)Even at Stage 1 of the LMM, any professional with the grasp of the concepts presented here (and the additional reading I’ll recommend) canbegin the process of proving learning value within their L&D unit, as it applies to the business units and within the enterprise. Start simply andincrementally and work towards making your unit a true business partner with the other enterprise units. Begin by: 1. Understanding business unit goals and how L&D can support them by allying with business unit managers and including yourself in business unit meetings. 2. Understanding the overall goals of the company (as stated in annual reports and CEO messages—and if you are advanced to Stage 5, an actual seat at the C-Suite table). 3. Prioritizing your training interventions according to those goals and objectives. 4. Using your training needs analysis (and gap information derived from it) to plan the training, learning objectives, formative assessments and summative assessments. 5. Deciding upon your method of proving value. Do you want to use Kearns’ approach? Do you still think the Phillips method work for you? Do you need an alternative? 13 Page
  14. 14. MEASURING TRAINING EFFECTIVENESS KISS Method – An alternative way to think about Proving Learning Value If you really want a simple way to provide reasonable metrics without going down the endless path of measure, measure, measure (sometimes forgetting why we are measuring), here’s an idea: use the classic KISS (in this case, Keep It Simple Suggestions) method. Consider the following, borrowed from Kirkpatrick, with caution to evaluate first: 1. Did the learners like the training? (Level 1 in the Kirkpatrick/Phillips methods, but we can maximize a Level 1 through learner engagement and formative assessments.) Did they feel engaged in the learning experience? When they asked if they felt they learned, what was their response? In my opinion, smile sheets are vastly underrated. Given the time and resources to create engaging learning, what better use of an instrument than as a tool for the learner to express their satisfaction? Remember that this level of assessment is perfectly fine for some training. Compliance, regulations and mandatory training having a certain percentage of correct choices required can all be fine for this kind of assessment, because the company has to do this training regardless. The questions then center on ensuring a good learning experience to make sure employees like learning, and that they haven’t felt the experience was a waste of their time. 2. Did the learners learn the content? What method(s) will you use to prove that? Will you pre-test to find out the level of their knowledge before they took the training, then use a post-learning assessment to demonstrate a change? It really could be that simple. Will you use Certainty Based Marking (CBM), a relatively new assessment method that measures the confidence of the learner in what they’ve learned (see Create Active Assessments With Certainty-Based Marking)?14Page
  15. 15. MEASURING TRAINING EFFECTIVENESSIf the learner is confident and right, you probably solidify the learning experience for them (as much as 95% according to the few studiesavailable). If the learner is confident in his or her answer but he or she is wrong, you may have a problem with the training, or you mayneed to reroute the learner to other materials to clear up learner’s confusion.Either way, it is a simple process to implement this method of assessment, and it provides excellent information for proving the value ofthe training with pre-planned reporting mechanisms. It is great for leadership training, and medical institutions have used CBMassessments in medical training (where uncertainty in the answers can have potentially disastrous effects) for years. The Level 1 (orsmile sheet) assessments come back with tremendously positive responses as well, because the assessments serve the purposeof taking ambiguity on the content out of a learner’s mind. 3. Can learners perform the trained tasks on the job? Have you set up preplanned support mechanisms online, with managers, in job aids and in work process alignment to ensure they can perform once the intervention is over? If so, can the line managers give you metrics obtained prior to the training? Did you design your learning objectives, modules and formative assessments in alignment to those metrics? If so, post-intervention performance ratings and appraisals should easily demonstrate learners can perform on the job, and you should see metrics improvements as well. Again, not so hard, and easily defensible with proper planning. 15 Page
  16. 16. MEASURING TRAINING EFFECTIVENESS 4. Did the learning intervention affect the bottom line? Did you evaluate the goal of the intervention within the confines of corporate revenues and/or risk reduction and/or cost reduction? If you did, there should be published metrics on each of these areas that clearly demonstrate impact on the bottom line. Again, did you construct the objectives, modules and formative assessments to support those goals? Are there support mechanisms in place to ensure success after the interventions? Can you demonstrate positive change? We can apply this level of measurement when the whole corporation is all rowing together to achieve the corporate goals. As an example, if you can demonstrate that the learning intervention reduced help desk calls by a significant percentage, you can show that percentage in reduction of cost. Help desk managers always have metrics, and allying yourself with those managers gives you easy access to them.16Page
  17. 17. MEASURING TRAINING EFFECTIVENESSSuggested Reading List:  Paul Kearns Training Journal 12 Part Series:  Measuring For Success – What CEOs Really Think About o Part 01 – Organisational learning maturity Learning Investments, Jack J. Phillips and Patricia Pulliam o Part 02 – Strategic learning Phillips o Part 03 – The attitude of learning consultants  ROI is Like a Box of Chocolates, Ron Drew Stone, CLO Media Chief o Part 04 – Learning cycles Learning Officer Magazine, January 2011 o Part 05 – Evaluation  Ten Usability Heristics, Jakob Nielsen o Part 06 – Performance management o Part 07 – Business analysis o Part 08 – Creative designers o Part 09 – Delivering solutions o Part 10 – Consulting skills o Part 11 – Learning consultants as business partners o Part 12 – Organisation design and development  Evaluating the ROI From Learning, Paul Kearns  What CEOs Expect From Corporate Learning, William J. Rothwell, John E. Lindlholm, William G. Wallick (American Management Association) 17  Page
  18. 18. MEASURING TRAINING EFFECTIVENESS There’s no need for overkill. The observable fact is that some people embrace measurement of the effectiveness of training as a means of job protection, and in many organizations, measuring training effectiveness is a substitute for getting on with the real work. Sadly, this measurement, at worst, becomes an end in itself and, at the least, it is a distraction from our real jobs: getting sound instructional materials to employees to support them in their jobs, and contributing to company goals. Provided we include all the other components of good instructional design and usability rules, and by using common sense methods of measurement without elaborate contortions, learning professionals can ensure training effectiveness soars. Michaels & Associates brings the experience and know-how of solid instructional and media design to every project. Feel free to contact us to assist with your next training endeavor! Michaels & Associates—where your business is your specialty and improving your business is ours. toll-free: 877-614-844018Page
  19. 19. MEASURING TRAINING EFFECTIVENESS About the Author Sherry Michaels is a veteran in the learning industry of more than thirty years and President of Michaels & Associates, a company specializing in instructional, media and writing design and content development for learning. Sherry founded the company in 1998 and developed a staff and network of consultants with development and project management expertise across all disciplines of corporate and academic learning. Sherry has presented several workshops for the American Society for Training and Development (ASTD), the Society for Pharmaceutical and Biological Training (SPBT) and the Society for Technical Communications (STC). Michaels & Associates provides custom training and documentation solutions for a client list that includescompanies such as Aetna-Schaller Anderson, Activator Methods, Inc., Avnet, automätik education (BMW MINI Cooper, Honda), Banner Health, CoxCommunications, Defense Acquisition University (DAU), Dow Jones, EMCOR, Excellus Blue Cross/Blue Shield, McKesson Pharmaceutical, MetLife,Pegasus Solutions, Pfizer, Scottsdale Insurance, Standard Pacific Homes, TriZetto Software and Universal Technical Institute. 19 Page