3. Lesson ObjectivesLesson Objectives
You will be able to
• Identify 4 levels of evaluation
• Apply methods for each level
• Develop an evaluation strategy
3
5. Why evaluate?Why evaluate?
• Ensure objectives are achieved
• Reinforce key program points
• Assess value of training
• Identify areas needing improvement
• Sell program
• Identify appropriate audience for future
5
6. When to evaluate?When to evaluate?
During
Training
End of
Training
Back at
Job/Home
6
7. Who is involved?Who is involved?
• Trainer
• Learners
• Coworkers
• Manager
7
22. Apply to your lessonApply to your lesson
Take a few minutes
• Find “Levels of Evaluation” Worksheet (p 7-10)
• Choose one evaluation level
• Jot down some methods for your plan
Table 7.1 (p 7-3) and Table 7.2 (p 7-7)
22
23. Group ExerciseGroup Exercise
• Find a partner
• Share your evaluation plan
• Pick one to illustrate
• Present to group
Objective, level, method
23
24. Develop an Evaluation StrategyDevelop an Evaluation Strategy
1) Create a Plan
• Pick an evaluation level
• Identify information to be collected
• Select method
24
25. Develop an Evaluation StrategyDevelop an Evaluation Strategy
2) Collect and Analyze Information
• Design an instrument to collect data
Search online for “sample training
evaluation forms”
• Decide how to analyze data
25
26. Some Tips on Form DesignSome Tips on Form Design
• Keep it short
• Tie questions to objectives
• Focus on actionable items
• Mainly closed-end questions
• Use open-ends to find gaps
• Make it anonymous
26
27. Develop an Evaluation StrategyDevelop an Evaluation Strategy
3) Present Evaluation Results
• Highlight lessons learned
• Present results
with recommendations
to management
27
31. Best of Luck with yourBest of Luck with your
Presentations!Presentations!
Contact Information:
Joan Serpico
Manager Special Projects, Mount Laurel Library
jserpico@mtlaurel.lib.nj.us 856-234-7319 x314
Barb Hauck-Mah
Director, Lodi Memorial Library
hauckmah@bccls.org 973-365-4044 x7
31
Editor's Notes
HI. My name is Joan Serpico and I am the Manager of Special Projects at Mount Laurel Library. I graduated from TTT in 2004 and I am happy to continue to be involved in it today. My experience with TTT has been very positive – not only did it improve my training, but I got to know dozens or smart helpful people in my field who I have come to lean on for ideas and enjoy working with on projects. It has been a great experience for me.
Hi. I’m Barb Hauck-Mah, Director of Lodi Memorial Library, a public library in Bergen County. I graduated from TTT in 2009 and share Joan’s enthusiasm for the value of this special program
At this point you are probably all getting a little more nervous as your presentations get closer. (That is how we all felt.) But I guarantee you that you will be so happy that you took this training. Do you sense that already?
Well, like all of the trainers, we are here to help you make your future training better. This section is Evaluating Training.
Carol talked about evaluating your training in her section on lesson plans and course objectives. It’s good to think about perfecting your technique throughout your planning (timing, clarity, etc.)
This section focuses on evaluating your training based on the impact that the training has on your learners. What are they taking away from your training? We’ll talk about how you can determine this.
Though Barb and I will be sharing some thoughts and techniques, I know you have some experiences with these and I hope you will share them as we go along. (By now you realize that some of the most valuable things you will learn at TTT come from your fellow attendees.)
By the end of this lesson, you will be able to evaluate your training.
Specifically, you will be able to …
Identify four levels of evaluation
Identify evaluation methods that can be used for each level
Develop an evaluation strategy
I’ll be keeping the floor open to hear what methods you have already used and what has worked and what hasn’t – especially what hasn’t. (These make for valuable learning opportunities and the best stories.)
The most important question is why bother? It is extra work. Why not present some flawless training, take a deep breath and celebrate?
Barb/Joan is going to help record why we would decide to bother with this.
(Flipchart floosies records answers from audience.)
Let’s see how many we identified and if there are any extras here. (Instructor to underline answers on flipchart that correspond to items in list.)
To ensure training objectives are achieved
To review & reinforce key program points for learners
To assess value of training
To identify areas needing improvement
To sell program to management & learners
To identify appropriate audience for future
OK. Those seem to indicate it would be worth some time to do this step. Let’s look at when we would evaluate…
When you evaluate depends on what level and method you select. You may evaluate during training and at the end of the
session and later on.
Next, let’s talk about who would be involved in the evaluating...
Who are some of the people who can be helpful in the evaluation?
Remember, rather than judging you the evaluation process is designed to
give you information to improve future training.
So we know why, when and who. What kind of evaluating are we going to do?
Donald Kirkpatrick introduced a model for evaluating training in 1959.
It has become the standard for trainers.
In Donald Kirkpatrick’s model, level 1-the Reaction level -tells how learners felt about the training. Level 2 – the Learning level – evaluates the new skills the learners have acquired. Level 3 – the Behavior level – takes a look at how the learners behavior has changed either on the job or at home. Level 4 – the Results level – takes a look at the organizational impact of a training.
Let’s take a closer look at each level and some methods we can use for each. We’ll apply a training example together for each level and then later, you will have some time to apply a level and a method for your practice training.
The training example we will use throughout our discussion of evaluation levels will be an eReaders class for Library patrons – how to download our ever popular ebooks onto various devices. How many of you are offering some kind of training for your patrons on downloading ebooks? What kind of training is it? How is it going? Throughout the explanation of evaluation, we will imagine that we are planning an evaluation of a class on downloading ebooks. Let’s get started examining the different levels.
The reaction evaluation measures how the learner liked the training. How did they feel about it?
It is subjective and does not evaluate what they have learned but how they felt about it.
This is done right after the training sessions. (Make sure to leave time for learners to complete the evaluation.)
It can be done via an evaluation form also known as a smile sheet or an interview.
There are two evaluation samples in your course book that are examples of Level 1 evaluation. They are located on pages 7- 5 and 7-6.
How many of you have filled out evaluations like this before? How many of you have used these before in your own training? What do you think?
Let’s imagine that we were giving a training session on downloading ebooks for patrons. Take a look at the sample level 1 evaluation forms in your coursebooks. Which of these could be adapted to be used for this ereader training? (Instructor waits for responses.)
Right. Either of them could be used. Feel free to adopt any of these samples for your own training.
An alternative to paper evaluation forms is a free online evaluation survey from websites such as Survey Monkey or Zoomerang.
The survey can be emailed as a link, embedded on a webpage or sent as an APP as soon as training is completed.
How many of you have filled out an online evaluation before? How many have used online evaluations in your own training? What do you think?
Most survey websites have evaluation templates you can customize. Both multiple choice and open end questions can be included.
We’ve provided a handout on popular online survey sites to get you started.
The learning evaluation level measures what the learners are now able to do or what they know as a result of the training. In short, what have they learned. It can be both subjective or objective and can take place during training or before/after training as in a pre-test/post-test.
Evaluating participants’ learning can take place through observation of how they are applying their skills – role plays, case studies, exercises. Interviews can also gather this kind of information.
Who has used this level of evaluation before? How have they worked for you?
What would be a way that we could apply level 2 learning to the Ereaders training example? Feel free to look at Table 7-2 (Level 2 Evaluation Methods) on page 7-7 for some ideas. (Instructor to wait for responses.) Possible answers – exercises, observation
The Behavior evaluation level measures how the learners actually apply
what they have learned as a result of the training. In short, how they are
performing differently. How they have transferred what they have learned
to the work or home situation?
It is usually done a few months after a training.
Evaluating behavior can be done through observation, interviews or via surveys.
When observing, it makes sense to see how the learners behaved during training vs after. A checklist is a good idea to maintain consistency in observations.
You could also survey or interview the learners or the people that they work with to see how behavior has changed.
Has anyone ever done this kind of evaluating?
How could we apply this level of evaluation to our ereaders example to determine if the training has altered behavior? (Sample answers – interview or survey learners about numbers of ebooks downloaded.)
Level 4 evaluation (Results) determines the impact of the training on the organization.
It is performed by the organization.
It may not be appropriate for all training (like the ereader training). I mention it mostly so you are aware of it.
If your organization wanted to evaluate a Customer Service training, for
example, they might compare the customer complaints gathered before vs
after the training.
The organization could do an analysis of pre and post training statistics or a cost-benefit analysis to see the impact of training on goals.
(Examples of items to be measured include absenteeism, sales, or turnover rate, etc)
Going back to Donald Kirkpatrick’s model as a review, level 1-the Reaction level -tells how learners felt about the training. Level 2 – the Behavior level – evaluates the new skills the learners have acquired. Level 3 – the Behavior level – takes a look at how the learners behavior has changed either on the job or at home. Level 4 – the Results level – takes a look at the organizational impact of a training.
Does that make sense to everyone?
Now let’s look at some evaluation methods and make sure we understand what level of evaluation they measure. Please turn to the worksheet on page 7-9. It’s titled “Determining Levels of Evaluation.” Take a few minutes to complete it and then compare your answers with a partner.
Take about 10 minutes to do this and then we’ll go over these briefly as a group.
After 10 minutes when all are finished with the worksheet and comparing with a partner.
Did anyone have any disagreements about the levels? (If so, go over those first. If not, review answers.)
Great! Do you feel like you have an understanding of the levels, what they determine and how the methods can be used to measure this?
Now let’s apply these methods to your training. Go to page 7-10 “Levels of Evaluation” worksheet. List your lesson objective and choose which level of evaluation you wish to use. Then jot down a few possible methods of evaluation that would work for you at that level. Use Tables 7.1 and 7.2 as “cheat sheets.” Take about 5-10 minutes for this.
Great. Let’s take a few minutes to share our ideas with each other. Find your partner and discuss your choices. Feel free to share your experiences with these levels and different methods and how they have worked for you already.
Choose one method between the two of you and illustrate it on a piece of flipchart paper. When we come together, one of you will describe your partner’s objective, chosen level of evaluation and method and why it was chosen.
Here is an example of what I’m talking about. We talked about how survey would be one level 3 evaluation method for our ereaders training evaluation. Here is an artistic rendering of that. (Instructor shows flipchart paper example.) The trainer is sending an email to the learners about 3 months after an ereaders class to ask about their current ebook downloading habits.
Let’s take about 10-15 minutes for this. We’ll be around to answer questions, if needed.
The next steps are outlined in How to Create an Evaluation Strategy found on page 7-11. Let’s take a look at that.
First: Develop a Plan
Pick an evaluation level
Decide what information needs to be collected
Decide on a method
(We’ve done all that.)
Once you have an evaluation plan, you will need to design an instrument.
You do not need to do this from scratch. Feel free to borrow one and
personalize it so it works for your training. Sample evaluation forms can
be found in your coursebook.
When designing an evaluation form, here are a few tips:
Keep the form simple and short. A page or two that could be completed in a few minutes is recommended.
Focus questions on aspects of the training that could be modified.
Offer a mix of rating, multiple choice and open ended questions. This approach may reveal issues or strengths you hadn’t identified.
If possible, make the evaluation anonymous to ensure honesty. Or, offer identifcation as an option.
Finally, you will collect the data, analyze it and you may decide to share
your results with colleagues or your supervisor. All of this is designed to
help improve your training.
Presenting your evaluation findings and describing how you plan to
adjust your training as a result can go a long way toward convincing your managers
administration of the value of the training and the relevance to your learners.
As we have discussed, evaluation is a crucial part of improving your
training. Your Evaluation Strategy involves deciding what you want to
evaluate, picking an appropriate level and method, designing an
instrument to collect the information, and analyzing and presenting your
findings.
Evaluation is simply part of a continuing cycle of the training process that
is ongoing to allow you to be the best trainer you can be.
To that end, Joanne is up next to talk about practice training.
Please feel free to contact us if we can ever be of help now or in the future.
