Developing Valid Level 1 Evaluation Forms

Developing Valid Level 1 Evaluation Forms Presented by: Ken Phillips Phillips Associates May 22, 2008

OBJECTIVES

Examine core purpose of Level 1 evaluations

Analyze strengths and shortcomings of 3 sample Level 1 evaluation forms

Apply 6 tips to design more valid Level 1 evaluations

QUESTIONS

What is the core purpose of Level 1 evaluations?

How valid are the results?

ANSWERS

Improve a

learning program

not prove

something

Not very valid &

SAMPLE

EVALUATION

FORMS

6 TIPS

TIP 1

Where appropriate, match up qualitative questions with quantitative measures

EXAMPLE

In a word, how would you describe this

session? ____________________________

Using a number, how would you describe this session? No Value Great Value 1 2 3 4 5

TIP 2

When collecting quantitative data using a Likert scale, create a response scale with numbers at regularly spaced intervals and words only at each end*

* Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”, Harvard

Business Review , 2002.

EXAMPLE

How would you rate the overall effectiveness of

the learning activities used in this session?

Not Very

Effective Effective

1 2 3 4 5

TIP 3

Use only one response scale with an odd number of points (e.g. 5, 7,9, 11)*

* Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”

ODD vs EVEN SCALE

TIP 4

Use small numbers at the low or left end of the scale and larger numbers at the right or high end of the scale

TIP 5

Write items either as a continuum or as a statement

How effectively did the facilitator use AV

materials?

Not Very

Effectively Effectively

1 2 3 4 5

EXAMPLES The facilitator effectively used AV materials. Strongly Strongly Disagree Agree 1 2 3 4 5

TIP 6

Include at least one item asking participants how relevant the learning event/material was to them and their job

EXAMPLE

How would you rate the overall relevance of the

session to you and your job?

Not at all Very

relevant relevant

1 2 3 4 5

REFERENCES

Boehle, Sarah, “Remember that mean, crabby teacher in

high school?”, Training , August 2006, pps. 17-22.

Phillips, Ken, “Six Tips on Developing Valid Level 1 Evaluation Forms”, Training Today , Fall 2007, pps. 8 & 14. Morrel-Samuels, Palmer, “Getting the Truth into Workplace Surveys”, Harvard Business Review , February 2002, pps. 111-118. Miner, Nanette, “The Art of Test Creation”, T + D , July 2007, pps. 27-29.