CONSTRUCT THE ULTIMATE       MULTIPLE CHOICE TEST!    STOP MEASURING RECALL         AND START MEASURING                   ...
AGENDA➤  Review Kirkpatrick/Phillips five level   evaluation model➤  Examine Level 2 evaluation facts➤  Analyze multiple ch...
KIRKPATRICK/PHILLIPS EVALUATION MODEL                                    Levels of Evaluation          Measurement Focus  ...
% LIVE CLASSROOM VS. TECH. BASEDPROGRAMS EVALUATED AT LEVELS 1 – 5*                                         Level         ...
LEVEL 2 EVALUATION FACTS*                                     ➤  81% of organizations evaluate at least   some learning pr...
WHY THE DISCONNECT?➤  Knowledge tests created are overly   simplistic➤  Knowledge test administered too close to   learnin...
WHATʼS THE SOLUTION➤  Create valid, scientifically sound Level 2   knowledge tests that measure job   application not just ...
SAMPLE LEVEL 2 MULTIPLE CHOICE                          KNOWLEDGE                             TEST    Phillips Associates ...
Whatʼs one thing that all 16 multiple           choice test questions have in                      common?Phillips Associa...
They test for knowledge or recall – the          lowest level of cognitive abilityPhillips Associates                     ...
TEST CREATION TIP 1                                         ➤  Test for application or understanding not   just knowledge ...
EXAMPLES➤  What does TV stand for?➤  What is the main function of a TV?➤  What physical principle is used to display   ima...
EXAMPLEYou sent a detailed program design documentto a group of SMEs to solicit their feedback,and have incorporated a num...
WHATʼS WRONG WITH THIS?10.  Effective listening is defined as?      A.  Attempting to understand the person          speaki...
TEST CREATION TIP 2                                            ➤  Keep response options equivalent in       length & writi...
WHATʼS WRONG WITH THIS?4.  A Collaborative Communication Style is    also known as a?     A.  Win/Win communication style....
TEST CREATION TIP 3                                         ➤  Question should not reveal correct    answer Phillips Assoc...
EXAMPLE4.  A Collaborative Communication Style is    also known as a(n)?    A.  Win/Win communication style.    B.  Low co...
WHATʼS WRONG WITH THIS?14.  What communication technique is used     to consolidate what has been previously     discussed...
TEST CREATION TIP 4                                         ➤  All response choices must be plausible Phillips Associates ...
WHATʼS WRONG WITH THIS?6.  A benefit of being an effective listener    is?       A.  Facilitates getting to know another   ...
TEST CREATION TIP 5                                         ➤  Avoid “All the above/None of the above” Phillips Associates...
WHATʼS WRONG WITH THIS?12.  Which is not one of the four    communication styles?    A.  Intuitive    B.  Avoiding    C.  ...
TEST CREATION TIP 6                                         ➤  Write questions same way material taught Phillips Associate...
WHATʼS WRONG WITH THIS?9.      A perception check is a?       A.  Technique for moving a conversation           from the o...
TEST CREATION TIP 7                                         ➤  Include central idea and most of phrasing in    stem  Phill...
EXAMPLE9. A Perception Check communication   technique is used to?       A.  Move a conversation from the           object...
WHATʼS WRONG WITH THIS?2.  ____________________ are used to report    specific, observable actions or details in    an obje...
TEST CREATION TIP 8                                         ➤  Write stem as a question – responses can    finish question,...
WHATʼS WRONG WITH THIS?1.  Interpersonal communication is often    rendered ineffective due to the presence of    communic...
TEST CREATION TIP 9                                            ➤  Eliminate excessive verbiage or irrelevant   information...
EXAMPLE1.  What are the three categories of    communication barriers?   A.  Environmental, Interpersonal and       Intrap...
TEST CREATION TIP 10                                             ➤  Spread correct answers across A – D           Phillips...
OVERALL SUMMARY➤  Develop Level 2 Multiple Choice Tests   that are:   •  Valid   •  Fair to the learner           •  Fair ...
FREE ARTICLESPhillips, Ken, “Developing Valid Level 2 Evaluations”,Training Today, Fall 2009, pps. 6-8.Phillips, Ken, “Eig...
Ken Phillips                                                                  Phillips Associates                         ...
Upcoming SlideShare
Loading in …5
×

CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application

1,449 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,449
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
60
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application

  1. 1. CONSTRUCT THE ULTIMATE MULTIPLE CHOICE TEST! STOP MEASURING RECALL AND START MEASURING APPLICATION Presented by: Ken Phillips Phillips Associates August 16, 2011Phillips Associates
  2. 2. AGENDA➤  Review Kirkpatrick/Phillips five level evaluation model➤  Examine Level 2 evaluation facts➤  Analyze multiple choice test creation errors in a sample Level 2 knowledge test➤  Learn 10 tips for creating valid, scientifically sound multiple choice test questions Phillips Associates 2
  3. 3. KIRKPATRICK/PHILLIPS EVALUATION MODEL Levels of Evaluation Measurement Focus Time FrameLevel 1: Reaction Participant reaction to a Conclusion of program learning programLevel 2: Learning Degree to which participants Conclusion of program or acquired new knowledge, within 6 to 8 weeks after skills or attitudesLevel 3: Behavior Degree to which participants 2 to 12 months applied back-on-the-job what was learnedLevel 4: Results Degree to which targeted 9 to 18 months business outcomes were achievedLevel 5: ROI Degree to which monetary 9 to 18 months program benefits exceed program costs Phillips Associates 3
  4. 4. % LIVE CLASSROOM VS. TECH. BASEDPROGRAMS EVALUATED AT LEVELS 1 – 5* Level Live Classroom Tech. Based 1-Reaction 81% 52% 2-Learning 50% 43% 3-Behavior 25% 15% 4-Results 14% 10% 5- ROI 6% 6% *ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 Phillips Associates 4
  5. 5. LEVEL 2 EVALUATION FACTS* ➤  81% of organizations evaluate at least some learning programs at Level 2➤  55% of organizations view data collected as having high or very high value 
 *ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 Phillips Associates 5
  6. 6. WHY THE DISCONNECT?➤  Knowledge tests created are overly simplistic➤  Knowledge test administered too close to learning event➤  Test items focus on recall not application➤  The data collected is not used Phillips Associates 6
  7. 7. WHATʼS THE SOLUTION➤  Create valid, scientifically sound Level 2 knowledge tests that measure job application not just recall➤  Include knowledge tests as part of learning program design and use for both evaluation and reinforcement➤  Aggregate knowledge test score data and use to demonstrate program value as well as improve test item validity Phillips Associates 7
  8. 8. SAMPLE LEVEL 2 MULTIPLE CHOICE KNOWLEDGE TEST Phillips Associates 8
  9. 9. Whatʼs one thing that all 16 multiple choice test questions have in common?Phillips Associates 9
  10. 10. They test for knowledge or recall – the lowest level of cognitive abilityPhillips Associates 10
  11. 11. TEST CREATION TIP 1 ➤  Test for application or understanding not just knowledge Phillips Associates 11
  12. 12. EXAMPLES➤  What does TV stand for?➤  What is the main function of a TV?➤  What physical principle is used to display images on a TV?➤  Your TV is not working properly. What is the most likely cause of the following set of symptoms:…? Phillips Associates 12
  13. 13. EXAMPLEYou sent a detailed program design documentto a group of SMEs to solicit their feedback,and have incorporated a number of theirsuggestions into the program design. This isan example of what element in the ADDIEmodel?A.  Analysis D. ImplementB.  Design E. EvaluationC.  Development Phillips Associates 13
  14. 14. WHATʼS WRONG WITH THIS?10.  Effective listening is defined as? A.  Attempting to understand the person speaking from his or her perspective and not your own. B.  Encouraging another person to talk. C.  Consolidating the important parts of a discussion. D.  Creating a win-win communication situation. Phillips Associates 14
  15. 15. TEST CREATION TIP 2 ➤  Keep response options equivalent in length & writing style Phillips Associates 15
  16. 16. WHATʼS WRONG WITH THIS?4.  A Collaborative Communication Style is also known as a? A.  Win/Win communication style. B.  Low concern for self and high concern for other communication style. C.  Engaging and outgoing communication style. D.  Intermediate concern for self and high concern for other communication style. Phillips Associates 16
  17. 17. TEST CREATION TIP 3 ➤  Question should not reveal correct answer Phillips Associates 17
  18. 18. EXAMPLE4.  A Collaborative Communication Style is also known as a(n)? A.  Win/Win communication style. B.  Low concern for self and high concern for other communication style. C.  Engaging and outgoing communication style. D.  Intermediate concern for self and high concern for other communication style. Phillips Associates 18
  19. 19. WHATʼS WRONG WITH THIS?14.  What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another? A.  Arguing B.  Interrupting C.  Summarizing D.  Initiating Phillips Associates 19
  20. 20. TEST CREATION TIP 4 ➤  All response choices must be plausible Phillips Associates 20
  21. 21. WHATʼS WRONG WITH THIS?6.  A benefit of being an effective listener is? A.  Facilitates getting to know another person better. B.  Increases the likelihood of having a mutual exchange. C.  Helps another person to get to know himself or herself better. D.  All the above. Phillips Associates 21
  22. 22. TEST CREATION TIP 5 ➤  Avoid “All the above/None of the above” Phillips Associates 22
  23. 23. WHATʼS WRONG WITH THIS?12.  Which is not one of the four communication styles? A.  Intuitive B.  Avoiding C.  Aggressive D.  Collaborative Phillips Associates 23
  24. 24. TEST CREATION TIP 6 ➤  Write questions same way material taught Phillips Associates 24
  25. 25. WHATʼS WRONG WITH THIS?9.  A perception check is a? A.  Technique for moving a conversation from the objective, where the focus is on facts, to the subjective, where the focus is on feelings. B.  Technique for pulling together the important parts of one long conversation. C.  Technique for helping someone to express his or her real self. D.  Technique for conveying interest in what another person is saying. Phillips Associates 25
  26. 26. TEST CREATION TIP 7 ➤  Include central idea and most of phrasing in stem Phillips Associates 26
  27. 27. EXAMPLE9. A Perception Check communication technique is used to? A.  Move a conversation from the objective to the subjective. B.  Pull together the important parts of one long conversation. C.  Help someone express his or her real self. D.  Convey interest in what another person is saying. Phillips Associates 27
  28. 28. WHATʼS WRONG WITH THIS?2.  ____________________ are used to report specific, observable actions or details in an objective, non-evaluative manner. A.  Behavior descriptions B.  Open probes C.  Evaluative statements D.  Direct statements Phillips Associates 28
  29. 29. TEST CREATION TIP 8 ➤  Write stem as a question – responses can finish question, but donʼt use fill-in-the- blank Phillips Associates 29
  30. 30. WHATʼS WRONG WITH THIS?1.  Interpersonal communication is often rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers? A.  Environmental, Interpersonal and Intrapersonal B.  Environmental, Interpersonal and Cultural C.  Structural, Interpersonal and Cultural D.  Organizational, Intrapersonal and Structural Phillips Associates 30
  31. 31. TEST CREATION TIP 9 ➤  Eliminate excessive verbiage or irrelevant information from the stem Phillips Associates 31
  32. 32. EXAMPLE1.  What are the three categories of communication barriers? A.  Environmental, Interpersonal and Intrapersonal B.  Environmental, Interpersonal and Cultural C.  Structural, Interpersonal and Cultural D.  Organizational, Intrapersonal and Structural Phillips Associates 32
  33. 33. TEST CREATION TIP 10 ➤  Spread correct answers across A – D Phillips Associates 33
  34. 34. OVERALL SUMMARY➤  Develop Level 2 Multiple Choice Tests that are: •  Valid •  Fair to the learner •  Fair to the organization & •  Test for understanding Phillips Associates 34
  35. 35. FREE ARTICLESPhillips, Ken, “Developing Valid Level 2 Evaluations”,Training Today, Fall 2009, pps. 6-8.Phillips, Ken, “Eight Tips on Developing Valid Level 1
Evaluation Forms”, Training Today, Fall 2007,
pps. 8 & 14. Phillips Associates 35
  36. 36. Ken Phillips Phillips Associates 34137 N. Wooded Glen Drive Grayslake, Illinois 60030 (847) 231-6068 www.phillipsassociates.com ken@phillipsassociates.comPhillips Associates 36

×