Your SlideShare is downloading. ×
0
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

CETS 2011, Ken Phillips, slides for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application

786

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
786
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
46
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. CONSTRUCT THE ULTIMATE MULTIPLE CHOICE TEST! STOP MEASURING RECALL AND START MEASURING APPLICATION Presented by: Ken Phillips Phillips Associates August 16, 2011Phillips Associates
  • 2. AGENDA➤  Review Kirkpatrick/Phillips five level evaluation model➤  Examine Level 2 evaluation facts➤  Analyze multiple choice test creation errors in a sample Level 2 knowledge test➤  Learn 10 tips for creating valid, scientifically sound multiple choice test questions Phillips Associates 2
  • 3. KIRKPATRICK/PHILLIPS EVALUATION MODEL Levels of Evaluation Measurement Focus Time FrameLevel 1: Reaction Participant reaction to a Conclusion of program learning programLevel 2: Learning Degree to which participants Conclusion of program or acquired new knowledge, within 6 to 8 weeks after skills or attitudesLevel 3: Behavior Degree to which participants 2 to 12 months applied back-on-the-job what was learnedLevel 4: Results Degree to which targeted 9 to 18 months business outcomes were achievedLevel 5: ROI Degree to which monetary 9 to 18 months program benefits exceed program costs Phillips Associates 3
  • 4. % LIVE CLASSROOM VS. TECH. BASEDPROGRAMS EVALUATED AT LEVELS 1 – 5* Level Live Classroom Tech. Based 1-Reaction 81% 52% 2-Learning 50% 43% 3-Behavior 25% 15% 4-Results 14% 10% 5- ROI 6% 6% *ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 Phillips Associates 4
  • 5. LEVEL 2 EVALUATION FACTS* ➤  81% of organizations evaluate at least some learning programs at Level 2➤  55% of organizations view data collected as having high or very high value 
 *ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 Phillips Associates 5
  • 6. WHY THE DISCONNECT?➤  Knowledge tests created are overly simplistic➤  Knowledge test administered too close to learning event➤  Test items focus on recall not application➤  The data collected is not used Phillips Associates 6
  • 7. WHATʼS THE SOLUTION➤  Create valid, scientifically sound Level 2 knowledge tests that measure job application not just recall➤  Include knowledge tests as part of learning program design and use for both evaluation and reinforcement➤  Aggregate knowledge test score data and use to demonstrate program value as well as improve test item validity Phillips Associates 7
  • 8. SAMPLE LEVEL 2 MULTIPLE CHOICE KNOWLEDGE TEST Phillips Associates 8
  • 9. Whatʼs one thing that all 16 multiple choice test questions have in common?Phillips Associates 9
  • 10. They test for knowledge or recall – the lowest level of cognitive abilityPhillips Associates 10
  • 11. TEST CREATION TIP 1 ➤  Test for application or understanding not just knowledge Phillips Associates 11
  • 12. EXAMPLES➤  What does TV stand for?➤  What is the main function of a TV?➤  What physical principle is used to display images on a TV?➤  Your TV is not working properly. What is the most likely cause of the following set of symptoms:…? Phillips Associates 12
  • 13. EXAMPLEYou sent a detailed program design documentto a group of SMEs to solicit their feedback,and have incorporated a number of theirsuggestions into the program design. This isan example of what element in the ADDIEmodel?A.  Analysis D. ImplementB.  Design E. EvaluationC.  Development Phillips Associates 13
  • 14. WHATʼS WRONG WITH THIS?10.  Effective listening is defined as? A.  Attempting to understand the person speaking from his or her perspective and not your own. B.  Encouraging another person to talk. C.  Consolidating the important parts of a discussion. D.  Creating a win-win communication situation. Phillips Associates 14
  • 15. TEST CREATION TIP 2 ➤  Keep response options equivalent in length & writing style Phillips Associates 15
  • 16. WHATʼS WRONG WITH THIS?4.  A Collaborative Communication Style is also known as a? A.  Win/Win communication style. B.  Low concern for self and high concern for other communication style. C.  Engaging and outgoing communication style. D.  Intermediate concern for self and high concern for other communication style. Phillips Associates 16
  • 17. TEST CREATION TIP 3 ➤  Question should not reveal correct answer Phillips Associates 17
  • 18. EXAMPLE4.  A Collaborative Communication Style is also known as a(n)? A.  Win/Win communication style. B.  Low concern for self and high concern for other communication style. C.  Engaging and outgoing communication style. D.  Intermediate concern for self and high concern for other communication style. Phillips Associates 18
  • 19. WHATʼS WRONG WITH THIS?14.  What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another? A.  Arguing B.  Interrupting C.  Summarizing D.  Initiating Phillips Associates 19
  • 20. TEST CREATION TIP 4 ➤  All response choices must be plausible Phillips Associates 20
  • 21. WHATʼS WRONG WITH THIS?6.  A benefit of being an effective listener is? A.  Facilitates getting to know another person better. B.  Increases the likelihood of having a mutual exchange. C.  Helps another person to get to know himself or herself better. D.  All the above. Phillips Associates 21
  • 22. TEST CREATION TIP 5 ➤  Avoid “All the above/None of the above” Phillips Associates 22
  • 23. WHATʼS WRONG WITH THIS?12.  Which is not one of the four communication styles? A.  Intuitive B.  Avoiding C.  Aggressive D.  Collaborative Phillips Associates 23
  • 24. TEST CREATION TIP 6 ➤  Write questions same way material taught Phillips Associates 24
  • 25. WHATʼS WRONG WITH THIS?9.  A perception check is a? A.  Technique for moving a conversation from the objective, where the focus is on facts, to the subjective, where the focus is on feelings. B.  Technique for pulling together the important parts of one long conversation. C.  Technique for helping someone to express his or her real self. D.  Technique for conveying interest in what another person is saying. Phillips Associates 25
  • 26. TEST CREATION TIP 7 ➤  Include central idea and most of phrasing in stem Phillips Associates 26
  • 27. EXAMPLE9. A Perception Check communication technique is used to? A.  Move a conversation from the objective to the subjective. B.  Pull together the important parts of one long conversation. C.  Help someone express his or her real self. D.  Convey interest in what another person is saying. Phillips Associates 27
  • 28. WHATʼS WRONG WITH THIS?2.  ____________________ are used to report specific, observable actions or details in an objective, non-evaluative manner. A.  Behavior descriptions B.  Open probes C.  Evaluative statements D.  Direct statements Phillips Associates 28
  • 29. TEST CREATION TIP 8 ➤  Write stem as a question – responses can finish question, but donʼt use fill-in-the- blank Phillips Associates 29
  • 30. WHATʼS WRONG WITH THIS?1.  Interpersonal communication is often rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers? A.  Environmental, Interpersonal and Intrapersonal B.  Environmental, Interpersonal and Cultural C.  Structural, Interpersonal and Cultural D.  Organizational, Intrapersonal and Structural Phillips Associates 30
  • 31. TEST CREATION TIP 9 ➤  Eliminate excessive verbiage or irrelevant information from the stem Phillips Associates 31
  • 32. EXAMPLE1.  What are the three categories of communication barriers? A.  Environmental, Interpersonal and Intrapersonal B.  Environmental, Interpersonal and Cultural C.  Structural, Interpersonal and Cultural D.  Organizational, Intrapersonal and Structural Phillips Associates 32
  • 33. TEST CREATION TIP 10 ➤  Spread correct answers across A – D Phillips Associates 33
  • 34. OVERALL SUMMARY➤  Develop Level 2 Multiple Choice Tests that are: •  Valid •  Fair to the learner •  Fair to the organization & •  Test for understanding Phillips Associates 34
  • 35. FREE ARTICLESPhillips, Ken, “Developing Valid Level 2 Evaluations”,Training Today, Fall 2009, pps. 6-8.Phillips, Ken, “Eight Tips on Developing Valid Level 1
Evaluation Forms”, Training Today, Fall 2007,
pps. 8 & 14. Phillips Associates 35
  • 36. Ken Phillips Phillips Associates 34137 N. Wooded Glen Drive Grayslake, Illinois 60030 (847) 231-6068 www.phillipsassociates.com ken@phillipsassociates.comPhillips Associates 36

×