CETS 2011, Ken Phillips, handout for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application
Upcoming SlideShare
Loading in...5

CETS 2011, Ken Phillips, handout for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application






Total Views
Slideshare-icon Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    CETS 2011, Ken Phillips, handout for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application CETS 2011, Ken Phillips, handout for Construct the Ultimate Multiple-Choice Test! Stop Measuring Recall and Start Measuring Application Document Transcript

    • The “How-To” Measurement & Evaluation Series Increased productivity throughskill building Construct the Ultimate Multiple Choice Test! Stop Measuring Recall and Start Measuring Application Chicago e-Learning & Technology Showcase August 16, 2011
    • Today’s PresentationDo you feel like you know how questions that are too simple, that “Great speaker and subject matter.much participants truly learn from are overly difficult, or that test Ken’s expertise really made the eventyour elearning programs? If not, knowledge and not application. In worthwhile.” –South Central Wisconsin ASTDyou’re not alone. According to a this session, you will learn the art Chapter Member2009 ASTD research study, 81% of and science of Level 2 multipleorganizations measure participants’ choice test creation. Together, we “This was very helpful. Obviouslylearning (Level 2 evaluation), yet will analyze a sample test and learn many people write tests for adults thatonly 55% of these organizations how to identify and correct common can be flawed. This session providedview the data collected to be of real mistakes. Whether you are new to good guidelines for writing valid tests.”value. The problem is that the elearning or an advanced –South Central Wisconsin ASTDvalidity of our results is only as professional, you will walk away Chapter Membergood as the quality of our test with skills to help you improve yourquestions. Often, we develop online Level 2 multiple-choice tests.About Your Presenter — KEN PHILLIPS Ken Phillips is founder and CEO of Phillips Associates, a consulting company with expertise in performance Ken is a frequent presenter at management, measurement and evaluation of learning and ASTD chapter meetings and other sales performance. Ken has more than 25 years experience industry events. Here’s a preview of designing learning instruments and assessments and has where he’ll be speaking next. authored more than a dozen published instruments. He also regularly speaks to American Society for Training and Development (ASTD) groups and university classes. For 16 Chicago e-Learning & the past four years, he has spoken at the ASTD Technology Showcase International Conference on topics related to measurement Topic: Construct theand evaluation of learning. Ultimate Multiple-ChoicePrior to pursuing a Ph.D. in the combined fields of organizational behavior and Test! Stop Measuring Recalleducational administration at Northwestern University, Ken held management and Start Measuringpositions with two colleges and two national corporations. In addition, he has Applicationwritten articles that have appeared in T+D magazine, Training Today andHR.com.Currently Ken is serving his seventh year on the board of directors of the 14 Training & PerformanceChicagoland Chapter of the American Society for Training and Development Management PDN(CCASTD), and was chapter president in 2008. He earned the Certified (Chicago)MeetingProfessional in Learning and Performance (CPLP) credential from national Topic: Learn 12 SurprisingASTD in 2006 as a pilot pioneer and was recertified in 2009. Ken is also Techniques for Obtainingcertified in Measuring and Evaluating Learning and Test Design and Delivery Powerful Data from Level 1by ASTD and in the Kirkpartick Four LevelsTM Evaluation. In January 2009, Evaluationshe also was appointed to serve a three-year term (2009-2011) as a member ofthe ASTD National Advisors for Chapters (NAC) committee.Ken can be reached at (847) 231-6068 or ken@phillipsassociates.com.Copyright ©2011 by Phillips AssociatesAll rights reserved. Printed in the United States of America. Except as permitted under the United States Copyright Act of 1976, no partof the publication may be reproduced or distributed in an form or by any means, or stored in a database or retrieval system, without theprior written permission from Phillips Associates
    • Sample Level 2 Multiple Choice Knowledge Test Interpersonal communication is often rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers? Correct Answer: A. A. Environmental, Interpersonal and Intrapersonal Environmental, 1. Interpersonal B. Environmental, Interpersonal and Cultural and Intrapersonal C. Structural, Interpersonal and Cultural D. Organizational, Intrapersonal and Structural _________________________ are used to report specific, observable actions or details in an objective, non-evaluative manner. A. Behavior descriptions Correct Answer: 2. A. Behavior B. Open probes descriptions C. Evaluative statements D. Direct statements What are questions that begin with words such as is, are, could, would, have and shall known as? A. Clarifying questions Correct Answer: 3. C. Close-ended B. Open-ended questions questions C. Close-ended questions D. Bridging questions A Collaborative Communication Style is also known as a? A. Win/Win communication style Correct Answer: B. Low concern for self and high concern for other A. Win/Win 4. communication style communication style C. Engaging and outgoing communication style D. Intermediate concern for self and high concern for other communication style 1 www.phillipsassociates.com  
    • What are “You” messages? A. Non-ownership statements that convey feelings, Correct Answer: but do not actually describe the feelings. A. Non- ownership B. Statements that contain the pronoun “you” two or statements that 5. more times. convey feelings, but do not C. Statements that point out what another person actually has done or was supposed to do. describe the D. Statements that describe behavior and begin with feelings. the word “You.” A benefit of being an effective listener is? A. Facilitates getting to know another person better B. Increases the likelihood of having a mutual Correct Answer: 6. exchange D. All the above C. Helps another person to get to know himself or herself better D. All the above What are Hidden Agenda Questions? A. An indirect or “pseudo” form of communication. Correct Answer: B. A question used to clarify what someone has A. An indirect or 7. said. “pseudo” form of C. A question used to direct a conversation in a communication. different direction. D. A technique for getting another person to talk. What are questions that begin with words such as what, where, when, how and why known as? A. Bridging questions Correct Answer: 8. D. Open-ended B. Clarifying questions questions C. Leading questions D. Open-ended questions 2 www.phillipsassociates.com  
    • A perception check is a? A. Technique for moving a conversation from the Correct Answer: objective, where the focus is on facts, to the A. Technique subjective, where the focus is on feelings. for moving a conversation B. Technique for pulling together the important parts from the 9. of one long conversation or a series of objective, where conversations. the focus is on facts, to the C. Technique for helping someone to express his or subjective, her real self. where the focus D. Technique for conveying interest in what another is on feelings. person has said. Effective listening is defined as? Correct Answer: A. Attempting to understand the person speaking A. Attempting to from his or her perspective and not your own. understand the 10. person speaking B. Encouraging another person to talk. from his or her C. Consolidating the important parts of a discussion. perspective and not your own. D. Creating a win-win communication situation. A proven technique for improving interpersonal communication effectiveness is? A. Replace generalizations and evaluative statements with behavior descriptions. Correct Answer: 11. B. Seek first to understand before attempting to be D. All the understood. above. C. Replace hidden agenda or pseudo questions with direct statements. D. All the above. Which is not one of the four communication styles? A. Intuitive Correct Answer: 12. B. Avoiding A. Intuitive C. Aggressive D. Collaborative 3 www.phillipsassociates.com  
    • Which is not a use for paraphrasing? A. Pulling together the important parts of one long Correct Answer: conversation or a series of conversations. A. Pulling together the B. Conveying interest in what another person has to important parts 13. say. of one long C. Testing for understanding when accuracy and conversation or clarity of communication is important. a series of conversations. D. Helping to achieve mutual understanding when strong feelings are present. What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another? A. Arguing Correct Answer: 14. C. Summarizing B. Interrupting C. Summarizing D. Initiating ____________________ are non-verbal cues such as head nods and brief verbal comments such as “right”, “okay” and “umhum”. A. Minimal encouragements Correct Answer:15. A. Minimal B. Communication fillers encouragements C. Confirmations D. Acknowledgements According to Robert Merabian, what percentage of interpersonal communication is transmitted non-verbally through tone of voice and body language? A. 90 percent. Correct Answer: 16. A. 90 percent B. 60 percent C. 75 percent D. 50 percent   4 www.phillipsassociates.com  
    • Level 2 Multiple Choice Test Question Job Aid % of live classroom vs. technology based programs evaluated at Levels 1-5 * Live Classroom Technology Based Level 1 81% ______% Level 2 50% ______% Level 3 25% ______% Level 4 14% ______% Level 5 6% ______% Level 2 evaluation facts * • 81% of organizations evaluate some learning programs at Level 2 • 55% of organizations view data collected as having high or very high value Multiple Choice Test Item Pros ** • Versatility in measuring all levels of cognitive ability • Highly reliable • Easy to score • Minimizes guessing factor Multiple Choice Test Item Cons ** • Difficult and time consuming to create • Lead to creation of test items that measure recall of simple facts 5 www.phillipsassociates.com
    •  Multiple choice test creation tips **   • Avoid use of “All the above” “None of the above” • If administering electronically, use diminishing response technique • Write stem as question – response can finish question but don’t use fill-in-the- blank • Test for understanding not just knowledge • Keep response options equivalent in length and writing style • Include central idea and most of phrasing in stem • Question should not reveal correct answer • Be sure there is only one correct response • Spread correct answers across A – D • All response choices must be plausible • Eliminate excessive verbage or irrelevant information from stem • Write questions same way material taught – avoid “null” questions* ASTD research study, “The Value of Evaluation: Making Training Evaluations MoreEffective,” 2009.** Adapted from ASTD Test Design and Delivery Certificate program participantworkbook, 2006. 6 www.phillipsassociates.com
    • Training Resources Available from Phillips AssociatesInsightful Learning ProductsTurnkey training resources for organizations and individualsPhillips Associates offers an array of learning instruments, multi-rater assessment systems, handbooks (job aids),manuals and games designed to develop specific skills in the areas of performance management and salesperformance. Some of our best-sellers include:No Pain—All Gain! Performance Coaching to Win: Coaching Meeting InventoryAppraisal Meeting Inventory The Coaching Meeting Inventory assesses a manager’sHelp managers conduct performance strengths and development needs in eight skill areas andappraisal meetings that improve outlines a logical step-by-step process for conductingemployee performance and are effective coaching discussions.viewed as valuable and worthwhileusing the Performance Appraisal Superior Service Starts Here: Customer Service SkillsMeeting Inventory. The instrument Inventory (CSSI)pinpoints a manager’s strengths and development needs CSSI consists of 32 statements divided into four keys ofin six skill areas and outlines a simple six-step process providing superior customer service: Developing afor conducting effective performance appraisal meetings. relationship of Interpersonal Trust, Listening, Problem Solving and Professionalism. Based on how they answerGetting On the Same Page: Objective Setting the statements, customer service reps gain valuableHandbooks insight into their greatest strengths and most pressingEasy to read booklets (one for managers and one for development needs in delivering superior customerdirect reports) filled with proven tips and techniques on service.transforming the objective setting process from littlemore than an annual ritual into a valuable, worthwhile Taking Stock for Better Communication:experience for everyone involved. Includes a set of Communication Style Inventory (CSI)worksheets the reader can use to prepare a set of CSI consists of 24 sets of paired statements describingobjectives. the four most commonly used communication styles:Acting the Part: Performance Management Role Plays Aggressive, Avoiding, Accommodating and Collaborative.Ready-to-use role play scenarios designed to help By choosing the statement that most nearly describesmanagers, supervisors and team leaders apply and how they would communicate in a particular situation,practice the skills needed for coaching effectively and respondents generate a profile of their use of the fourconducting productive performance appraisals. communication styles.Includes half-day workshop design and PowerPoint slides.For descriptions of all Phillips Associates Learning Products, visit www.phillipsassociates.comPERFORMANCE MANAGEMENT SALES PERFORMANCE PRODUCTS MULTI-RATER ASSESSMENT PRODUCTSPRODUCTS • Face-to-Face Sales Skills Inventory • Performance Management Profile• Performance Appraisal Meeting System Inventory • Cross-Selling Skills Inventory • Territory Management Style • Selling Skills Profile System• Managing Your Own Performance Appraisal Inventory Inventory FOUNDATION SKILLS PRODUCTS • Communication Style Inventory• Coaching Meeting Inventory • Customer Service Skills Inventory• Keys to Effective Coaching• Objective Setting Inventory Browse and buy online• Interpersonal Feedback Inventory from the PHILLIPS ASSOCIATES• Objective Setting Games Training Resources Online Catalog• Performance Management Role Plays www.phillipsassociates.com• Objective Setting Handbooks Save 10% on your entire order• Performance Appraisal Handbooks when you mention this code: CELTS811 (offer expires 9/16/11)
    • Performance Management and Sales Performance ProgramsIn addition to our full offering of learning products, • Exceeding Expectations: Managing the PerformancePhillips Associates delivers interactive programs to help Appraisal Processemployees achieve and maintain better on-the-job • From Basics to Best in Class: Customer Service Skillsperformance. Our highly participative approach teachesmore than just concepts and practices—it builds lasting • Coaching to Win: Coaching Employee Performanceskills. Some of our most popular programs include: For a complete list of programs offered, please visit us at www.phillipsassociates.com/seminars. The “How-To” Measurement & Evaluation Series Only Phillips Associates shows you how! Presentations (75-90 minute learning programs with audience participation) • Learn 12 Surprising Techniques for Obtaining Powerful Data from Level 1 Evaluations • Crack the Code of Test Question Design and Take Your Level 2 Evaluations to 2.5! • How to Capture Elusive Level 3 Data: The Secrets of Survey Design Workshop (1 or 2 day presentation with participant interaction) • Demonstrate Value and Get Your “Seat at the Table” 51 Tips and Best Kept Secrets for Designing Level 1 – 4 Evaluations For more information contact Ken Phillips at 847.231.6068 or ken@phillipsassociates.com 34137 N. Wooded Glen Drive | Grayslake, IL 60030 847.231.6068 | Info@phillipsassociates.com www.phillipsassociates.com I N T E R N AT I O N A L D I S T R I B U T O R S CANADA UNITED KINGDOM Training & Development Materials of Management Learning Resources, Ltd. Canada PO Box 28, Carmarthen 372 Moonstone road east, PO Box 38 Dyfed SA31 1DT Moonstone, ontario L0K 1N0 Wales, United Kingdom 800.283.4964 01267-281661