Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cets 2013_ gander hale_write_test_questions_session

508 views

Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Cets 2013_ gander hale_write_test_questions_session

  1. 1. WRITING TEST QUESTIONS FOR ANY TEST ENGINE Sharon Gander, M.Ed., CPT Judith Hale, Ph.D., CPT
  2. 2. OVERVIEW  Overview and presenter introductions  Test engine challenges  Conditions & scenarios  Questions and responses  Quality Distractors  Quality checks  Test Engine Debrief THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 6
  3. 3. TAKE TIME TO: 1 – OPEN A WORD PROCESSOR 2 – OPEN OR DOWNLOAD A TEST ENGINE 3 – DRAFT 2-3 TEST QUESTIONS ON ANY TOPIC 4 – CHECK THAT YOU HAVE RECEIVED THE HANDOUTS 3 – WORK WITH A PARTNER ON ACTIVITIES – FIND YOUR PARTNER NOW! Prepare for this session by doing the steps above!
  4. 4. OBJECTIVES 1. Write a one-paragraph scenario opening to a question. 2. Write a question with one right answer. 3. Write three plausible answers and one correct answer to the posed question. 4. Identify strategies for validating the preferred response to posed question. 5. Check answers for “tells” that give away the right or wrong answer. 6. Explain the value of using Plain Language and Global English in writing test questions. 7. Leave with three tools to apply in writing test items for learning or certifications. 8. Recognize test engine characteristics that impact item design, delivery and validation. 9. Prototype one or more questions in a free test-engine. THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 8 Immediately after this session, begin building scenario-based questions for courses, certifications and test-outs as opportunities arise.
  5. 5. TEST ENGINE CHALLENGES What challenges do you experience in writing test items for online test- engine delivery?
  6. 6. TEST ENGINE CHALLENGES  Developer familiarity with test engine while writing  The need to write clear questions and responses  It is difficult to get valid content  Need to overcome the strong tendency to write recall (knowledge/facts)  Difficult to write questions that make sense to the test candidate  Continuous error checking dulls the senses  Difficult to use appropriate question type THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 10
  7. 7. CONDITIONS AND SCENARIOS How do you create performance- based test questions that work in any test engine?
  8. 8. PERFORMANCE-BASE QUESTIONS The heart of a performance-based question is a condition. You could use:  Given a specific situation and resources… Or  Describe the situation in 1-2 paragraph story (a scenario) THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 12
  9. 9. Given the fact that test engines and survey engines handle question sets and answer evaluation very differently and the fact that you need to produce quality assessment questions as quickly as possible. Which two question types are most standard across all engines? a) True/false and multiple-choice. b) Fill-in-the-blank and ordering. c) Multi-select and ranking. d) Essay and matching. 13THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
  10. 10. SOAR – A SCENARIO TECHNIQUE Situation – essential facts of the setting Obstacles – the hurdles that will prevent action or require unique actions; problems and challenges Action – the key decisions and/or action steps are required Result – the desired outcome THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 14
  11. 11. ACTIVITY #1: PREPARE QUESTION 1. Think of a situation of your own (or use the one below). 2. Describe the conditions using your SOAR template 3. Work with partner – 3 minutes Write for the action. What action will overcome the obstacle to gain the desired result THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 15 Can’t think of a situation? Try this: You are tasked with assessing SME’s skills for writing content. Your organization wants SMEs to apply adult learning theory to content that will be used in e-learning solutions. Describe a scenario that will assess this skill.
  12. 12. QUESTIONS AND RESPONSES Why use multiple-choice questions with only one correct response?
  13. 13. WHY ONE CORRECT ANSWER? Clarity  Decreases time readers spend figuring out what the question is  Decreases the time spent guessing at what the question writer intended Defensibility  Can be supported by unambiguous documented references  Provides clear evidence of knowledge when answering knowledge-type questions  Provides clear evidence of performance when answering performance-based questions Technology  Scoring and tracking answers  Works with every test engine THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 17
  14. 14. WHAT IS A CLEAR QUESTION? One that gives the reader clear signals about “what to answer” by:  Starting with a question word (5Ws/H)  Who, what, when, where, why, or how  Ending with a question mark  Avoiding the hanging question that results from using a colon that leaves the reader trying to read your mind  Using only one verb  Using action verbs  Using either the present or past tense  Avoiding the passive “is” “was”  Using either an actor, “you” or implied “you” THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 18
  15. 15. ACTIVITY #2: WRITE A QUESTION 1. Using your SOAR scenario and the item writing template 2. Write a question with ONE right answer More about distractors and incorrect responses coming soon. 3. Work with partner – 3 minutes THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 19 Write for the action. Find a way to ask, what action will overcome an obstacle to gain the desired result?
  16. 16. WHAT IS A QUALITY DISTRACTOR Should one response be obviously wrong? Should I use the “ none of the above” or “all of the above responses? Should I use answers like “a and c only”?
  17. 17. QUALITY DISTRACTORS Are plausible – all of them are plausible  Real and real-life Force discrimination  Requires test-taker to evaluate options Separate the less experienced from the highly experienced  Requires test-taker to have experience (and education) to choose the most effective response Technically accurate  May distract with variance in technical details  Forces accuracy in technical aspects THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 21
  18. 18. MULTIPLE CHOICE RESPONSES One Correct Response  With documented reference of correctness  Not just an experts “feeling” that it’s right Three Plausible Responses  One mistake that most novices make (a common error)  One mistake that soon-to-be experts tend to make  One mistake that even many experts make THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 22
  19. 19. ANSWER VALIDATION STRATEGIES Need to validate your preferred response?  Document references that validate the correct response (#1)  Ask an acknowledged expert to write a white paper on the subject  Poll experts and write a white paper about the polled results  Give question to experts and novices; determine which answer the experts choose  Other? Anything that documents a preferred response other than “I believe this to be true.” THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 23
  20. 20. In a multiple choice question, the optimal answer set is four answers with one right answer. What should the other three answers be? a) Two wrong answers plus one “all correct” answer. b) Two plausible answers plus one obviously wrong answer. c) One plausible answer, one wrong answer, plus an “all correct” answer. d) Three plausible answers typical of different skill levels. 24THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
  21. 21. ACTIVITY #3: PLAUSIBLE RESPONSES 1. Write 3 plausible and 1 correct response to your question 2. Enter your question in your test engine. 3. View your question in the test engine as though you were a test- taker. 4. Work with partner – 5 minutes THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 25 Checkpoint Is there a difference between what you planned to write and what the test engine allowed or how the question displayed? Why?
  22. 22. QUALITY ASSURANCE How do you check your work before publishing it?
  23. 23. BASIC QUALITY ASSURANCE • Grammar, spelling, punctuation • Looking for “tells” (hints that give away the correct response) • Length of responses • Change in verb tense • Change in actor • Change in technical terms • Change in punctuation • Ensure that the question stem • Uses only one action verb • Starts with question work (5Ws/H) • Ends in question mark THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 27
  24. 24. Key words, changes in grammar style and verb tense can provide “tells” (hints to the right answer) among the plausible distractor answers. Which of the following techniques would also prevent a “tell” in your answer set? a) Use mainly very short one or two word answers. b) Use very precise technical terminology or phrases. c) Use answers that are the relatively the same length. d) Use only present tense verbs in plausible answers. 28THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
  25. 25. In a multiple choice question, the optimal answer set is four answers with one right answer and: a) 3 wrong answers b) 2 plausible answers +1 obviously wrong answr c) Use one plausible answer, onw very wrong answer, and one “all correct” answer. d) Three plausible answers typical of different skill levels. 29THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) What does the question really test? WHAT QA ISSUES?
  26. 26. ADVANCED QUALITY ASSURANCE Plain Language • Active voice (avoid “is” and “was”) – go for the action • Use strong clear noun – a person, place or thing (avoid “it” “they”) • Use simple sentence structures – end your thought with a period and limit the number of comma and colons. • Use “you” or the implied “you” command when the reader is required to take action • Spell out acronyms • Avoid jargon • Organize your points Global English • Avoid the use of country specific terms (e.g., USDA, Rte 66, M1, Kindergarten) • Describe common situations rather than using local terms (e.g., tsunamis, School District, US green building code) Reading Level (approx. 9th grade) • Watch the long names with many capitalized nouns in a row • Put periods at end of bullet-points • Avoid jargon • Limit use of technical terms • Use short paragraphs THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 30
  27. 27. Your current project must provide an assessment with many questions that will be used in many countries by professionals with many language preferences. What is the strongest reason for using Plain Language and Global English in writing test questions? a) Your question items will all be in one verb tense for reading ease. b) Your questions will be clear to anyone who reads English. c) Your answers are researched, justifiable and measurable. d) Your work can be used internationally without translation. 31THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
  28. 28. How did the test engine affect your question? What happened to your quality assurance when you entered your question into the test engine? What test engine issues did you experience? Did you put your questions in a work processor first? If so, how did that impact your results? TEST ENGINE IMPACT
  29. 29. ERROR PREVENTION Use a QA Checklist like the one provided  Check the writing, spelling and grammar Review questions (offline) with an expert audience who has not seen them • Do they validate the question, the correct answer and the validation resource (text, website, document) Load your items into the online format  Does the item display correctly?  Does the answer correctly evaluate as right or wrong?  Does the answer/score transfer to the LMS correctly? Run an online pilot with both expert and non-experts candidates  Feedback on questions and test process  Cut scores Modify the test as needed THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 33
  30. 30. YOU HAVE HAD THE CHANCE TO: 1. Write a one-paragraph scenario opening to a question. 2. Write a question with one right answer. 3. Write three plausible answers and one correct answer to the posed question. 4. Identify strategies for validating the preferred response to posed question. 5. Check answers for “tells” that give away the right or wrong answer. 6. Explain the value of using Plain Language and Global English in writing test questions. 7. Leave with three tools to apply in writing test items for learning or certifications. 8. Recognize test engine characteristics that impact item design, delivery and validation. 9. Prototype one or more questions in a free test-engine. THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 34
  31. 31. In the next 7 days, write at least 5 scenario-based, multiple-choice questions with 1 correct and 3 plausible responses. QA them with the checklist and enter them into a test engine. THE CHALLENGE!
  32. 32. WRITING TEST QUESTIONS FOR ANY TEST ENGINE Sharon Gander, M.Ed., CPT Judith Hale, Ph.D., CPT Contact us: Sharon Gander at performancepi.gander@gmail.com Judy Hale at judy@tifpi.org
  33. 33. THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 37 SESSION EVALUATION Please take the time to provide us with your session feedback using any of the following methods. QR Code 1. Download a QR reader to your Smartphone or tablet 2. Scan the QR code from this slide, your session evaluation sign, or your program 3. Launch and complete the evaluation URL Address 1. Locate the URL for this session from this slide, you session evaluation sign, or your program. 2. Enter the URL Code for this session into the browser of your laptop, Smartphone or tablet 3. Launch and complete the evaluation URL: https://www.questionmark.com/cets07

×