Successfully reported this slideshow.

Quality Criteria Test Questions


Published on


Published in: Education, Technology
  • Be the first to comment

Quality Criteria Test Questions

  1. 1. Bismarck Public Schools Being “TestWise”… Ensuring Quality Assessment [email_address] November, 2011
  2. 2. Ensuring Quality Assessments Questions… Being TESTWISE <ul><li>Test questions should follow the following criteria: </li></ul><ul><li>Be matched to the standard </li></ul><ul><li>Offer an opportunity for students to learn </li></ul><ul><li>Be free from bias </li></ul><ul><li>Are written at appropriate levels </li></ul><ul><li>Are rigorous and reliable </li></ul>(Nebraska Department of Education and Buros Center for Testing)
  3. 3. Quality Selected Response Items <ul><li>True and False </li></ul><ul><li>Matching </li></ul><ul><li>Multiple Choice </li></ul>
  4. 4. True/False <ul><li>Related to a single idea </li></ul><ul><li>Absolutely true.. no shades of gray </li></ul><ul><li>Absent of qualifiers/opinions, negatives </li></ul><ul><li>“ Each of the following are true EXCEPT ”…….. “Which is NOT the answer” </li></ul><ul><li>Use sparingly, as students have a 50-50 chance of “guessing the correct answer” </li></ul><ul><li>Test companies advise against use of T/F </li></ul><ul><li>Stems must be clear and unambiguous. </li></ul><ul><li>Alleviate vague phrases like “is useful for”, “is important for”, “may be or could be”, “usually/frequently” </li></ul>
  5. 5. <ul><li>Write true or false before each sentence. </li></ul><ul><li>_____ 1. The town we live in is a suburb. </li></ul><ul><li>_____ 2. An urban area has very few people. </li></ul><ul><li>_____ 3. A rural area often has farms. </li></ul><ul><li>_____ 4. Chicago is an urban area. </li></ul>
  6. 6. Matching <ul><li>Homogenous in content </li></ul><ul><li>Matching set no more than 7 </li></ul><ul><li>Uneven number of items to be matched OR items may be used more than once </li></ul><ul><li>Ordered logically </li></ul>
  7. 7. <ul><li>Match the definition on the right to the vocabulary word on the left. </li></ul><ul><li>Write the correct letter on the blank in front of the word. </li></ul>
  8. 8. Multiple-choice <ul><li>A single task is articulated in the question </li></ul><ul><li>Begin with question-starter (who, what, where, how, why) </li></ul><ul><li>All response options are plausible (no throw-aways) </li></ul><ul><li>Write them in the present tense </li></ul><ul><li>Avoid negatives (e.g., which is NOT) </li></ul><ul><li>Avoid using absolutes like always, never , and all in the options </li></ul>
  9. 9. Comprehension in Multiple Choice <ul><li>One reason teachers dislike multiple choice is the belief these items are only good for measuring simple recall of facts. (low level recall of vocabulary and facts) </li></ul><ul><li>The real value of multiple choice, however, is application of higher-level thinking to include comprehension , application , and analysis. </li></ul>
  10. 10. <ul><li>Poor Example of a multiple choice question: </li></ul><ul><li>Idaho is widely known as: </li></ul><ul><li>A. The largest producer of potatoes in the US </li></ul><ul><li>B. The location of the tallest U.S. mountain </li></ul><ul><li>C. The state with a beaver on its flag. </li></ul><ul><li>D. The “Treasure State” </li></ul>Why is it poor?
  11. 11. <ul><li>Better Example: </li></ul><ul><li>Idaho is widely known for its: </li></ul><ul><li>A. Apples. </li></ul><ul><li>B. Corn. </li></ul><ul><li>C. Potatoes. </li></ul><ul><li>D. Wheat. </li></ul>Why is this better?
  12. 12. <ul><li>A word used to describe a noun is called an: </li></ul><ul><li>a. Adjective. </li></ul><ul><li>b. Conjunction. </li></ul><ul><li>c. Pronoun. </li></ul><ul><li>d. Verb. </li></ul><ul><li>A word used to describe a noun is called… </li></ul><ul><li>a. An adjective. </li></ul><ul><li>b. A conjunction. </li></ul><ul><li>c. A pronoun. </li></ul><ul><li>d. A verb. </li></ul>
  13. 13. <ul><li>Avoid grammatical cues: one or more distractors don’t follow </li></ul><ul><li>grammatically from the stem . </li></ul><ul><li>Stem: A 60 year old man, who was found on the sidewalk unconscious, is brought to the emergency room by the police. After opening an airway, the first step in management should be intravenous administration of </li></ul><ul><li>(Distractors:) </li></ul><ul><li>a. Examination of spinal fluid </li></ul><ul><li>b. Glucose with vitamin B1 </li></ul><ul><li>c. CT scan of the head </li></ul><ul><li>d. Phenytoin </li></ul><ul><li>e. diazepam </li></ul>
  14. 14. <ul><li>(STEM :) Use the list of numbers below to answer the </li></ul><ul><li>question. </li></ul><ul><li>22, 25, 14, 11, 23, 27, 46, 23, 24, 35 </li></ul><ul><li>(Lead-in:) What is the mode? </li></ul><ul><li>(Distractors) </li></ul><ul><li>A. 23 </li></ul><ul><li>B. 24 </li></ul><ul><li>C. 35 </li></ul><ul><li>D. no mode </li></ul>
  15. 15. Preparing for Better Multiple-Choice Test Items <ul><li>Multiple True/False : </li></ul><ul><li>A virus: </li></ul><ul><li>T F Can cause disease. </li></ul><ul><li>T F Can reproduce by itself. </li></ul><ul><li>T F Is composed of large living cells. </li></ul><ul><li>T F Lives in plant and animal cells. </li></ul>
  16. 16. Advantages/Disadvantages of… Multiple Choice Questions <ul><li>Advantages: </li></ul><ul><li>Reliability: </li></ul><ul><li>Less susceptible to guessing like true and false. </li></ul><ul><li>Scoring is more clear cut </li></ul><ul><li>Not affected by scorer inconsistencies as in an essays </li></ul><ul><li>Immune to the influence of student “bluffing” </li></ul>
  17. 17. Advantages/Disadvantages of… Multiple Choice Questions <ul><li>Disadvantages: </li></ul><ul><li>Multiple choices limits the student’s ability to: </li></ul><ul><li>Articulate what they have learned </li></ul><ul><li>Organize personal thought process </li></ul><ul><li>Perform a task </li></ul><ul><li>Produce original ideas </li></ul><ul><li>Provide examples </li></ul>
  18. 18. And the Research says? <ul><li>In 46 authoritative references in educational measurement, 35 authors agreed that the multiple choice should have only one correct answer. (Haladyna & Downing, 1989) </li></ul><ul><li>Other conclusions: </li></ul><ul><li>Multiple choice are found to have lower reliability and higher in difficulty (Frisbie, 1990) depending how they are written </li></ul>
  19. 19. Checklist for reviewing multiple choice items <ul><li>Has the item been constructed to access a single learning objective? </li></ul><ul><li>Is the item stated clearly in the stem? </li></ul><ul><li>Does the stem avoid including irrelevant material? </li></ul><ul><li>Is the stem stated in a positive form? </li></ul><ul><li>Are the distractors homogeneous in content? </li></ul><ul><li>Are all distractors feasible? </li></ul><ul><li>Has unnecessary difficult vocabulary been avoided? </li></ul><ul><li>Are the grammar, punctuation, and spelling correct? </li></ul>
  20. 20. Quality Constructed Response Items
  21. 21. Fill-in-the-blank <ul><li>Position the blank at the end of the sentence when possible </li></ul><ul><li>Limit the number of blanks in an item </li></ul><ul><li>Blanks should be same length </li></ul><ul><li>Be sure information prior to/surrounding the blank is adequate </li></ul><ul><li>May use a word bank </li></ul>
  22. 22. <ul><li>Find is to locate as under is to _______________. </li></ul><ul><li>Lincoln is to _______________ as _______________ is to Wisconsin. </li></ul>
  23. 23. <ul><li>You are creating a display in a museum about a native American culture. Choose one artifact from each of the seven tribes studied to include in your display. Write a brief paragraph that includes each of the following: </li></ul><ul><li>the artifact. </li></ul><ul><li>how the artifact was used. </li></ul><ul><li>how the artifact reflects the way in which that culture adapted to its environment. </li></ul><ul><li>Scoring Guide : Each component is worth one point for a total of three points per tribe (21 points for the entire question). </li></ul>
  24. 24. <ul><li>YOUR TURN! </li></ul>
  25. 25. Back at the ranch……. <ul><li>Review your assessment items. </li></ul><ul><li>Are there any you might revise due to the format? </li></ul><ul><li>Make note of that, and revise accordingly. </li></ul>cutting-edge research concrete strategies sustainable success
  26. 26. “ Fairness is compromised when teachers assess something that hasn’t been taught or use assessment methods that are incongruent with instruction.” cutting-edge research concrete strategies sustainable success — Jay McTighe, NEA, Professional and Standards and Practice, (1994)
  27. 27. Free From Bias <ul><li>Pervasive reductions in performance can be caused by perceptions of unfairness in the test questions or offensiveness that harm motivation or attention on later test items. </li></ul><ul><li>Bias is the presence of some characteristics of an item that results in differential performance for two individuals of the same ability but from different subgroups. </li></ul>cutting-edge research concrete strategies sustainable success
  28. 28. Types of Bias <ul><li>Offensive content </li></ul><ul><li>Stereotyping </li></ul><ul><li>Unfair representation </li></ul><ul><li>Use of situations that may be unfamiliar to subgroups </li></ul><ul><li>Poorly written items </li></ul><ul><li>Literacy bias (using too sophisticated of language for the nature of the content)—David Meisner. </li></ul>cutting-edge research concrete strategies sustainable success
  29. 29. Use of Unfamiliar Situations cutting-edge research concrete strategies sustainable success
  30. 30. Offensive Content cutting-edge research concrete strategies sustainable success Last year 200 women and children in our community were served by the Safe Haven shelter. Due to the rise of domestic abuse, the shelter is experiencing a 30% increase in residents. Current food costs are listed below. Project the anticipated funds needed for food for the remainder of the year.
  31. 31. Stereotyping <ul><li>Maria Yellowhawk needs to purchase beads for necklaces she is making. She has 3 dozen packages of beads, each containing twelve beads. If each necklace requires 120 beads, how many necklaces can she make? </li></ul>cutting-edge research concrete strategies sustainable success
  32. 32. Unfair Representation <ul><li>Graph the following salaries of employees of the Acme Chemical Company: </li></ul><ul><li>Roger Smith, CEO $250,000 </li></ul><ul><li>Mary Kelly, Secretary $ 17,500 </li></ul><ul><li>Dr. Frank Hatfield, Chemist $ 98,000 </li></ul><ul><li>Francisco Juarez, Custodian $ 24,000 </li></ul>cutting-edge research concrete strategies sustainable success