Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Item Analysis, Design and Test Formats


Published on

Published in: Education, Technology

Item Analysis, Design and Test Formats

  1. 3. <ul><li>Item: a question used to test a piece of knowledge. </li></ul><ul><li>Instruction(s): An explanation of how to complete a task. It contains a meaningful situation and the actual task to perform. </li></ul><ul><li>Stem: The linguistic environment that works as the context to elicit the right answer in multiple choice (MC), gap-filling, and other objective formats. In MC items, it leads into the cue (key + distracters) </li></ul><ul><li>Cue: The MC item’s response options, usually 4. </li></ul><ul><li>Key: The correct choice in MC. </li></ul><ul><li>Distracters (or distractors): The response options in MC that are possible but incorrect . </li></ul>
  2. 4. <ul><li>Poor distracter </li></ul><ul><li>A response option selected by less than 10 percent of the respondents to the items. Under-selected distracters indicate either that the distracters were ineffective or that the content areas behind the distracters were well understood by the respondents so that these distracters were not “distracting” at all. </li></ul><ul><li>Standard MC item </li></ul><ul><li>t he format describing an item that includes one simple positively stated stem that directly leads to the response options. </li></ul>
  3. 6. <ul><li>Before you start designing your items, answer the following 2 questions: </li></ul><ul><li>What am I going to assess? (aspects of language) </li></ul><ul><li>How am I going to assess it? (test formats) </li></ul><ul><li>Write your items considering the following 4 aspects: </li></ul><ul><li>Task (behavior that exhibits the knowledge to be assessed. Make it realistic -similar to real life tasks) </li></ul><ul><li>Context (meaningful) </li></ul><ul><li>Instructions (clear and concise. Examples if necessary) </li></ul><ul><li>Stem (unambiguous elicitation of the correct answer) </li></ul><ul><li>Options (cues and key well taken care of). </li></ul>
  4. 8. <ul><li>Depending on the nature of the content being assessed, choose the most appropriate test format </li></ul><ul><li>READING AND LISTENING </li></ul><ul><li>Gap-filling (open/with options) </li></ul><ul><li>Multiple choice </li></ul><ul><li>Matching </li></ul><ul><li>True/false (don’t know/not mentioned) </li></ul><ul><li>Open-ended questions </li></ul><ul><li>Closed questions </li></ul><ul><li>Table completion (information transfer) </li></ul><ul><li>Linking (for reference words) </li></ul><ul><li>Discrepancies </li></ul><ul><li>Listing/Sequencing </li></ul>
  5. 9. <ul><li>WRITING AND SPEAKING </li></ul><ul><li>Essay questions </li></ul><ul><li>Guided writing </li></ul><ul><li>Punctuation </li></ul><ul><li>Summary </li></ul><ul><li>Note-taking </li></ul><ul><li>Dictation (?) </li></ul><ul><li>Free interview </li></ul><ul><li>Picture description </li></ul><ul><li>Information transfer </li></ul><ul><li>Role-play </li></ul><ul><li>Oral presentation </li></ul>
  6. 10. <ul><li>LANGUAGE COMPONENTS </li></ul><ul><li>Cloze test -Ctest (blank every n th word) </li></ul><ul><li>Gap-filling </li></ul><ul><li>Sentence arrangement </li></ul><ul><li>Sentence writing </li></ul><ul><li>Table completion </li></ul><ul><li>Matching sentence halves </li></ul><ul><li>Matching words/definitions </li></ul><ul><li>Multiple choice </li></ul><ul><li>Open-ended questions </li></ul><ul><li>Closed questions </li></ul><ul><li>Dialogue completion </li></ul>
  7. 12. <ul><li>Problem: For every item, all cues must be grammatically and/or semantically consistent and homogeneous. </li></ul><ul><li>Wrong: John _________ flowers to the party last night. </li></ul><ul><li>Carries </li></ul><ul><li>Carried </li></ul><ul><li>Lifts </li></ul><ul><li>Lifted </li></ul><ul><li>Better: John _________ flowers to the party last night. </li></ul><ul><li>Carries </li></ul><ul><li>Carried </li></ul><ul><li>Has carried </li></ul><ul><li>Is carrying </li></ul>
  8. 13. <ul><li>Problem: The stem should not introduce the correct option. </li></ul><ul><li>Wrong: He wrote an _________ in his laptop. </li></ul><ul><li>Letter </li></ul><ul><li>Post card </li></ul><ul><li>E-mail </li></ul><ul><li>Memo </li></ul><ul><li>Better: He wrote a brief _________ in his laptop. </li></ul><ul><li>Letter </li></ul><ul><li>Post card </li></ul><ul><li>E-mail </li></ul><ul><li>Memo </li></ul>
  9. 14. Problem: In communicative tests, instructions should not be based on grammar, but on meaningful contexts. Wrong: Write sentences using make, let, and be allowed. Better: You got admitted in a university away from hometown so you have to move to a student residence. Write sentences expressing the things that are prohibited or permitted in the residence (you may have to use make, let , and be allowed ).
  10. 15. <ul><li>Problem: The cues in a multiple-choice format should be all grammatically possible. </li></ul><ul><li>Wrong: Katherine is a very _________ student. She gets nothing but straight A’s. </li></ul><ul><li>Smartly </li></ul><ul><li>Intelligent </li></ul><ul><li>Preoccupied </li></ul><ul><li>Considerate </li></ul><ul><li>Better: Katherine is a very _________ student. She gets nothing but straight A’s. </li></ul><ul><li>Thoughtful </li></ul><ul><li>Intelligent </li></ul><ul><li>Preoccupied </li></ul><ul><li>Considerate </li></ul>
  11. 16. Problem: It is never suggested to use made-up words or grammatical structures as distracters. Wrong: People say that my children are malraised/spoiled because I buy them whatever they ask for. Better: People say that my children are consented/spoiled because I buy them whatever they ask for.
  12. 17. <ul><li>Problem: Language tests assess overall language proficiency, not cognitive skills or creativity. </li></ul><ul><li>Wrong: Unscramble the jumbled letters below to form English words. </li></ul><ul><li>MEAX </li></ul><ul><li>GMAEUZ </li></ul><ul><li>TAC </li></ul><ul><li>ALAVGI </li></ul><ul><li>Better: Unscramble the jumbled words below to form sentences expressing frequency. </li></ul><ul><li>gym often go I the to _____________________________ </li></ul><ul><li>she ever television hardly watches___________________ </li></ul><ul><li>my mom home never at is _________________________ </li></ul><ul><li>cooks Franklin dinner always_______________________ </li></ul>
  13. 18. Problem: The difficulty in an item should lie in the task itself, not in the instructions or the stem. Wrong: When a police officer arrests someone, the officer must inform the person of a certain rights, including the right to remain silent and the right to an attorney. Why, according to the text, is this required? Better: According to the text, why must police officers inform suspects of their rights during an arrest?
  14. 19. <ul><li>Problem: Review options require considerable concentration (is this what you are assessing?) and focus on the small information presented in the previous opnes. Avoid such options (d, e, f, g) </li></ul><ul><li>Wrong: The person in the listening said he had left his land because he… </li></ul><ul><li>Wanted to seek his fortune </li></ul><ul><li>Wished to avoid his creditors </li></ul><ul><li>Preferred the new land </li></ul><ul><li>All of the above </li></ul><ul><li>None of the above </li></ul><ul><li>A and B, but not C </li></ul><ul><li>B and C, but not A </li></ul><ul><li>Better: The person in the listening said he had left his land because he… </li></ul><ul><li>Wanted to seek his fortune </li></ul><ul><li>Wished to avoid his creditors </li></ul><ul><li>Preferred the new land </li></ul><ul><li>Had his family abroad already </li></ul>
  15. 20. <ul><li>Problem: All cues should be about the same length. Another option is having two short cues and two long ones. </li></ul><ul><li>Wrong: Based on the recording, what is the best way to reduce inflation? </li></ul><ul><li>Cutting on income taxes. </li></ul><ul><li>Increasing government expenses. </li></ul><ul><li>Reducing the number of bank operating all over America. </li></ul><ul><li>Raising local interest rates. </li></ul><ul><li>Better: Based on the recording, what is the best way to reduce inflation? </li></ul><ul><li>Cutting on income taxes. </li></ul><ul><li>Increasing government expenses. </li></ul><ul><li>Reducing the federal deficit. </li></ul><ul><li>Raising local interest rates. </li></ul>
  16. 21. Problem: The correct answer should be based on the content of a passage and not on previous knowledge or common sense. Wrong: Read the text and say if these sentences are TRUE or FALSE. a) The three basic principles to be environmentally friendly are reduce, reuse and recycle . ______   Better: Read the text and answer the following questions. a) What are the three basic principles to be environmentally friendly? _________________________________________
  17. 23. <ul><li>Once the item is designed, it is important to determine its appropriateness in terms of… </li></ul><ul><li>VALIDITY: Items test what we want to test. Test-takers are familiar with the format. </li></ul><ul><li>RELIABILITY: Items’ results are consistent and replicable. </li></ul><ul><li>PRACTICALITY: Items are easy to complete and correct. </li></ul><ul><li>BACKWASH: Items are meaningful and motivating. </li></ul>
  18. 24. An item is VALID if it assesses authentic communication and interaction, and also reflects what students do in class. It has POSITIVE BACKWASH if it adds value to the educational process due to its desirable impact on learning. Wrong:   Dictation . Listen to the instructions to start a weblog and write the information on the space provided. Then translate the information into your native language. Better: You want to create a weblog and a friend is giving you the information to set up an account as the 1 st step. Listen to the instructions and fill in the missing information to learn how to start your own blog.
  19. 25. An item is RELIABLE if it measures students’ performance consistently. It is PRACTICAL if it takes little time to design, organize and mark, and involves resources that are easily available. Wrong: In your opinion, how can computers help us? _______________________________________________________________________________________________________________________________________ Better: Mention three ways in which computers can help firefighters in case of an emergency. 1)____________________________________________ 2)____________________________________________ 3)____________________________________________
  20. 26. Integrative and open-ended test formats are usually very VALID and also have a POSITIVE BACKWASH, since they involve communication and interaction in real-life situations. Example: While reading today’s newspaper, you find two interesting articles. Express your opinion about them. However, integrative and open-ended test formats are often impractical (difficult to interpret) and unreliable (open and subjective).
  21. 27. On the other hand, discrete-point test formats are usually very RELIABLE and PRACTICAL because correct answers are limited and they are quick and easy to mark. Example: It is said that pit-bulls are a breed/race of dog that is particularly violent. Nevertheless, discrete item formats frequently lack validity (meaningful communication) and positive backwash (luck is as important as hard work ).