Item Analysis


Published on

Published in: Education, Technology
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Item Analysis

  1. 3. <ul><li>Item: A question used to test a piece of knowledge. </li></ul><ul><li>Instruction: An explanation of how to complete a task. It contains a meaningful situation and the task. </li></ul><ul><li>Stem: The beginning of a multiple choice test item. In multiple choice items, it leads into four response options, made of three distracters and one key. </li></ul><ul><li>Cue: The item’s response options. </li></ul><ul><li>Key: The response option that is correct. </li></ul><ul><li>Distracter: The response option that is not the correct answer </li></ul>
  2. 4. <ul><li>Normal item: The format describing an item that includes one simple positively stated stem that directly leads to the response options. It is the recommended format. </li></ul><ul><li>Not item: The format describing an item with a negatively stated stem. It typically asks for what is “not,” least” or “worst” or use words like “except” or “but.” It should be avoided particularly when participants are taking the test in their native language. </li></ul><ul><li>Poor distracter: A response option selected by less than 10 percent of the respondents to the items. Under-selected distracters indicate either that the distracters were ineffective or that the content areas behind the distracters were well understood by the respondents so that these distracters were not “distracting” at all. </li></ul>
  3. 6. <ul><li>Before you start designing your items, answer the following 2 questions: </li></ul><ul><li>What am I going to assess? (aspects of language) </li></ul><ul><li>How am I going to assess it? (test formats) </li></ul><ul><li>Write your items considering the following 4 aspects: </li></ul><ul><li>Task (aspect assessed) </li></ul><ul><li>Context (meaningful) </li></ul><ul><li>Instructions (clear and concise) </li></ul><ul><li>Stem (question or statement) </li></ul><ul><li>Options (cues and key) </li></ul>
  4. 8. <ul><li>Depending on the nature of the content being assessed, choose the most appropriate test format </li></ul><ul><li>READING AND LISTENING </li></ul><ul><li>Gap-filling (open/with options) </li></ul><ul><li>Multiple choice </li></ul><ul><li>Matching </li></ul><ul><li>True/false (don’t know/not mentioned) </li></ul><ul><li>Open-ended questions </li></ul><ul><li>Closed questions </li></ul><ul><li>Table completion </li></ul><ul><li>Linking </li></ul><ul><li>Discrepancies </li></ul><ul><li>Listing/Sequencing </li></ul>
  5. 9. <ul><li>WRITING AND SPEAKING </li></ul><ul><li>Essay questions </li></ul><ul><li>Guided writing </li></ul><ul><li>Punctuation </li></ul><ul><li>Summary </li></ul><ul><li>Note taking </li></ul><ul><li>Dictation </li></ul><ul><li>Free interview </li></ul><ul><li>Picture description </li></ul><ul><li>Information transfer </li></ul><ul><li>Role-play </li></ul><ul><li>Oral Presentation </li></ul>
  6. 10. <ul><li>LANGUAGE COMPONENTS </li></ul><ul><li>Cloze exercise (blank every th word) </li></ul><ul><li>Gap-filling </li></ul><ul><li>Sentence arrangement </li></ul><ul><li>Sentence writing </li></ul><ul><li>Table completion </li></ul><ul><li>Matching sentence halves </li></ul><ul><li>Matching (words/definitions) </li></ul><ul><li>Multiple choice </li></ul><ul><li>Open-ended questions </li></ul><ul><li>Closed questions </li></ul><ul><li>Dialogue completion </li></ul>
  7. 12. <ul><li>Problem: For every item, all cues must be grammatically and/or semantically consistent and homogeneous. </li></ul><ul><li>Wrong: John _________ flowers to the party last night. </li></ul><ul><li>Carries </li></ul><ul><li>Carried </li></ul><ul><li>Lifts </li></ul><ul><li>Lifted </li></ul><ul><li>Better: John _________ flowers to the party last night. </li></ul><ul><li>Carries </li></ul><ul><li>Carried </li></ul><ul><li>Has carried </li></ul><ul><li>Is carrying </li></ul>
  8. 13. <ul><li>Problem: The stem should not introduce the correct option. </li></ul><ul><li>Wrong: He wrote an _________ in his laptop. </li></ul><ul><li>Letter </li></ul><ul><li>Post card </li></ul><ul><li>E-mail </li></ul><ul><li>Memo </li></ul><ul><li>Better: He wrote a brief _________ in his laptop. </li></ul><ul><li>Letter </li></ul><ul><li>Post card </li></ul><ul><li>E-mail </li></ul><ul><li>Memo </li></ul>
  9. 14. Problem: In communicative tests, instructions should not be based on grammar, but on meaningful contexts. Wrong: Write sentences using make, let, and be allowed. Better: You got admitted in a university away from for hometown so you have to move to a student residence. Write sentences expressing the things that are prohibited or permitted in this context.
  10. 15. <ul><li>Problem: The cues in a multiple-choice format should be all grammatically possible. </li></ul><ul><li>Wrong: Katherine is a very _________ student. She gets nothing but straight A’s. </li></ul><ul><li>Smartly </li></ul><ul><li>Intelligent </li></ul><ul><li>Preoccupied </li></ul><ul><li>Considerate </li></ul><ul><li>Better: Katherine is a very _________ student. She gets nothing but straight A’s. </li></ul><ul><li>Thoughtful </li></ul><ul><li>Intelligent </li></ul><ul><li>Preoccupied </li></ul><ul><li>Considerate </li></ul>
  11. 16. Problem: It is never suggested to use made-up words or grammatical structures as distracters. Wrong: People say that my children are malraised/spoiled because I buy them whatever they ask for. Better: People say that my children are consented/spoiled because I buy them whatever they ask for.
  12. 17. <ul><li>Problem: Language tests assess overall language proficiency, not cognitive skills or creativity. </li></ul><ul><li>Wrong: Unscramble the jumbled letters below to form English words. </li></ul><ul><li>MEAX </li></ul><ul><li>GMAEUZ </li></ul><ul><li>TAC </li></ul><ul><li>ALAVGI </li></ul><ul><li>Better: Unscramble the jumbled words below to form sentences expressing frequency. </li></ul><ul><li>gym often go I the to _____________________________ </li></ul><ul><li>she ever television hardly watches___________________ </li></ul><ul><li>my mom home never at is _________________________ </li></ul><ul><li>cooks Franklin dinner always_______________________ </li></ul>
  13. 18. Problem: The difficulty in an item should lie in the task itself, not in the instructions or the stem. Wrong: When a police officer arrests someone, the officer must inform the person of a certain rights, including the right to remain silent and the right to an attorney. Why, according to the text, is this required? Better: According to the text, why must police officers inform suspects of their rights during an arrest?
  14. 19. <ul><li>Problem: For every item, there should always be one (and only one) correct response. </li></ul><ul><li>Wrong: The person in the listening said he had left his land because he… </li></ul><ul><li>Wanted to seek his fortune </li></ul><ul><li>Wished to avoid his creditors </li></ul><ul><li>Preferred the new land </li></ul><ul><li>All of the above </li></ul><ul><li>None of the above </li></ul><ul><li>A and B, but not C </li></ul><ul><li>B and C, but not A </li></ul><ul><li>Better: The person in the listening said he had left his land because he… </li></ul><ul><li>Wanted to seek his fortune </li></ul><ul><li>Wished to avoid his creditors </li></ul><ul><li>Preferred the new land </li></ul><ul><li>Had his family abroad already </li></ul>
  15. 20. <ul><li>Problem: All cues should be about the same length. Another option is having two short cues and two long ones. </li></ul><ul><li>Wrong: Based on the recording, what is the best way to reduce inflation? </li></ul><ul><li>Cutting on income taxes. </li></ul><ul><li>Increasing government expenses. </li></ul><ul><li>Reducing the number of bank operating all over America. </li></ul><ul><li>Raising local interest rates. </li></ul><ul><li>Better: Based on the recording, what is the best way to reduce inflation? </li></ul><ul><li>Cutting on income taxes. </li></ul><ul><li>Increasing government expenses. </li></ul><ul><li>Reducing the federal deficit. </li></ul><ul><li>Raising local interest rates. </li></ul>
  16. 21. Problem: The correct answer should be based on the content of a passage and not on previous knowledge or common sense. Wrong: Read the text and say if these sentences are TRUE or FALSE. a) The three basic principles to be environmentally friendly are reduce, reuse and recycle . ______   Better: Read the text and answer the following questions. a) What are the three basic principles to be environmentally friendly? _________________________________________
  17. 23. <ul><li>Once the item is designed, it is important to determine its appropriateness in terms of… </li></ul><ul><li>VALIDITY: Items test what we want to test. </li></ul><ul><li>RELIABILITY: Items’ results are consistent and replicable. </li></ul><ul><li>PRACTICALITY: Items are easy to design, complete and correct. </li></ul><ul><li>BACKWASH: Items are meaningful and motivating. </li></ul>
  18. 24. An item is VALID if it assesses authentic communication and interaction, and also reflects what students do in class. It has POSITIVE BACKWASH if students feel they have been tested fairly and become aware of the importance of language proficiency. Wrong:   Dictation . Listen to the instructions to start a weblog and write the information on the space provided. Then translate the information into your native language. Better: You want to create a weblog and a friend of yours is giving you the information to set up an account. Listen to the instructions and fill in the missing information to learn how to start your own blog.
  19. 25. An item is RELIABLE and if it measures students’ performance consistently. It is PRACTICAL if it takes little time to design, organize and mark, and involves resources that are easily available. Wrong: In your opinion, how can computers help us? _______________________________________________________________________________________________________________________________________ Better: Mention three ways in which computers can help firefighters in case of an emergency? 1)____________________________________________ 2)____________________________________________ 3)____________________________________________
  20. 26. Integrative and open-ended test formats are usually very VALID and also have a POSITIVE BACKWASH, since they involve communication and interaction in real-like situations. Example: While reading today’s newspaper, you find two interesting articles. Express your opinion about them. However, integrative and open-ended test formats are often impractical (difficult to interpret) and unreliable (open and subjective).
  21. 27. On the other hand, discrete item test formats are usually very RELIABLE and PRACTICAL, BECAUSE they provide enough information, correct answers are limited and they are quick and easy to mark. Example: It is said that pit-bulls are a breed/race of dog that is particularly violent. Nevertheless, discrete item formats frequently lack validity (meaningful communication) and positive backwash (luck is as important as hard work).