Item analysis, design and test formatsPresentation Transcript
Item: a question used to test a piece ofknowledge. Instruction(s): An explanation of how tocomplete a task. It contains a meaningful situationand the actual task to perform. Stem: The linguistic environment that works asthe context to elicit the right answer in multiplechoice (MC), gap-filling, and other objective formats.In MC items, it leads into the cue (key + distracters).
Cue: The MC item’s response options, usually 4. Key: The correct choice in MC. Distracters (or distractors): The responseoptions in MC that are possible but incorrect .What is the main idea of the passage “Frogs and Toads”? [Stem]A Frogs & toads share differences & similarities. [Key]B Frogs live closer to water than many toads. [Distractor]C Frogs and toads behave just like cousins. [Distractor]D Frogs may live different kinds of lives than toads. [Distractor] Adapted from: http://tiny.cc/797ndw
Poor distracterA response option selected by less than 10 percent of therespondents to the items. Under-selected distracters indicateeither that the distracters were ineffective or that the contentareas behind the distracters were well understood by therespondents so that these distracters were not “distracting” atall.Standard MC item the format describing an item that includes one simplepositively stated stem that directly leads to the responseoptions.
Before you start designing your items, answer thefollowing 2 questions: What am I going to assess? (aspects of language) How am I going to assess it? (test formats)Write your items considering the following 5 aspects: Task (behavior that exhibits the knowledge to beassessed. Make it realistic -similar to real life tasks) Context (meaningful) Instructions (clear and concise. Examples if necessary) Stem (unambiguous elicitation of the correct answer) Options (cues and key well taken care of).
Depending on the nature of the content beingassessed, choose the most appropriate test formatREADING AND LISTENINGGap-filling (open/with options)Multiple choiceMatchingTrue/false (don’t know/not mentioned)Open-ended questionsClosed questionsTable completion (information transfer)Linking (for reference words)DiscrepanciesListing/Sequencing
LANGUAGE COMPONENTSCloze test -Ctest (blank every nth word)Gap-fillingSentence arrangementSentence writingTable completionMatching sentence halvesMatching words/definitionsMultiple choiceOpen-ended questionsClosed questionsDialogue completion
Problem: For every item, all cues must be grammaticallyand/or semantically consistent and homogeneous.Wrong: John _________ flowers to the party last night.a) Carriesb) Carriedc) Liftsd) LiftedBetter: John _________ flowers to the party last night.a) Carriesb) Carriedc) Has carriedd) Is carrying
Problem: The stem should not introduce the correctoption.Wrong: He wrote an ________ in his yahoo account.a) Letterb) Post cardc) E-maild) MemoBetter: He wrote a brief ________ in his yahoo account.a) Letterb) Post cardc) E-maild) Memo
Problem: In communicative tests, instructions should notbe based on grammar, but on meaningful contexts.Wrong: Write sentences using make, let, and beallowed.Better: You got admitted in a university away fromhometown so you have to move to a student residence.Write sentences expressing the things that are prohibitedor permitted in the residence (you may have to usemake, let, and be allowed).
Problem: The cues in a multiple-choice format should beall grammatically possible.Wrong: Katherine is a very _________ student. She getsnothing but straight A’s.a) Smartlyb) Intelligentc) Preoccupiedd) ConsiderateBetter: Katherine is a very _________ student. She getsnothing but straight A’s.a) Thoughtfulb) Intelligentc) Preoccupiedd) Considerate
Problem: It is never suggested to use made-up words orgrammatical structures as distracters.Wrong:People say that my children are malraised/spoiled because I buy them whatever they ask for.Better:People say that my children are consented/spoiledbecause I buy them whatever they ask for.
Problem: Language tests assess overall languageproficiency, not cognitive skills or creativity.Wrong: Unscramble the jumbled letters below to form Englishwords.a) MEAXb) GMAEUZc) TACd) ALAVGIBetter: Unscramble the jumbled words below to formsentences expressing frequency.a) gym often go I the to _____________________________b) she ever television hardly watches___________________c) my mom home never at is _________________________d) cooks Franklin dinner always_______________________
Problem: The difficulty in an item should lie in the taskitself, not in the instructions or the stem.Wrong: When a police officer arrests someone, theofficer must inform the person of a certainrights, including the right to remain silent and the right toan attorney. Why, according to the text, is this required?Better: According to the text, why must police officersinform suspects of their rights during an arrest?
Problem: Review options require considerable concentration (isthis what you are assessing?) and focus on the small informationpresented in the previous opnes. Avoid such options (d, e, f, g)Wrong: The person in the listening said he had left his land becausehe…a) Wanted to seek his fortuneb) Wished to avoid his creditorsc) Preferred the new landd) All of the abovee) None of the abovef) A and B, but not Cg) B and C, but not ABetter: The person in the listening said he had left his land becausehe…a) Wanted to seek his fortuneb) Wished to avoid his creditorsc) Preferred the new landd) Had his family abroad already
Problem: All cues should be about the same length. Anotheroption is having two short cues and two long ones.Wrong: Based on the recording, what is the best way toreduce inflation?a) Cutting on income taxes.b) Increasing government expenses.c) Reducing the number of bank operating all over America.d) Raising local interest rates.Better: Based on the recording, what is the best way toreduce inflation?a) Cutting on income taxes.b) Increasing government expenses.c) Reducing the federal deficit.d) Raising local interest rates.
Problem: The correct answer should be based on the contentof a passage and not on previous knowledge or commonsense.Wrong: Read the text and say if these sentences areTRUE or FALSE.a) The three basic principles to be environmentally friendly arereduce, reuse and recycle. ______Better: Read the text and answer the following questions.a) What are the three basic principles to be environmentallyfriendly? _________________________________________
Once the item is designed, it is important todetermine its appropriateness in terms of…VALIDITY: Items test what we want to test.Test-takers are familiar with the format.RELIABILITY: Items’ results are consistentand replicable.PRACTICALITY: Items are easy to completeand correct.BACKWASH: Items are meaningful andmotivating.
An item is VALID if it assesses authenticcommunication and interaction, and also reflectswhat students do in class. It has POSITIVEBACKWASH if it adds value to the educationalprocess due to its desirable impact on learning.Wrong: Dictation. Listen to the instructions to start aweblog and write the information on the space provided.Then translate the information into your native language.Better: You want to create a weblog and a friend is givingyou the information to set up an account as the 1st step.Listen to the instructions and fill in the missinginformation to learn how to start your own blog.
An item is RELIABLE if it measures students’performance consistently. It is PRACTICAL if it takeslittle time to design, organize and mark, and involvesresources that are easily available.Wrong: In your opinion, how can computers help us?_______________________________________________________________________________________________________________________________________Better: Mention three ways in which computers canhelp firefighters in case of an emergency.1)____________________________________________2)____________________________________________3)____________________________________________
Integrative and open-ended test formats areusually very VALID and also have a POSITIVEBACKWASH, since they involve communicationand interaction in real-life situations.Example: While reading today’s newspaper, youfind two interesting articles. Express youropinion about them.However, integrative and open-ended testformats are often impractical (difficult tointerpret) and unreliable (open and subjective).
On the other hand, discrete-point test formatsare usually very RELIABLE and PRACTICALbecause correct answers are limited and they arequick and easy to mark.Example: It is said that pit-bulls are abreed/race of dog that is particularly violent.Nevertheless, discrete item formats frequentlylack validity (meaningful communication) andpositive backwash (luck is as important as hardwork).