![Writing and Correcting Communicative Exams JoAnn Miller Editorial Macmillan [email_address]](https://image.slidesharecdn.com/examwriting-slideshare-090622204141-phpapp02/75/Exam-Writing-Slideshare-1-2048.jpg)
![Which version? Why? Match the letters (a to e) with the numbers (1 to 5). 1. Your mother’s husband is your___. 2. Your mother’s father is your___. 3. Your mother’s brother is your___. 4. Your uncle’s son is your___. 5. Your father’s sister is your ___. uncle cousin aunt father grandfather Underline the word in each pair that completes the conversation correctly. My (1)[ uncle / aunt ] likes (2) [ playing / going to ] movies. He is my father’s (3)[ sister / brother] . He’s (4)[ heavy / average ] and he has (5)[ blue / brown ] hair. His birthday is on October (6)[twelve / twelfth ] .](https://image.slidesharecdn.com/examwriting-slideshare-090622204141-phpapp02/75/Exam-Writing-Slideshare-18-2048.jpg)
Uploaded byJoAnn MIller
Exam Writing Slideshare
The document discusses various aspects of writing and correcting communicative language exams, including: 1. It provides an overview of topics like what skills to test, test formats, and correction techniques. 2. It discusses the benefits of using exam banks which can provide more standardization and criteria. 3. It offers guidance on determining exam sections, point values, and balancing accuracy and fluency, production and recognition, and objective and subjective questions.
More Related Content
Similar to Exam Writing Slideshare
Exam Writing Slideshare
- 1. Writing and CorrectingCommunicative Exams JoAnn Miller Editorial Macmillan [email_address]
- 2. Overview IntroductionWhat to test Types of tests Communicative testing Content Analysis Written tests Balance Formats Correction
- 3. Alternative Testing Notpaper and pencil Constant not punctual The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm
- 4. Portfolios a collectionof student production over time shows the stages in the learning process and the stages of the student’s growth. The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm
- 5. Portfolios? more subjectiveacceptance physical limits
- 6. Why written tests?Not the only way and maybe not the fairest. But easiest with large numbers of students More objective Accepted by institutions, parents, students
- 7. Exam Banks Acollection of exams for classroom use maintained by the institution itself. Written by the teachers themselves or a special committee Following institutional guidelines Could be various “cycles” covering the same material
- 8. The benefits ofan exam bank Less work for teachers More standardization in large one-campus schools and in multi-campus schools Criteria, instructions and grading Face validity Format Uniform length
- 9. What are yourexams going to be like? Many variables: Institution Students Teachers Text Time
- 10. Teachers Level ofEnglish Lack of mathematical skills Time factor Ease of grading Answer key
- 11. Students Younger studentsMore images Shorter exams Humor Older students Professionalism Humor
- 12. How many pointsfor each skill? If institution tells you, just follow through If not, base exam on the textbook (the common denominator) General text analysis How much time is spent on each skill Count exercises in a few units, determine percent Keep institutional goals in mind
- 13. Reasons for TestingPlacement tests student’s suitability to take a specific course based on specific textbook Proficiency tests check students’ progress in general TOEFL, First Certificate, etc. Achievement tests check how much a student has learned based on what a student has studied in a specific course
- 14. Communicative testing Weteach “communicatively” but we test “traditionally”. What IS communicative testing? Communicative testing means testing in context .
- 15. Grammar? What willyou test?
- 16. Which version? Why?Circle the correct answer 1. Do you like __________? swimming b. to swum c. swim 2. Where ________ live? does she b. she does c. she 3. I _________ speak French. no speak b. doesn’t c. don’t 4. What __________? a. does he do b. does he c. he does do Write the correct forms of the words in parentheses. Alice: Where (1)______ you _________ (live)? Bart: Acapulco. Alice: My brother (2)___________ (go) there every summer on vacation, but he (3)_________(not speak) Spanish. Bart: Acapulco (4)_________ (attract) tourists from all over the world. Many people there (5)___________(speak) English very well. What about you, (6)______ you ________(speak) Spanish? Alice: A little.
- 17. Vocabulary? What willyou test?
- 18. Which version? Why?Match the letters (a to e) with the numbers (1 to 5). 1. Your mother’s husband is your___. 2. Your mother’s father is your___. 3. Your mother’s brother is your___. 4. Your uncle’s son is your___. 5. Your father’s sister is your ___. uncle cousin aunt father grandfather Underline the word in each pair that completes the conversation correctly. My (1)[ uncle / aunt ] likes (2) [ playing / going to ] movies. He is my father’s (3)[ sister / brother] . He’s (4)[ heavy / average ] and he has (5)[ blue / brown ] hair. His birthday is on October (6)[twelve / twelfth ] .
- 19. Functions? What willyou test?
- 20. What is afunction? The communicative purpose of the users of the language. How language is used. Usually expressed as gerunds: introducing, apologizing, asking directions, requesting
- 21. Examples of aFunctional Cycle Function: Requesting (1) Open the window, please. (2) Would you open the window? (3) Could you please open the window? (4) Would you mind opening the window? (5) I was wondering if you would mind opening the window. (6) I’d be grateful if you opened the window. Each time the difference in register is emphasized.
- 22. How to test? Complete the conversation. Complete the conversation logically. Use the words in parentheses. Miriam: Tell me about your new apartment. Mary: (1)____________________(big / living room). Miriam: (2)___________________(how / bedrooms)? Mary: There are two, but (3)________(any furniture) in one of them. Or: Miriam: Tell me about your new apartment. Mary: (1)_______________________(living room). Or: Miriam: Tell me about your new apartment. Mary: (1)________________________________.
- 23. Content Validity Assessmentshould be based on a content-analysis of the text being used
- 24. Content Analysis Youmust test only material students have seen The only common denominator is the textbook Analysis of percent of time spent on each topic (grammar structure, vocabulary item, function, etc.)
- 25. Content Analysis: Informationfrom the contents Functions (10 points): Talking about imitation products Talking about food and food festivals Discussing the movie industry Making a business plan Grammar (5 points): Nouns in groups Indefinite Pronouns Vocabulary (10 points): Food Business language ///// /// ///// ///// ///// //// ///// /// ///// ///// /// 8 10 9 5 32 3 5 8 5 3 8 8 / 32 =__% 25% 31% 28% 16% 25% X 10 = ___pts 2.5 pts 3 pts 3 pts 1.5 pts 37% 2 pts 63% 3 pts 63% 6 pts 37% 4 pts
- 26.
- 27. Practice vs Testing In class practice During exam Goals Content Learner activity Teacher activity Class-room climate learning feedback on learning process oriented product oriented open ended close ended ss know material students might not know success-oriented success/failure oriented peer teaching no peer teaching helps performance gives tasks cooperative competitive relaxed tense intrinsic motivation extrinsic motivation
- 28. Balance Ideally anexam will balance: Accuracy and fluency Production and recognition Objective and subjective sections
- 29. Accuracy and FluencyFluency The ability to produce written and / or spoken language with ease Communicate ideas effectively The ability to use vocabulary chunks (phrases) to facilitate communication Accuracy Ability to produce grammatically correct sentences
- 30. Production and RecognitionProduction Student writes more than one word Can be creative / involves more “mental” work More than one answer may be possible Recognition Student recognizes correct answer Not creative Only one correct answer
- 31. Objective and SubjectiveSections Subjective There is more than one possible answer Corrector must be trained and experienced There can be surprises Students can protest grading Objective There is only one answer Anyone can correct the exam No surprises No argument from students
- 32. Grammar / Vocabulary/ Functions? Accuracy or fluency? Production or recognition? Subjective or objective?
- 33. Grammar / Vocabulary/ Functions? Accuracy or fluency? Production or recognition? Subjective or objective?
- 34. Grammar / Vocabulary/ Functions? Accuracy or fluency? Production or recognition? Subjective or objective?
- 35. Grammar / Vocabulary/ Functions? Accuracy or fluency? Production or recognition? Subjective or objective?
- 36. Grammar / Vocabulary/ Functions? Accuracy or fluency? Production or recognition? Subjective or objective?
- 37. Grammar / Vocabulary/ Functions? Accuracy or fluency? Production or recognition? Subjective or objective?
- 38. What is anexam section? A certain number of items testing the same skill / aspect To be communicative, they should be written as a conversation, note, letter, or some “real” type of discourse All items in a section should be worth the same number of points and test a similar skill (all grammar, all vocabulary, all functions, etc.)
- 39. Determining sections Youcan combine point values within the same aspect. Put two grammar structures in the same section You can divide point values between two sections Divide the points from one structure and put them in two sections
- 40.
- 41. Multiple Choice Partsof question Stem Options Distractors XXXXXXXXXXXXXXXXXX? YYYYYYYYY ZZZZZZZZ AAAAAAAA { { Correct answer
- 42. Stems 1. Before writing, identify the one point to be tested by that item. 2. The stem should either be an incomplete statement or a direct question 3. Don’t include words that do not contribute to the basis for choosing among the options. For example, The American flag has three colors. One of them is (1) red (2) green (3) black vs. One of the colors of the American flag is (1) red (2) green (3) black Kehoe, Jerard. Writing Multiple-Choice Test Items. ERIC/AE Digest Series EDO-TM-95-3, October 1995. http://www.ericdigests.org/1997-1/test.html
- 43. 4. Include asmuch information in the stem and as little in the options as possible. 5. Restrict the use of negatives in the stem. Negatives in the stem usually require that the answer be a false statement. 6. Avoid irrelevant clues to the correct option. Grammatical construction
- 44. Options (Kehoe)1. Use three or four options. 2. Construct distractors that are comparable in length complexity grammatical form 3. After the options are written, vary the location of the answer randomly.
- 45. Ordering Multiple ChoiceItems Numerical a. 1939 b. 1940 c. 1941 d. 1942 Burton, Steven J. Richard R. Sudweeks, Paul F. Merrill, Bud Wood. How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty , Brigham Young University Testing Services and The Department of Instructional Science. 1991. http://testing.byu.edu/info/handbooks/betteritems.pdf Sequential a. Heating ice from -100°C to 0°C. b. Melting ice at 0°C. c. Heating water from 0°C to 100°C. d. Evaporating water at 100°C. e. Heating steam from 100°C to 200°C. Sequential a. Heating ice from -100°C to 0°C . b. Melting ice at 0°C . c. Heating water from 0°C to 100°C. d. Evaporating water at 100°C . e. Heating steam from 100°C to 200°C. Alphabetical a. C hanging a from .01 to .05. b. D ecreasing the degrees of freedom. c. I ncreasing the spread of the exam scores. d. R educing the size of the treatment effect.
- 46. True / FalseAdvantages : Can test large amounts of content Students can answer 3-4 questions per minute Disadvantages: They are easy Students have a 50-50 chance of getting the right answer by guessing It is difficult to discriminate between students that know the material and students who don't Need a large number of items for high reliability Designing Test Questions, Grayson H. Walker Teaching Resource Center, The University of Tennessee at Chattanooga, h ttp://www.utc.edu/Administration/WalkerTeachingResourceCenter/FacultyDevelopment/Assessment/test-questions.html
- 47. How to “save”a T/F section… Add a third option “ Not mentioned” OR Have student correct F answers But only if students have practiced this version in the textbook
- 48. Cloze-type sections Writewords or phrases from a box Write the correct forms of verbs, comparatives, etc. Even Complete the Conversation
- 49. True Cloze A cloze test is a special type of fill-in exercise where, for example, every 5th word in a paragraph of about 150 words is deleted. (It could be every 6th word, or every 7th word, and so on.)
- 50.
- 51. Boxes In thesesections, students are given a text with certain words omitted The omitted words / phrases (and perhaps some distractors) are put in a box at the top or side of the text More difficult if extra options are supplied The student completes the “Cloze” exercise with the words presented Can be used with grammar, vocabulary or functions (chunks)
- 52.
- 53. Fill in theblank A set of sentences or a text which has blanks in it for the students to complete with the correct or appropriate word. Example: He walked _____ school. He ______ (walk) to school. Fill-in-the blank exercises are a good way of reinforcing new grammar and vocabulary. Also called: fill-in the gap, fill-in
- 54.
- 55. Ordering: text andsentence Writing a text in order (paragraph, story, conversation) evaluates the student’s ability to recognize discourse cues (pronouns, connectors, chunks, etc.) Writing a sentence in the correct order evaluates a students knowledge of syntax, which is considered part of grammar. cream / I / ice / like
- 56. Complete the conversationThese sections evaluate a student’s ability to communicate ideas if they are corrected for communication and not for accuracy. They can be written with different degrees of cueing.
- 57. Complete the conversation.Complete the conversation logically. Use the words in parentheses. Miriam: Tell me about your new apartment. Mary: (1)____________________(big / living room). Miriam: (2)___________________(how / bedrooms)? Mary: There are two, but (3)________(any furniture) in one of them. Or: Miriam: Tell me about your new apartment. Mary: (1)_______________________(living room). Or: Miriam: Tell me about your new apartment. Mary: (1)________________________________.
- 58. Point Values Givemore points to… Production items Give fewer points to… Recognition items Give partial credit in Fluency / Production sections Use fractions only if your teachers are mathematical
- 59. Instructions Keep themsimple You can use Spanish in lower levels You can translate them into Spanish whenever necessary. Use the same wording in all your exams Use examples whenever necessary But be careful they don’t give away the pattern Remember: The instructions are NOT part of the exam…
- 60.
- 61. Correcting Grammar, Reading,Vocabulary, Listening In general these sections are all right or all wrong . We rarely give partial credit. These sections test accuracy . Communicative sections You can give partial credit These sections test fluency. Ask yourself if the S’s answer communicates what the S wants to say .
- 62. Examples of partialcredit Correct answer: What’s your name? Student writes: What you name? Correct answer: If you invited me, I’d go. Student writes: If you invite me, I go. Correct answer: I went to the movies yesterday. Student writes: I go to the movies yesterday. I go to the movies.
Editor's Notes
- #3 Go over the Overview and have students predict what each theme refers to…
- #4 There are testing alternatives [click] that aren’t the traditional “paper and pencil” exams. [click] You can evaluate students’ progress constantly, not punctually (at specific times) [click] Definitions from REF 1: The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm
- #5 Reference 1 (REF1) From The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm
- #6 Portfolio assessment is very popular in places like the US where classes are small and teachers have office hours and preparation time [click] But they are more subjective, you can’t guarantee that two teachers will be grading the same way and [click] Teachers in Mexico have physical limitations: lots of students, lots of groups, little free time to correct [click] Would they be accepted by the SEP, schools, parents and students? Have participants discuss in pairs and then compare responses.
- #7 Why do we rely on written tests here in Mexico? [click] It isn’t the only way to test and it probably isn’t the fairest…some students don’t do well on written tests [click] But they are the easiest way to test with the large numbers of students we have here [click] It is more objective and can be designed so that results can be fairly consistent among teachers at the same institution [click] And they are accepted by institutions (official study plans, schools) parents and the students themselves
- #8 Only show the title of the slide. Ask teachers how many have exam banks at their schools (If some participants don’t know what exam banks are, ask for a definition). Put students into groups, try to get at least one teacher who has worked with an exam bank in each group. Have them discuss the advantages and disadvantages of the exam banks. When they finish, listen to some of their conclusions. Then continue on with the slide. Exam banks have many advantages for large schools. They are a collection of exams [click] that could be written by the teachers or a special committee [click]. They must be based on institutional guidelines. [click] It is possible to have various cycles of the exams.
- #9 Benefits: Teachers don’t have to write all their own exams. Instead of writing 5, 10 or more exams a semester, they write one or two. It save time, energy and quality. [click] They are useful in larger schools because they help standardize performance assuring all the students who pass from one level to another are ready. The criteria, instructions and grading are uniform. [click] Face validity (an exam looks like the teachers and students expect it to). The format is similar. All exams in the exam bank follow the same rules so there are no surprises. They even look similar. [click] Exams are also about the same length. In reality exams banks are the most time-efficient, reliable ways to test. Each exam can be written with the same criteria and the results between groups is much more reliable. Various cycles can be created so that cheating is minimized and each individual teacher works less since instead of writing an exam for every class they are teaching, they might just write a couple of exams for the exam bank. In pairs: Do they have exam banks? How do they work? If not, would they work?
- #10 So, if you are going to give a written test, you need to first decide what you are going to test [click] And that depends on many variables [click] The institution: often your school decides what you will test or gives you the exams you will use [click] Your students: Exams should reflect the age, interests, goals, and abilities of your students [click] Other teachers: If you are writing for an institutional exam bank, you have to write exams other teachers can use. They can’t just reflect what you do in class. [click] The text: Exams vary depending on the textbook you use. If you change texts, you can’t continue using the same old exams since the material covered is probably quite different [click] Time: It makes a big difference if your students have 30 or 50 minutes or an hour and a half to take an exam.
- #11 Writing for an exam bank requires the writer to think about the other teachers who will be using the exam. There are many criteria that need to be considered: [click] Level of English: Not all your colleagues speak English as well as you do. [click] Lack of mathematical skills: We all became teachers because we couldn’t do math. Therefore, don’t make exams practice in adding fractions. Don’t assign point values of 5/6, ¾, 8/9, etc. Teachers will not appreciate it. [click] Time factor: Pay attention to how much time is available to take the test. Remember, 50 minutes for an exam also includes a lot of organizational activity: seating arrangements, handing out the exam, going over the instructions, etc. [click] Ease of grading: Teachers don’t have that much free time to correct exams. Make them as easy to correct as possible without relying entirely on multiple choice and true/false. [click] Answer key: Provide the answers. Not all teachers will be able to read your mind. Understand why: Be sure other teachers understand why you are testing in a specific way BEFORE they give the test. It will save time and misunderstandings later.
- #12 You always have to adapt your test-writing to your students [click] If they are younger, use more images and make the exams shorter (students will work slower and have shorter attention spans), and always include humor. It relaxes them and makes an unenjoyable experience a bit more tolerable. [click] If they are older, consider making the exams more professional, reflecting what the students are studying or doing in their everyday lives and again, include humor. Even older students like to laugh occasionally.
- #13 You also have to decide how to divide up the points you have to cover the skills your students have practiced and emphasis they have given to grammar and vocabulary training. For example, maybe you are using a text in reading comprehension that includes some grammar and vocabulary work. If that were true, you’d want to devote more points to testing reading than to grammar and vocabulary. But if you were using a “four-skills” text that concentrates on vocabulary and grammar development and just has one reading practice per unit, you would devote many fewer points to reading. [click] If your institution tells you how many points to devote to each skill, you just follow through. [click] However, it your have to determine how many points to assign yourself, use your textbook as a guide. If you are writing exams other teachers will be using, it is only fair since it is the only common denominator. If you love teaching vocabulary and always give students extra vocabulary lists and a colleague prefers to teach grammar and doesn’t have access to your creative lists, imagine using his exam that ignores your beloved vocabulary items that you had the students review every night. Even if you write exams just for your students, it is only fair to base them on the text. Imagine the poor student who gets chicken pox and has to miss two weeks of class. She might not get your beautifully designed lists. All she can study from is the text and when she gets back to class and you give her your exam with 50% of the points dedicated to vocabulary you only presented in class, she’ll be very shocked to realize her hours of self-study were useless and she can’t understand what you are testing. [click] So, base your analysis on a general text analysis [click] Look at about how much time is spent on each skill [click] Count how many grammar, vocabulary, listening, reading, etc. exercises there are in each unit and calculate what percent of the time spent in the book is dedicated to each skill. [click] However, always keep the institutional goals in mind. Often the text chosen by the administration doesn’t really reflect the goals they have in mind for the students.
- #14 Why should we test? [click] Exams can indicated if a student is able to take a specific course, as in a placement exam. [click] Exams can check a students’ general progress, as in proficiency exams or diagnostics [click] These are exams like the TOEFL…they can exempt students from studying, but they can’t place them in a spcific course. [click] Exams can check how much a student has learned in a specific course. These are the exams we give every unit, every three units or every semester. These exams are called achievement exams. Each type of exam has a different purpose and they cannot be interchanged. For example, the TOEFL is a proficiency exam written to determine if the student’s level of English is high enough to undertake a course of study in the United States. It isn’t a placement exam for a specific course. A placement exam should be developed in direct relationship with the text used so that the student can be placed in the appropriate level. When you change texts, you change placement exams. Achievement exams are written to cover a limited amount of material to determine if the student has mastered it satisfactorily.
- #15 Ask participants what they think “communicative testing” refers to. After hearing some possibilities, tell them: The “Communicative Approach” has been in existence since the late 80s. We now accept the ideas of teaching communicatively: group work, teaching language in context, etc, but we continue testing traditionally [click] Ask participants what “traditional testing” is (isolated sentences, transformation, grammar-based) [click] Communicative testing means testing in context…not isolated sentences
- #16 HO2-- Let’s look at some examples. What will you test? [click] Grammar? Refer participants to HO2. Have them look at the grammar examples and compare them in pairs. What are the differences ? When they finish giving you some differences go to the next slide.
- #17 [click] The example on the left is traditional. There is no context. It is OK for simple structures like this one, but what about more complex structures such as If clauses or present perfect/past? Here is an example you can give them: If I __________ (be invited) to your party, I __________ (go). What is the answer? If I am invited to her party, I’ll go (If 1) If I were invited to her party, I’d go (If 2) If I had been invited to her party, I would have gone (If 3) Since there is no context, any of them would be correct. [click] In the other example the grammar is presented in context, a conversation. If this had been about If clauses, it might have been: “I’m sorry. I didn’t know she was going to have a party. If I ______________ (be invited), I ___________ (go).” Obviously If 3.
- #18 What will you test? [click] Vocabulary? Refer participants to the HO2. Have them look at the vocabulary examples and compare them in pairs. What are the differences? When they finish giving you some differences go to the next slide.
- #19 [click] The example on the left is a traditional vocabulary section. It tests if they learned a vocabulary list, but it doesn’t test if they really know how to use the vocabulary. [click] The example on the right tests many different problems students can have when they work with vocabulary: Difficult pairs: (1) (3) (6) Collocation: (2) (4) (5) The context lets you test more.
- #20 Ask participants how they could test functions…collect some ideas.
- #21 Go over the definition of functions just in case someone doesn’t know what they are.
- #22 Go through the examples. Emphasize how the following aspects become more complex as we go down the list: Grammar structures (easy to more difficult) Length of sentences (short to long) Register (from informal to formal)
- #23 This is the best type of section to test students’ knowledge of functions. (In HO2) [click] Students complete a conversation (or paragraph) with sentences (or even phrases) that communicate the correct function logically. Go over the first example. Show that there isn’t one correct answer. The first one could be: It has / I have / There is a big living room. Elicit possible answers for # 2 and 3. The section can be even more open as the other two option show. Elicit possible answers. Have participants compare this with the grammar and vocabulary sections they have seen. What are the differences? Emphasize that here the purpose is communication and that communication can occur even if the students don’t use complete sentences or make some grammar errors. This will be seen later in the workshop when they learn to correct these sections.
- #24 Base the content analysis on the textbook for the reasons we mentioned previously…Slide 13
- #25 You can’t test students on something they haven’t seen. [click] If you are writing for other teachers, the only thing you have in common is the textbook. If you are only writing for your students and they are absent, the only resource they have is their textbooks[click] You need to make an analysis of how much time is spent on each aspect you want to test.
- #27 Here is an example of a final content analysis. If you are using an exam bank, all the teachers should have it. The students should also have it. It can help them study. Look at the Grammar section. Notice that one point of the 15 points on that section of the exam is dedicated to the comparative. If the student didn’t know only one point was dedicated to that structure, he would know that it was more important to study BE and the simple present (7 points). This can also help the teachers since no one would spend hours teaching all aspects of the comparative if it is not represented with more than one point on the exam. This would be a positive washback effect.
- #28 It is important that participants realize they can’t just take exercise sections from textbooks and use them to test. The purposes behind textbook exercises and examinations are the same as those between “in class practice” and “during exam” activities. The exam sections have different purposes and, therefore, different structures. Go over this chart carefully… Aims: In class the purpose is to learn, in exams it is to get feedback on the learning Content: activities in class are process oriented (the “doing” is usually more important than the result), exams are product oriented. In language exams it doesn’t matter how you get to a result....it’s the result that is graded. activities in class are also open-ended, there isn’t always a result, they can continue for days; on exams they have to close. Learner activities: in class students basically know the material or can use books and dictionaries to find out what they don’t know, on exams they often don’t know the material classroom activities are success-oriented, students are helped to succeed, exams are often success oriented, but not for all students you can have peer-teaching in classroom activities, but working in groups is ususally frowned on during exams. Teacher activity: In class, the teacher helps students improve their performance, on exams the teachers might help the student understand what he is supposed to do, but the teacher doesn’t help the student directly with the answers. Classroom climate: The climate in class is cooperative, students help each other, but on exams it could be competitive Classroom activities are relaxed, exams are tense There is intrinsic motivation in class (students often are motivated to do well by their own internal desires), but it is extrinsic during exams (parents, schools, grade pressure, etc usually motivate students to do well).
- #29 Balance means it adds up to 100% For example, you don’t need to have a 50-50 balance. 80-20, 70-30, 40-60 are also balances. The balance depends on the school situation…. BUT all three should be present in an exam.
- #30 Go over the definitions…be sure participants understand them… Accuracy and Fluency balance depends on what the students will be doing with English. Ask the participants: What accuracy/fluency balance would you recommend for tourism students? What is more important for them? (Fluency-maybe 70-30 over accuracy). What balance for students who will be translators? (Accuracy—maybe 80-20 over fluency).
- #31 In production, the student writes more than one word (remind them of Complete the Conversation for testing functions) The student can be more creative and the teacher has to be more alert because more than one answer might be correct. In recognition (for example, multiple choice or true/false), there is only one correct answer and it isn’t creative. Production means more work. Teachers with large number of students can’t handle a 50-50 balance. A good exam could be 20-80 or 30-70 (production/recognition) and still be fair. TOEFL and other similar exams are all recognition…they never test whether the student can produce language. Years ago this led to a big influx of Asian students into US universities. They had studied grammar and reading, but no speaking or writing. They did great recognizing correct answers on the TOEFL and got very high scores, but when they arrived in the US, authorities realized they couldn’t say two words in English….The TOEFL exam was revised and now includes writing sections and often oral interviews are required to study in the US.
- #32 Sections can also be objective or subjective. It is good to have some subjective sections, but an exam with sections that are all graded subjectively make it difficult to judge student ability between different teachers…. However, even though they are more difficult to grade, they do give more information about students’ ability. The limitations can be overcome if the graders are trained…. This is a very common problem with oral grading.
- #33 HO 4 ( Put participants into pairs and have them look at each exam section. For each one, they identify it as Accuracy/fluency, production/recognition and subjective/objective. About 15 mintues. Then go over all of them using the slides.) We are going to look at some exam sections and identify if they test accuracy/fluency, if they are designed to be production or recognition sections and if they are to be graded subjectively or objectively. Have participants go over each section in pairs using the handout. Then use this slide and the following ones to go over them. Answers: Fluency: Have participants show you there are more than one possible answer for most of the items. (for example, (1) What’s your name? / Your name, please? / Name, please / etc.) Production: Students write complete sentences or phrases. Subjective: The grader has to understand possible answers.
- #34 Answers: Fluency: it practices functions through vocabulary chunks which are essential for fluent conversations. Recognition: There is only one right answer. Objective: Anyone could grade it given an answer key
- #35 Answers: Accuracy: This is testing correct use of the present tense. Recognition: In reality all the student needs to do is find the subject and put the verb in the correct form Objective: There is only one correct answer.
- #36 Answers: Fluency: This is similar to the first example, but it is limited by cues. However, the students still have some freedom to be creative: (1) Look at those earrings / Look at these earrings / Look earrings ) Production: They are writing phrases Subjective: The grader must understand English to tell if a student’s answer communicates clearly.
- #37 Answers: Accuracy: tests grammar structures taught in the text Recognition: Just find the answer in the box and write it. Objective: One correct answer
- #38 Answers: Fluency: This requires the students to use discourse cues to order the conversation. It goes beyond simple accurate sentences. Discourse cues are essential for effective communication. Recognition: They just copy the sentences Objective: There is one correct order.
- #39 Go over slide with participants…. If necessary, go back and look at the examples on the previous slides and on HO 4.
- #40 All of these comments refer to results from a content analysis…. The next slide gives an example….
- #41 Now, we’ll look into some more difficult formats in more detail
- #42 Get examples so you can be sure they understand what the stem (first part) and options are (A.b.c)
- #43 REF 3: From Kehoe, Jerard. Writing Multiple-Choice Test Items. ERIC/AE Digest Series EDO-TM-95-3, October 1995. http://www.ericdigests.org/1997-1/test.html Go through the points one-by-one…checking comprehension as you go
- #44 Continued
- #45 You might want to discuss how many items they feel they should use. Kehoe recommends 3-4, but some really formal exams use 5 options.
- #46 These are the different orders possible…. Talk about which seem best and why? Do multiple Choice handout (HO5) in pairs (30 minutes). These are the poor examples from the Burton article. Have participants think about what they think is wrong with them and how they could be improved. Then go over them. Use REF 4 to correct… REF 4: Burton, Steven J. Richard R. Sudweeks, Paul F. Merrill, Bud Wood. How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty, Brigham Young University Testing Services and The Department of Instructional Science. 1991. http://testing.byu.edu/info/handbooks/betteritems.pdf When they finish HO5, have them write a multiple choice section using their content analyses and the section divisions they recently did. When they finish, have them compare and criticize their work. Be careful they are using multiple choice for reasonable sections…(20 minutes)
- #47 Go over with participants. Ask for their opinions. From (REF5) Designing Test Questions, Grayson H. Walker Teaching Resource Center, The University of Tennessee at Chattanooga, http://www.utc.edu/Administration/WalkerTeachingResourceCenter/FacultyDevelopment/Assessment/test-questions.html
- #49 Get examples from them…
- #50 Macmillan English Dictionary…on line Go over the definition and be sure everyone understands.
- #51 Answers: Accuracy: tests grammar structures taught in the text Recognition: Just find the answer in the box and write it. Objective: One correct answer
- #52 Look for examples
- #53 Answers: Accuracy: This is testing correct use of the present tense. Recognition: In reality all the student needs to do is find the subject and put the verb in the correct form Objective: There is only one correct answer.
- #54 Go over Macmillan English Dictionary…on line
- #55 Answers: Fluency: This requires the students to use discourse cues to order the conversation. It goes beyond simple accurate sentences. Discourse cues are essential for effective communication. Recognition: They just copy the sentences Objective: There is one correct order.
- #56 Find examples When you finish the discussion , have them write two exam sections using their content analyses and the section divisions they recently did. They decide on which types of sections they want to write. When they finish, have them compare and criticize their work. Be careful they are using formats that are reasonable for what they are testing…(20 minutes)
- #57 Go over this with the participants. It will be the hardest section to write since they have very little experience with communicative exams.
- #58 This is the best type of section to test students’ knowledge of functions. (In HO2) [click] Students complete a conversation (or paragraph) with sentences (or even phrases) that communicate the correct function logically. Go over the first example. Show that there isn’t one correct answer. The first one could be: It has / I have / There is a big living room. Elicit possible answers for # 2 and 3. The section can be even more open as the other two option show. Elicit possible answers. Have participants compare this with the grammar and vocabulary sections they have seen. What are the differences? Emphasize that here the purpose is communication and that communication can occur even if the students don’t use complete sentences or make some grammar errors. This will be seen later in the workshop when they learn to correct these sections.
- #59 This is all logical, but we often forget when we are writing exams [click] Production items should have more points because the student is required to do more [click] Recognition items are easier (it’s easier to recognize a correct answer than to think of it) so they should be worth less [click] Production sections also require more points because you should give partial credit [click] Keep away from complex fractions Have them assign point values to the sections they have written.
- #60 Go over these points one-by-one and let participants comment. Some are a bit controversial. (The instructions are not the exam. This is the basis of this slide…) Instructions should be as simple as possible. They aren’t testing the student’s knowledge of English, the items are. [click] Instructions can be very intimidating for lower level students. Write them on the exam in Spanish. They are not testing English, the items are. [click] In higher levels, if the instructions are in English and a student just can’t understand what to do, translate them. The instructions are NOT the test. [click] Always give the same instructions using the same wording. Students will learn them and save time. [click] Be careful when using examples. Some structures are based on patterns and examples can give away the pattern. For example, want someone to do something, if clauses, question formation, etc. [click] And finally….the instructions are NOT the exam Have them write instructions for each of their sections. Compare and discuss. (10 minutes)
- #61 It’s easy to correct an “A, b, c” section, but how do you correct a “complete the conversation” section so that it really tests communicative ability and competence?
- #62 Go over the summary
- #63 These are examples of partial credit. Go over then one-by-one. Ask participants if they would give partial credit or not if they were grading communication… First example: Does it communicate? Would you be able to answer the student’s questions? Probably. If it were a C1 student on the first exam, I’d give credit, but write in the corrections. On later exams, I’d probably not be a generous. Second example: The meaning is different. In the correct answer, you didn’t invite me, in the student’s answer you might do so in the future. Although the grammar is correct, it doesn’t communicate and I wouldn’t give any credit. Third example: Click slowly, discussing as you go. The first one communicates and if it were the first time students had worked with the past I’d accept it and just write in the correction. They use “yesterday” to indicate the tense. [click] This could also be accepted if it were in answer to the question. “what did you do yesterday?”. Correcting these sections, you have to ask “does the answer communicate the idea required by the conversation.” If so, accept it or give partial credit depending on the level of the students.