Communicative testing written and oral-cccn--wb


Published on

Talk given at the CCCN, Costa Rica, June 24, 2010

Published in: Education, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Why should we test? [click] Exams can indicated if a student is able to take a specific course, as in a placement exam. [click] Exams can check a students’ general progress, as in proficiency exams or diagnostics [click] These are exams like the TOEFL…they can exempt students from studying, but they can’t place them in a spcific course. [click] Exams can check how much a student has learned in a specific course. These are the exams we give every unit, every three units or every semester. These exams are called achievement exams. Each type of exam has a different purpose and they cannot be interchanged. For example, the TOEFL is a proficiency exam written to determine if the student’s level of English is high enough to undertake a course of study in the United States. It isn’t a placement exam for a specific course. A placement exam should be developed in direct relationship with the text used so that the student can be placed in the appropriate level. When you change texts, you change placement exams. Achievement exams are written to cover a limited amount of material to determine if the student has mastered it satisfactorily.
  • So, if you are going to give a written test, you need to first decide what you are going to test [click] And that depends on many variables [click] The institution: often your school decides what you will test or gives you the exams you will use [click] Your students: Exams should reflect the age, interests, goals, and abilities of your students [click] Other teachers: If you are writing for an institutional exam bank, you have to write exams other teachers can use. They can’t just reflect what you do in class. [click] The text: Exams vary depending on the textbook you use. If you change texts, you can’t continue using the same old exams since the material covered is probably quite different [click] Time: It makes a big difference if your students have 30 or 50 minutes or an hour and a half to take an exam.
  • Writing for an exam bank requires the writer to think about the other teachers who will be using the exam. There are many criteria that need to be considered: [click] Level of English: Not all your colleagues speak English as well as you do. [click] Lack of mathematical skills: We all became teachers because we couldn’t do math. Therefore, don’t make exams practice in adding fractions. Don’t assign point values of 5/6, ¾, 8/9, etc. Teachers will not appreciate it. [click] Time factor: Pay attention to how much time is available to take the test. Remember, 50 minutes for an exam also includes a lot of organizational activity: seating arrangements, handing out the exam, going over the instructions, etc. [click] Ease of grading: Teachers don’t have that much free time to correct exams. Make them as easy to correct as possible without relying entirely on multiple choice and true/false. [click] Answer key: Provide the answers. Not all teachers will be able to read your mind. Understand why: Be sure other teachers understand why you are testing in a specific way BEFORE they give the test. It will save time and misunderstandings later.
  • You always have to adapt your test-writing to your students [click] If they are younger, use more images and make the exams shorter (students will work slower and have shorter attention spans), and always include humor. It relaxes them and makes an unenjoyable experience a bit more tolerable. [click] If they are older, consider making the exams more professional, reflecting what the students are studying or doing in their everyday lives and again, include humor. Even older students like to laugh occasionally.
  • You also have to decide how to divide up the points you have to cover the skills your students have practiced and emphasis they have given to grammar and vocabulary training. For example, maybe you are using a text in reading comprehension that includes some grammar and vocabulary work. If that were true, you’d want to devote more points to testing reading than to grammar and vocabulary. But if you were using a “four-skills” text that concentrates on vocabulary and grammar development and just has one reading practice per unit, you would devote many fewer points to reading. [click] If your institution tells you how many points to devote to each skill, you just follow through. [click] However, it your have to determine how many points to assign yourself, use your textbook as a guide. If you are writing exams other teachers will be using, it is only fair since it is the only common denominator. If you love teaching vocabulary and always give students extra vocabulary lists and a colleague prefers to teach grammar and doesn’t have access to your creative lists, imagine using his exam that ignores your beloved vocabulary items that you had the students review every night. Even if you write exams just for your students, it is only fair to base them on the text. Imagine the poor student who gets chicken pox and has to miss two weeks of class. She might not get your beautifully designed lists. All she can study from is the text and when she gets back to class and you give her your exam with 50% of the points dedicated to vocabulary you only presented in class, she’ll be very shocked to realize her hours of self-study were useless and she can’t understand what you are testing. [click] So, base your analysis on a general text analysis [click] Look at about how much time is spent on each skill [click] Count how many grammar, vocabulary, listening, reading, etc. exercises there are in each unit and calculate what percent of the time spent in the book is dedicated to each skill. [click] However, always keep the institutional goals in mind. Often the text chosen by the administration doesn’t really reflect the goals they have in mind for the students.
  • Base the content analysis on the textbook for the reasons we mentioned previously…Slide 13
  • You can’t test students on something they haven’t seen. [click] If you are writing for other teachers, the only thing you have in common is the textbook. If you are only writing for your students and they are absent, the only resource they have is their textbooks[click] You need to make an analysis of how much time is spent on each aspect you want to test.
  • Here is an example of a final content analysis. If you are using an exam bank, all the teachers should have it. The students should also have it. It can help them study. Look at the Grammar section. Notice that one point of the 15 points on that section of the exam is dedicated to the comparative. If the student didn’t know only one point was dedicated to that structure, he would know that it was more important to study BE and the simple present (7 points). This can also help the teachers since no one would spend hours teaching all aspects of the comparative if it is not represented with more than one point on the exam. This would be a positive washback effect.
  • Ask participants what they think “communicative testing” refers to. After hearing some possibilities, tell them: The “Communicative Approach” has been in existence since the late 80s. We now accept the ideas of teaching communicatively: group work, teaching language in context, etc, but we continue testing traditionally [click] Ask participants what “traditional testing” is (isolated sentences, transformation, grammar-based) [click] Communicative testing means testing in context…not isolated sentences
  • HO2-- Let’s look at some examples. What will you test? [click] Grammar? Refer participants to HO2. Have them look at the grammar examples and compare them in pairs. What are the differences ? When they finish giving you some differences go to the next slide.
  • [click] The example on the left is traditional. There is no context. It is OK for simple structures like this one, but what about more complex structures such as If clauses or present perfect/past? Here is an example you can give them: If I __________ (be invited) to your party, I __________ (go). What is the answer? If I am invited to her party, I’ll go (If 1) If I were invited to her party, I’d go (If 2) If I had been invited to her party, I would have gone (If 3) Since there is no context, any of them would be correct. [click] In the other example the grammar is presented in context, a conversation. If this had been about If clauses, it might have been: “ I’m sorry. I didn’t know she was going to have a party. If I ______________ (be invited), I ___________ (go).” Obviously If 3.
  • What will you test? [click] Vocabulary? Refer participants to the HO2. Have them look at the vocabulary examples and compare them in pairs. What are the differences? When they finish giving you some differences go to the next slide.
  • [click] The example on the left is a traditional vocabulary section. It tests if they learned a vocabulary list, but it doesn’t test if they really know how to use the vocabulary. [click] The example on the right tests many different problems students can have when they work with vocabulary: Difficult pairs: (1) (3) (6) Collocation: (2) (4) (5) The context lets you test more.
  • Ask participants how they could test functions…collect some ideas.
  • Go over the definition of functions just in case someone doesn’t know what they are.
  • Go through the examples. Emphasize how the following aspects become more complex as we go down the list: Grammar structures (easy to more difficult) Length of sentences (short to long) Register (from informal to formal)
  • Get some examples of functions [click] Then ask where they can find the functions in the text (in the contents at the beginning of the book) Have teachers look at the Overview for Attitude 2 (SB)… Which functions are similar? Look at Units 4-5…how are the functions presented?
  • This is the best type of section to test students’ knowledge of functions. (In HO2) [click] Students complete a conversation (or paragraph) with sentences (or even phrases) that communicate the correct function logically. Go over the first example. Show that there isn’t one correct answer. The first one could be: It has / I have / There is a big living room. Elicit possible answers for # 2 and 3. The section can be even more open as the other two option show. Elicit possible answers. Have participants compare this with the grammar and vocabulary sections they have seen. What are the differences? Emphasize that here the purpose is communication and that communication can occur even if the students don’t use complete sentences or make some grammar errors. This will be seen later in the workshop when they learn to correct these sections.
  • It is important that participants realize they can’t just take exercise sections from textbooks and use them to test. The purposes behind textbook exercises and examinations are the same as those between “in class practice” and “during exam” activities. The exam sections have different purposes and, therefore, different structures. Go over this chart carefully… Aims: In class the purpose is to learn, in exams it is to get feedback on the learning Content: activities in class are process oriented (the “doing” is usually more important than the result), exams are product oriented. In language exams it doesn’t matter how you get to a’s the result that is graded. activities in class are also open-ended, there isn’t always a result, they can continue for days; on exams they have to close. Learner activities: in class students basically know the material or can use books and dictionaries to find out what they don’t know, on exams they often don’t know the material classroom activities are success-oriented, students are helped to succeed, exams are often success oriented, but not for all students you can have peer-teaching in classroom activities, but working in groups is ususally frowned on during exams. Teacher activity: In class, the teacher helps students improve their performance, on exams the teachers might help the student understand what he is supposed to do, but the teacher doesn’t help the student directly with the answers. Classroom climate: The climate in class is cooperative, students help each other, but on exams it could be competitive Classroom activities are relaxed, exams are tense There is intrinsic motivation in class (students often are motivated to do well by their own internal desires), but it is extrinsic during exams (parents, schools, grade pressure, etc usually motivate students to do well).
  • Balance means it adds up to 100% For example, you don’t need to have a 50-50 balance. 80-20, 70-30, 40-60 are also balances. The balance depends on the school situation…. BUT all three should be present in an exam.
  • Go over the definitions…be sure participants understand them… Accuracy and Fluency balance depends on what the students will be doing with English. Ask the participants: What accuracy/fluency balance would you recommend for tourism students? What is more important for them? (Fluency-maybe 70-30 over accuracy). What balance for students who will be translators? (Accuracy—maybe 80-20 over fluency).
  • In production, the student writes more than one word (remind them of Complete the Conversation for testing functions) The student can be more creative and the teacher has to be more alert because more than one answer might be correct. In recognition (for example, multiple choice or true/false), there is only one correct answer and it isn’t creative. Production means more work. Teachers with large number of students can’t handle a 50-50 balance. A good exam could be 20-80 or 30-70 (production/recognition) and still be fair. TOEFL and other similar exams are all recognition…they never test whether the student can produce language. Years ago this led to a big influx of Asian students into US universities. They had studied grammar and reading, but no speaking or writing. They did great recognizing correct answers on the TOEFL and got very high scores, but when they arrived in the US, authorities realized they couldn’t say two words in English….The TOEFL exam was revised and now includes writing sections and often oral interviews are required to study in the US.
  • Sections can also be objective or subjective. It is good to have some subjective sections, but an exam with sections that are all graded subjectively make it difficult to judge student ability between different teachers…. However, even though they are more difficult to grade, they do give more information about students’ ability. The limitations can be overcome if the graders are trained…. This is a very common problem with oral grading.
  • HO 4 ( Put participants into pairs and have them look at each exam section. For each one, they identify it as Accuracy/fluency, production/recognition and subjective/objective. About 15 mintues. Then go over all of them using the slides.) We are going to look at some exam sections and identify if they test accuracy/fluency, if they are designed to be production or recognition sections and if they are to be graded subjectively or objectively. Have participants go over each section in pairs using the handout. Then use this slide and the following ones to go over them. Answers: Fluency: Have participants show you there are more than one possible answer for most of the items. (for example, (1) What’s your name? / Your name, please? / Name, please / etc.) Production: Students write complete sentences or phrases. Subjective: The grader has to understand possible answers.
  • Answers: Fluency: it practices functions through vocabulary chunks which are essential for fluent conversations. Recognition: There is only one right answer. Objective: Anyone could grade it given an answer key
  • Answers: Accuracy: This is testing correct use of the present tense. Recognition: In reality all the student needs to do is find the subject and put the verb in the correct form Objective: There is only one correct answer.
  • Answers: Fluency: This is similar to the first example, but it is limited by cues. However, the students still have some freedom to be creative: (1) Look at those earrings / Look at these earrings / Look earrings ) Production: They are writing phrases Subjective: The grader must understand English to tell if a student’s answer communicates clearly.
  • Answers: Accuracy: tests grammar structures taught in the text Recognition: Just find the answer in the box and write it. Objective: One correct answer
  • Answers: Fluency: This requires the students to use discourse cues to order the conversation. It goes beyond simple accurate sentences. Discourse cues are essential for effective communication. Recognition: They just copy the sentences Objective: There is one correct order.
  • Go over slide with participants…. If necessary, go back and look at the examples on the previous slides and on HO 4.
  • It’s easy to correct an “A, b, c” section, but how do you correct a “complete the conversation” section so that it really tests communicative ability and competence?
  • Go over the summary
  • These are examples of partial credit. Go over then one-by-one. Ask participants if they would give partial credit or not if they were grading communication… First example: Does it communicate? Would you be able to answer the student’s questions? Probably. If it were a C1 student on the first exam, I’d give credit, but write in the corrections. On later exams, I’d probably not be a generous. Second example: The meaning is different. In the correct answer, you didn’t invite me, in the student’s answer you might do so in the future. Although the grammar is correct, it doesn’t communicate and I wouldn’t give any credit. Third example: Click slowly, discussing as you go. The first one communicates and if it were the first time students had worked with the past I’d accept it and just write in the correction. They use “yesterday” to indicate the tense. [click] This could also be accepted if it were in answer to the question. “what did you do yesterday?”. Correcting these sections, you have to ask “does the answer communicate the idea required by the conversation.” If so, accept it or give partial credit depending on the level of the students.
  • This is the exam section used in the activity (HO7). Go over it and get possible answers before they begin working. Have them work in pairs, grading the students’ work. (15 minutes) Then go over the worksheet, comparing how they graded.
  • Hello, I’m JoAnn Miller. In this talk I will share with you some oral exam procedures that I developed for a private university in Mexico.
  • First some definitions. Note that the accuracy/fluency distinction is valid for both oral and written production and can even be considered for the input skills of listening and reading. In this talk we will concentrate on oral production. I think the key terms in these definitions are: effective communication, continuous speech without a perfect command of the more technical aspects of oral production.
  • Accuracy is the opposite—it stresses grammatically correct sentences and does not always include the continuous speech aspect of fluency. Our students need to be able to use both abilities in their professional lives. Since, as we will see in a few minutes, accuracy assumes time for preparation, we can think of the following experiences your students might have to face.
  • Imagine one of your students works in a factory in Mexico that is expecting some Japanese visitors who don’t speak Spanish. Since no one at the factory speaks Japanese and the visitors do speak English, your student has been asked to do two different activities: The first is give the guests a tour of the factory. This involves fluency since although she can plan a bit about what she will say during the tour, there will be unexpected questions from the visitors. The visitors, also being second language learners, will be tolerant of any mistakes she might make just as long as they can understand what she wants to communicate. The presentation is later in the week and allows your student to prepare. She can write out her talk, prepare Power Point slides and even rehearse in front of her friends. This previous preparation allows for accuracy and the visitors (as well as her bosses) will be expecting a more formal, “correct” performance.
  • I became aware of the importance of the accuracy / fluency distinction when the university where I was working adopted a task-based syllabus. Here you can see what a task is. Note that students are not limited by accuracy when doing a task..Tasks are fluency practices.
  • Tasks are activities in which students experiment with the language in order to communicate their ideas without worrying directly about accuracy.
  • However, it is recognized that accuracy is also important. Accuracy comes through preparation.
  • This balance will depend on the needs of the students. Students who are planning to go into tourism will need more fluency practice. Students who are interested in translation will need more accuracy practice. Most students fall into the middle ground and will do well with practice in both skills.
  • I was coordinating the English language program at a large private university in Mexico. The courses were institutional—we supplied the programs, texts and exams for all students. The exams we will be seeing here were for the BA program. The students in this program have 5 hours of English courses a week: 4 in a classroom and one hour in a language lab. They usually have different teachers for the classroom and the lab. The oral exams are given in the lab. This puts less pressure on the students when practicing oral English in the classroom since they know that teacher will not be grading their production.
  • Since we were giving both fluency and accuracy practice in class, we felt this should be reflected in both the written and oral exams. In the written exams, besides the more traditional formats, students are also asked to produce parts of a conversation. These responses are graded based on if they communicate a logical idea or not. Minor errors in grammar are not counted, but errors in function are since it is possible for a student to make a number of grammatical errors and still communicate while an error in function usually impedes communication.
  • The oral exams are based on role plays. The accuracy exam is given first since students need preparation time. The fluency exam has no preparation. Both role plays are presented only for the teacher. The other students are either preparing their presentations or have already left. Students spend an average of 3 minutes per pair doing their role plays which are designed to be short. This allows the teacher to test the entire group of 30 in a 50 minute class period.
  • As students come into the lab they are put into pairs and given a role play card. They are told they will have AT LEAST five minutes to prepare. They are also told that as soon as they present their role plays for the teacher, they can leave. This encourages many of them to work fast. While they are preparing they can use any resource they want: textbooks, dictionaries or even ask questions. When they are ready, they come to the front of the class. They can have no notes or papers with them, so even if they wrote out their roles, they can’t read them. After they have performed the accuracy role play, the teacher gives them, either individually or in pairs, instructions for the fluency role play. They are given no preparation time.
  • This is the 5 points scale we used for grading. It was developed from numerous popular scales and adapted to our specific needs. Students can only get 0 if they do not present the exam. Most students get between 3 and 4 points on both scales.
  • These are some of the role plays that were used for testing accuracy. They are cut into cards. The role plays vary according to level and reflect functions practiced in the textbook. There are various cycles in use for each exam to discourage dishonesty, but in reality, since students are given preparation time, it doesn’t matter much if they get one of the cards before class since they can’t know which of the role plays they will be asked to do.
  • This is an example of the kinds of role plays used for the fluency part of the exam. The teacher can choose which role play to use to add variety and to not let the students know which they will get.
  • In order to make grading easier, this is the form used by the teachers to record the points. In conclusion, teachers and students have responded favorably to this kind of testing since they feel it allow comprehensive oral testing in a relatively friendly environment.
  • This is the 5 points scale we used for grading. It was developed from numerous popular scales and adapted to our specific needs. Students can only get 0 if they do not present the exam. Most students get between 3 and 4 points on both scales.
  • Communicative testing written and oral-cccn--wb

    1. 1. Communicative Testing JoAnn Miller Macmillan Publishers [email_address]
    2. 2. Kinds of Testing <ul><li>Placement tests </li></ul><ul><ul><li>student’s suitability to take a specific course </li></ul></ul><ul><ul><li>based on specific textbook </li></ul></ul><ul><li>Proficiency tests </li></ul><ul><ul><li>check students’ progress in general </li></ul></ul><ul><ul><li>TOEFL, First Certificate, etc. </li></ul></ul><ul><li>Achievement tests </li></ul><ul><ul><li>check how much a student has learned </li></ul></ul><ul><ul><li>based on what a student has studied in a specific course </li></ul></ul>
    3. 3. What are your exams going to be like? <ul><li>Many variables: </li></ul><ul><ul><li>Institution </li></ul></ul><ul><ul><li>Students </li></ul></ul><ul><ul><li>Teachers </li></ul></ul><ul><ul><li>Text </li></ul></ul><ul><ul><li>Time </li></ul></ul>
    4. 4. Teachers <ul><li>Level of English </li></ul><ul><li>Lack of mathematical skills </li></ul><ul><li>Time factor </li></ul><ul><li>Ease of grading </li></ul><ul><li>Answer key </li></ul>
    5. 5. Students <ul><li>Younger students </li></ul><ul><ul><li>More images </li></ul></ul><ul><ul><li>Shorter exams </li></ul></ul><ul><ul><li>Humor </li></ul></ul><ul><li>Older students </li></ul><ul><ul><li>Professionalism </li></ul></ul><ul><ul><li>Humor </li></ul></ul>
    6. 6. How many points for each skill? <ul><li>If institution tells you, just follow through </li></ul><ul><li>If not, base exam on the textbook (the common denominator) </li></ul><ul><ul><li>General text analysis </li></ul></ul><ul><ul><li>How much time is spent on each skill </li></ul></ul><ul><ul><li>Count exercises in a few units, determine percent </li></ul></ul><ul><li>Keep institutional goals in mind </li></ul>
    7. 7. Content Validity <ul><li>Assessment should be based on a content-analysis of the text being used </li></ul>
    8. 8. Content Analysis <ul><li>You must test only material students have seen </li></ul><ul><li>The only common denominator is the textbook </li></ul><ul><li>Analysis of percent of time spent on each topic (grammar structure, vocabulary item, function, etc.) </li></ul>
    9. 9. Content Analysis: Information from the contents Functions (10 points): Talking about imitation products Talking about food and food festivals Discussing the movie industry Making a business plan Grammar (5 points): Nouns in groups Indefinite Pronouns Vocabulary (10 points): Food Business language ///// /// ///// ///// ///// //// ///// /// ///// ///// /// 8 10 9 5 32 3 5 8 5 3 8 8 / 32 =__% 25% 31% 28% 16% 25% X 10 = ___pts 2.5 pts 3 pts 3 pts 1.5 pts 37% 2 pts 63% 3 pts 63% 6 pts 37% 4 pts
    10. 11. Communicative testing <ul><li>We teach “communicatively” but we test “traditionally”. </li></ul><ul><li>What IS communicative testing? </li></ul><ul><li>Communicative testing means testing in context . </li></ul>
    11. 12. Grammar? What will you test?
    12. 13. Which version? Why? <ul><li>Circle the correct answer </li></ul><ul><li>1. Do you like __________? </li></ul><ul><li>swimming b. to swum c. swim </li></ul><ul><li>2. Where ________ live? </li></ul><ul><li>does she b. she does c. she </li></ul><ul><li>3. I _________ speak French. </li></ul><ul><li>no speak b. doesn’t c. don’t </li></ul><ul><li>4. What __________? </li></ul><ul><li>a. does he do b. does he c. he does do </li></ul><ul><li>Write the correct forms of the words in parentheses. </li></ul><ul><li>Alice: Where (1)______ you </li></ul><ul><li>_________ (live)? </li></ul><ul><li>Bart: Acapulco. </li></ul><ul><li>Alice: My brother (2)___________ (go) </li></ul><ul><li>there every summer on vacation, </li></ul><ul><li>but he (3)_________(not speak) </li></ul><ul><li>Spanish. </li></ul><ul><li>Bart: Acapulco (4)_________ (attract) </li></ul><ul><li>tourists from all over the world. </li></ul><ul><li>Many people there </li></ul><ul><li>(5)___________(speak) English </li></ul><ul><li>very well. What about you, </li></ul><ul><li>(6)______ you ________(speak) </li></ul><ul><li>Spanish? </li></ul><ul><li>Alice: A little. </li></ul>
    13. 14. Vocabulary? What will you test?
    14. 15. Which version? Why? <ul><li>Match the letters (a to e) with the numbers (1 to 5). </li></ul><ul><li>1. Your mother’s husband is your___. </li></ul><ul><li>2. Your mother’s father is your___. </li></ul><ul><li>3. Your mother’s brother is your___. </li></ul><ul><li>4. Your uncle’s son is your___. </li></ul><ul><li>5. Your father’s sister is your ___. </li></ul><ul><li>uncle </li></ul><ul><li>cousin </li></ul><ul><li>aunt </li></ul><ul><li>father </li></ul><ul><li>grandfather </li></ul><ul><li>Underline the word in each pair that completes the conversation correctly. </li></ul><ul><li>My (1)[ uncle / aunt ] likes (2) [ playing / going to ] movies. He is my father’s (3)[ sister / brother] . He’s (4)[ heavy / average ] and he has (5)[ blue / brown ] hair. His birthday is on October (6)[twelve / twelfth ] . </li></ul>
    15. 16. Functions? What will you test?
    16. 17. What is a function? <ul><li>The communicative purpose of the users of the language. </li></ul><ul><li>How language is used. </li></ul><ul><li>Usually expressed as gerunds: introducing, apologizing, asking directions, requesting </li></ul>
    17. 18. Examples of a Functional Cycle <ul><li>Function: Requesting </li></ul><ul><li>(1) Open the window, please. </li></ul><ul><li>(2) Would you open the window? </li></ul><ul><li>(3) Could you please open the window? </li></ul><ul><li>(4) Would you mind opening the window? </li></ul><ul><li>(5) I was wondering if you would mind opening the window. </li></ul><ul><li>(6) I’d be grateful if you opened the window. </li></ul><ul><li>Each time the difference in register is emphasized. </li></ul>
    18. 19. Where can you find them in your textbook?
    19. 20. How to test? Complete the conversation. <ul><li>Complete the conversation logically. Use the words in parentheses. </li></ul><ul><li>Miriam: Tell me about your new apartment. </li></ul><ul><li>Mary: (1)____________________(big / living room). </li></ul><ul><li>Miriam: (2)___________________(how / bedrooms)? </li></ul><ul><li>Mary: There are two, but (3)________(any furniture) </li></ul><ul><li>in one of them. </li></ul>Or: Miriam: Tell me about your new apartment. Mary: (1)____________________(living room). Or: Miriam: Tell me about your new apartment. Mary: (1)___________________________.
    20. 21. Practice vs Testing In class practice During exam Goals Content Learner activity Teacher activity Class-room climate learning feedback on learning process oriented product oriented open ended close ended ss know material students might not know success-oriented success/failure oriented peer teaching no peer teaching helps performance gives tasks cooperative competitive relaxed tense intrinsic motivation extrinsic motivation
    21. 22. Balance <ul><li>Ideally an exam will balance: </li></ul><ul><ul><li>Accuracy and fluency </li></ul></ul><ul><ul><li>Production and recognition </li></ul></ul><ul><ul><li>Objective and subjective sections </li></ul></ul>
    22. 23. Accuracy and Fluency <ul><li>Fluency </li></ul><ul><ul><li>The ability to produce written and / or spoken language with ease </li></ul></ul><ul><ul><li>Communicate ideas effectively </li></ul></ul><ul><ul><li>The ability to use vocabulary chunks (phrases) to facilitate communication </li></ul></ul><ul><li>Accuracy </li></ul><ul><ul><li>Ability to produce grammatically correct sentences </li></ul></ul>
    23. 24. Production and Recognition <ul><li>Production </li></ul><ul><ul><li>Student writes more than one word </li></ul></ul><ul><ul><li>Can be creative / involves more “mental” work </li></ul></ul><ul><ul><li>More than one answer may be possible </li></ul></ul><ul><li>Recognition </li></ul><ul><ul><li>Student recognizes correct answer </li></ul></ul><ul><ul><li>Not creative </li></ul></ul><ul><ul><li>Only one correct answer </li></ul></ul>
    24. 25. Objective and Subjective Sections <ul><li>Objective </li></ul><ul><ul><li>There is only one answer </li></ul></ul><ul><ul><li>Anyone can correct the exam </li></ul></ul><ul><ul><li>No surprises </li></ul></ul><ul><ul><li>No argument from students </li></ul></ul><ul><li>Subjective </li></ul><ul><ul><li>There is more than one possible answer </li></ul></ul><ul><ul><li>Corrector must be trained and experienced </li></ul></ul><ul><ul><li>There can be surprises </li></ul></ul><ul><ul><li>Students can protest grading </li></ul></ul>
    25. 26. <ul><li>Grammar / Vocabulary / Functions? </li></ul><ul><li>Accuracy or fluency? </li></ul><ul><li>Production or recognition? </li></ul><ul><li>Subjective or objective? </li></ul>
    26. 27. <ul><li>Grammar / Vocabulary / Functions? </li></ul><ul><li>Accuracy or fluency? </li></ul><ul><li>Production or recognition? </li></ul><ul><li>Subjective or objective? </li></ul>
    27. 28. <ul><li>Grammar / Vocabulary / Functions? </li></ul><ul><li>Accuracy or fluency? </li></ul><ul><li>Production or recognition? </li></ul><ul><li>Subjective or objective? </li></ul>
    28. 29. <ul><li>Grammar / Vocabulary / Functions? </li></ul><ul><li>Accuracy or fluency? </li></ul><ul><li>Production or recognition? </li></ul><ul><li>Subjective or objective? </li></ul>
    29. 30. <ul><li>Grammar / Vocabulary / Functions? </li></ul><ul><li>Accuracy or fluency? </li></ul><ul><li>Production or recognition? </li></ul><ul><li>Subjective or objective? </li></ul>
    30. 31. <ul><li>Grammar / Vocabulary / Functions? </li></ul><ul><li>Accuracy or fluency? </li></ul><ul><li>Production or recognition? </li></ul><ul><li>Subjective or objective? </li></ul>
    31. 32. What is an exam section? <ul><li>A certain number of items testing the same skill / aspect </li></ul><ul><li>To be communicative, they should be written as a conversation, note, letter, or some “real” type of discourse </li></ul><ul><li>All items in a section should be worth the same number of points and test a similar skill (all grammar, all vocabulary, all functions, etc.) </li></ul>
    32. 33. Correcting communicative sections
    33. 34. Correcting <ul><li>Grammar, Reading, Vocabulary, Listening </li></ul><ul><li>In general these sections are all right or all wrong . </li></ul><ul><li>We rarely give partial credit. </li></ul><ul><li>These sections test accuracy . </li></ul><ul><li>Communicative sections </li></ul><ul><li>You can give partial credit </li></ul><ul><li>These sections test fluency. </li></ul><ul><li>Ask yourself if the S’s answer communicates what the S wants to say . </li></ul>
    34. 35. Examples of partial credit <ul><li>Correct answer: What’s your name? </li></ul><ul><li>Student writes: What you name? </li></ul><ul><li>Correct answer: If you invited me, I’d go. </li></ul><ul><li>Student writes: If you invite me, I go. </li></ul><ul><li>Correct answer: I went to the movies yesterday. </li></ul><ul><li>Student writes: I go to the movies yesterday. </li></ul><ul><li> I go to the movies. </li></ul>
    35. 36. IV. The clerk knows Cleopatra. Caesar asks the clerk about Cleo. Complete the conversation. Use the words in parentheses. ( 4 points, .5 each) Clerk : Yes, I know her. Julius : (1) _______________________________________ (work) ? Clerk : (2) ________________________________ ( palace downtown). Julius : (3) _______________________________________ ( do) ? Clerk : (4) ___________________________________ ( help people). Julius : (5) ___________________________________ ( close friend)? Clerk : Yes, (6) ____________________________________ (funny). Julius : (7) _______________________________________ ( sports)? Clerk : Yes, (8) ___________________________________ ( tennis ). Actual Student Responses on the worksheet
    36. 37. Example 1 <ul><li>Clerk : Yes, I know her. </li></ul><ul><li>Julius : (1) Where does she works? </li></ul><ul><li>Clerk : (2) She does work palace downtown . </li></ul><ul><li>Julius : (3) Where do she does? </li></ul><ul><li>Clerk : (4) She does help people. </li></ul><ul><li>Julius : (5) Where does she close friend ? </li></ul><ul><li>Clerk : Yes, (6) she does funny. </li></ul><ul><li>Julius : (7) Where does she lift sports ? </li></ul><ul><li>Clerk : Yes, (8) she does play tennis. </li></ul>
    37. 38. Example 2 <ul><li>Clerk : Yes, I know her. </li></ul><ul><li>Julius : (1) You do? </li></ul><ul><li>Clerk : (2) I’m work in the palace downtown . </li></ul><ul><li>Julius : (3) What do you do? </li></ul><ul><li>Clerk : (4) I’m help help people. </li></ul><ul><li>Julius : (5) How you close friend ? </li></ul><ul><li>Clerk : Yes, (6) they are funny </li></ul><ul><li>Julius : (7) ar you play sports ? </li></ul><ul><li>Clerk : Yes, (8) I’m play. </li></ul>
    38. 39. Example 3 <ul><li>  Clerk : Yes, I know her. </li></ul><ul><li>Julius : (1) Where does she work? </li></ul><ul><li>Clerk : (2) She work at the palace downtown . </li></ul><ul><li>Julius : (3) What does she does? </li></ul><ul><li>Clerk : (4) She helps people. </li></ul><ul><li>Julius : (5) Does she have a close friend ? </li></ul><ul><li>Clerk : Yes, (6) she does. She’s very funny.. </li></ul><ul><li>Julius : (7) Does she like sports ? </li></ul><ul><li>Clerk : Yes, (8) she play tennis. </li></ul>
    39. 40. Example 4 <ul><li>  Clerk : Yes, I know her. </li></ul><ul><li>Julius : (1) What does she works? </li></ul><ul><li>Clerk : (2) She works in palace downtown . </li></ul><ul><li>Julius : (3) What does she do? </li></ul><ul><li>Clerk : (4) Work with help people. </li></ul><ul><li>Julius : (5) Are you close friend ? </li></ul><ul><li>Clerk : Yes, (6) she is funny. </li></ul><ul><li>Julius : (7) Are you practice sports ? </li></ul><ul><li>Clerk : Yes, (8) she plays tennis. </li></ul>
    40. 41. Where written and oral meet <ul><li>Functions and their relation to communication </li></ul><ul><li>Accuracy vs. Fluency </li></ul><ul><li>Grading of communicative written sections </li></ul>
    41. 42. Oral Testing of Accuracy and Fluency
    42. 43. Fluency <ul><li>The ability to produce written and / or spoken language with ease </li></ul><ul><li>Speak with a good but not necessarily perfect command of intonation, vocabulary and grammar </li></ul><ul><li>Communicate ideas effectively </li></ul><ul><li>Produce continuous speech without causing comprehension difficulties or a breakdown in communication </li></ul>Longman Dictionary of Language Teaching and Applied Linguistics
    43. 44. Accuracy <ul><li>Ability to produce grammatically correct sentences </li></ul><ul><li>May not include the ability to speak or write fluently. </li></ul>Longman Dictionary of Language Teaching and Applied Linguistics
    44. 45. An example: foreign visitors <ul><li>Tour of factory—fluency; must be prepared for unexpected questions and explanations </li></ul><ul><li>Formal presentation of a product—accuracy; time to plan, rehearse </li></ul>
    45. 46. What is a task? <ul><li>A goal-oriented activity in which learners use language to achieve a real outcome . </li></ul><ul><li>Learners use whatever target language resources they have in order to solve a problem, do a puzzle, play a game, or share and compare experiences. </li></ul>
    46. 47. Fluency in Tasks <ul><li>“ Learners need opportunities to process language for communicative purposes as receivers and producers. </li></ul><ul><li>“ These opportunities should be unfettered by the perceived need to conform to teacher expectations in terms of the production of specific language forms.” </li></ul>Dave Willis, “Accuracy, fluency and conformity” Challenge and Change in Language Teaching, J. Willis and D. Willis, ed. Heinemann, 1996. P. 50
    47. 48. Accuracy in Tasks <ul><li>“ Whenever learners are involved in communication they are concerned with accuracy…making the best use of their language systems… </li></ul><ul><li>“ In spontaneous communication [they] have little time to reflect on the language they produce. </li></ul><ul><li>“ If…they are given time to prepare what they have to produce, there will be a concern for formal accuracy…” </li></ul>Dave Willis, “Accuracy, fluency and conformity” Challenge and Change in Language Teaching, J. Willis and D. Willis, ed. Heinemann, 1996. P. 50
    48. 49. Implications <ul><li>“ Teachers should balance issues of fluency and accuracy depending on the specific needs of learners and the resources of time and materials for instruction.” </li></ul>Miriam Eisenstein Ebsworth, “Accuracy Vs. Fluency: Which Comes First in ESL Instruction?” , ESL Magazine. 1:2, 24-26. March/April 1998.
    49. 50. The teaching situation <ul><li>Private, multi-level (high school, university, post-grad) university in Mexico (17 campuses throughout country) </li></ul><ul><li>Institutional EFL programs </li></ul><ul><li>Total 35,000+ students </li></ul><ul><li>Groups of 25-30 in language lab </li></ul><ul><li>Native and non-native teachers with varying abilities and experience </li></ul>
    50. 51. Grading <ul><li>Objective grading required as much as possible (by government, school and parents) </li></ul><ul><li>Each course: 2 written exams in classroom (functions, structures, vocabulary, reading, writing); 2 exams in lab (listening comprehension and oral production) </li></ul>
    51. 52. Oral Exam <ul><li>Accuracy </li></ul><ul><li>Role play </li></ul><ul><li>5 minutes+ preparation </li></ul><ul><li>Perform for teacher </li></ul><ul><li>1-2 minutes maximum </li></ul><ul><li>Fluency </li></ul><ul><li>Role play with teacher and another student </li></ul><ul><li>No preparation </li></ul><ul><li>Cues only </li></ul>
    52. 53. Oral Exam Organization <ul><li>Students (pairs) given role cards as they enter classroom </li></ul><ul><li>Told to prepare—when finish exam, can leave </li></ul><ul><li>Can use any reference or ask questions </li></ul><ul><li>Come to the front of the class, talk only to teacher </li></ul><ul><li>Perform </li></ul><ul><li>Either stay together or separate for second role-play </li></ul>
    53. 54. 5 point scale <ul><li>Accuracy </li></ul><ul><li>1 ---Little or no language produced. </li></ul><ul><li>2 --Poor vocabulary, serious mistakes in grammar, poor pronunciation. </li></ul><ul><li>3 --Adequate vocabulary, mistakes in grammar, adequate pronunciation. </li></ul><ul><li>4 --Good vocabulary, occasional errors in grammar, good pronunciation. </li></ul><ul><li>5 --Wide vocabulary, very few errors in grammar, very good pronunciation. </li></ul><ul><li>Fluency </li></ul><ul><li>1 ---Little or no communication. </li></ul><ul><li>2 ---Very hesitant and brief utterances, sometimes difficult to understand. </li></ul><ul><li>3 ---Communicates ideas, but hesitantly and briefly </li></ul><ul><li>4 ---Effective communication, but does not elaborate on response. </li></ul><ul><li>5 ---Easy and efficient communication. Elaborates on responses. </li></ul>
    54. 55. Student A: Imagine you area meeting and not in an exam. All of your classmates are at the meeting too. Your partner doesn’t know anyone. Tell him who the people are. at Student B: Imagine you are at a meeting and not in an exam. All of your classmates are at the meeting too. You don’t know anyone. Ask your partner who the people are. Student A: Tell your partner about an accident you or some member of your family had. When he/she tells you, ask some intelligent questions or make relevant comments. Student B: Tell your partner about an accident you or some member of your family had. When he/she tells you, ask some intelligent questions or make relevant comments. Student A: You are making a survey about what people think they will be able to do with telecommunications in twenty years. Ask your partner at least three questions about the topic. Student B: Your partner is making a survey. Answer his/her questions.
    55. 56. <ul><li>II. Roleplay with the teacher. Use to grade fluency. (5 points) (Do not show these questions to the Ss. ) </li></ul><ul><li>Keep students together. Ask them what they plan to do when they finish school. Then ask them to tell you the pros and cons of that job. </li></ul><ul><li>Separate students for a moment. You are going to give one student a message for the other student. For example, ask Student A to tell Student B you are going to meet him/her after class. Then have the student pass on the message. Make the messages a little bit complicated. When you finish, give Student B messages for Student A. </li></ul><ul><li>3. Separate students. Tell student to imagine his girlfriend / her boyfriend is angry. Ask him / her what he / she will do. Then ask a “what if” question: What if he doesn’t believe you? What if he goes out with someone else?, etc. </li></ul>
    56. 57. <ul><li>II. Oral Exam (10 points) </li></ul><ul><li>Role Play 1: 5 4 3 2 1 0 </li></ul><ul><li>Role Play 2: 5 4 3 2 1 0 </li></ul><ul><li>Total Oral points: ___/10 </li></ul>Part I is the listening comprehension exam. It is on the same page.
    57. 58. Practice <ul><li>The best way to understand what oral testing is all about is to experience it yourselves. </li></ul><ul><li>Get into groups of three. </li></ul>
    58. 59. Formal oral testing <ul><li>common and well-developed in standardized tests </li></ul><ul><li>less common and definitely underdeveloped in classroom testing. </li></ul><ul><li>According to Fulcher, it can cost up to £62,867.79 for a school to develop a good oral testing program. </li></ul>Glenn Fulcher in Testing Second Language Speaking (Pearson/Longman, 2003), pp. 159-160.
    59. 60. Problems with oral testing <ul><li>difficulty in investigating validity and reliability due to the nature of the exams themselves. </li></ul><ul><li>In his book, Fulcher examines various task types and rating scales, but is unable to recommend any of them completely due to the problems inherent in oral testing. </li></ul>
    60. 61. Task types <ul><li>Realistic communication. </li></ul><ul><ul><li>The production = relatively unpredictable; similar to what they do in the textbook. </li></ul></ul><ul><li>  Kind of production wanted </li></ul><ul><ul><li>statements only </li></ul></ul><ul><ul><li>question production </li></ul></ul>
    61. 62. Task Organization <ul><li>Individual </li></ul><ul><li>Pairs </li></ul><ul><li>Groups </li></ul>
    62. 63. Individual Task Types <ul><li>talk about a topic given by examiner (statements only) </li></ul><ul><li>talk about an image (statements only) </li></ul><ul><li>interview the examiner or an imaginary person (question production) </li></ul><ul><li>roleplay with examiner </li></ul>
    63. 64. Pair/Group Tasks Types <ul><li>have a conversation-occasionally role play (realistic communication) </li></ul><ul><li>build a conversation around an image-photo/map (realistic communication) </li></ul><ul><li>survey (question-answer) </li></ul><ul><li>problem solving –often role play (realistic communication) </li></ul><ul><li>interview (question-answer) </li></ul>
    64. 65. Practice <ul><li>Different type of presentation </li></ul><ul><li>Script </li></ul><ul><ul><li>For use with less experienced teachers </li></ul></ul><ul><li>Get into groups of four </li></ul>
    65. 66. Now you work… <ul><li>In pairs: </li></ul><ul><li>Choose a unit in the text. </li></ul><ul><li>Write: </li></ul><ul><ul><li>One individual </li></ul></ul><ul><ul><li>One pair </li></ul></ul><ul><ul><li>One group activity </li></ul></ul>
    66. 67. Grading oral exams <ul><li>Evaluating oral production is difficult for teachers </li></ul><ul><ul><li>insecurity in their own abilities </li></ul></ul><ul><ul><li>lack of time available </li></ul></ul><ul><ul><li>lack of understanding of grading instruments. </li></ul></ul><ul><li>Standardized testing </li></ul><ul><ul><li>intensive examiner training </li></ul></ul><ul><ul><li>not possible in most school scenarios. </li></ul></ul>
    67. 68. What can we do? <ul><li>simple grading instruments </li></ul><ul><li>not truly valid or reliable in a statistical sense </li></ul><ul><li>help the classroom teacher or department standardized oral grading. </li></ul>
    68. 69. Instruments <ul><li>Simple Scales </li></ul><ul><li>Formal Rubrics </li></ul><ul><li>A mixture </li></ul>
    69. 70. Simple scales <ul><li>1 --Little or no language produced. Little or no communication. </li></ul><ul><li>2 --Poor vocabulary, serious mistakes in grammar, poor pronunciation. Very hesitant and brief utterances, sometimes difficult to understand. </li></ul><ul><li>3 --Adequate vocabulary, mistakes in grammar, adequate pronunciation. Communicates ideas, but hesitantly and briefly </li></ul><ul><li>4 --Good vocabulary, occasional errors in grammar, good pronunciation. Effective communication, but does not elaborate on response. </li></ul><ul><li>5 --Wide vocabulary, very few errors in grammar, very good pronunciation. Easy and efficient communication. Elaborates on responses. </li></ul>
    70. 71. Problem of standardization <ul><li>examiner’s general knowledge of the production level expected for each course. </li></ul><ul><li>Most experienced teachers have this knowledge and I have always included this warning: </li></ul><ul><li>  </li></ul>
    71. 72. <ul><li>Note: Please take S's level into account. A Course 1 student cannot produce as much language as a Course 3 student. To get 3 points, the student should be able to use structures and vocabulary taught in the course he / she just finished. However, expect errors since the student has not fully acquired the material. To get 5 points, the student may still make a few isolated errors, but will speak much above a typical student at the same level.  </li></ul>
    72. 73. But…remember… <ul><li>Accuracy and Fluency??? </li></ul>
    73. 74. <ul><li>Accuracy </li></ul><ul><li>1 ---Little or no language produced. </li></ul><ul><li>2 --Poor vocabulary, serious mistakes in grammar, poor pronunciation. </li></ul><ul><li>3 --Adequate vocabulary, mistakes in grammar, adequate pronunciation. </li></ul><ul><li>4 --Good vocabulary, occasional errors in grammar, good pronunciation. </li></ul><ul><li>5 --Wide vocabulary, very few errors in grammar, very good pronunciation. </li></ul><ul><li>Fluency </li></ul><ul><li>1 ---Little or no communication. </li></ul><ul><li>2 ---Very hesitant and brief utterances, sometimes difficult to understand. </li></ul><ul><li>3 ---Communicates ideas, but hesitantly and briefly </li></ul><ul><li>4 ---Effective communication, but does not elaborate on response. </li></ul><ul><li>5 ---Easy and efficient communication. Elaborates on responses. </li></ul>
    74. 75. With 2 scales…. <ul><li>You can … </li></ul><ul><ul><li>give two exams…one accuracy and one fluency </li></ul></ul><ul><ul><li>give one exam….but grade both aspects. </li></ul></ul>
    75. 76. Rubrics <ul><li>&quot;a scoring tool that lists the criteria for a piece of work or 'what counts.” </li></ul><ul><li>Generally rubrics specify the level of performance expected for several levels of quality. </li></ul><ul><li>These levels of quality may be written as different ratings (e.g., Excellent, Good, Needs Improvement) or as numerical scores (e.g., 4, 3, 2, 1) which are then added up to form a total score which then is associated with a grade (e.g., A, B, C, etc).  </li></ul>
    76. 77. Combined System <ul><li>Similar to rubrics, but easier to grade </li></ul><ul><li>Example is based on Common European Framework “Can-do” Statements (B2) </li></ul><ul><li>Just check, yes/no. Can the student do it or not. </li></ul>
    77. 78. Thank you very much <ul><li>JoAnn Miller </li></ul><ul><li>[email_address] / [email_address] </li></ul><ul><li> </li></ul>