The document discusses alternative methods for evaluating students, such as portfolios and communicative testing, rather than solely relying on written exams. It provides examples of analyzing course content to determine an appropriate weighting of skills like grammar, vocabulary, and functions to test. The document also provides examples of how to test different language skills communicatively, such as functional language through role plays or conversations, rather than discrete-point grammar questions.

1. Me? Write an exam? How to write a communicative exam JoAnn Miller, Macmillan Publishers [email_address] www.efltasks.net

2. Written Oral Projects Portfolio What kinds of evaluations do you do?

3. Alternative Testing Not paper and pencil Constant not punctual Definition: “ the ongoing process involving the student and teacher in making judgments about the student’s progress in language using non-conventional strategies “ techniques that can be used within the context of instruction and can be easily incorporated into the daily activities of the school or classroom.” The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm

4. Portfolios a collection of student production over time shows the stages in the learning process and the stages of the student’s growth. The Reflective Portfolio: Two Case Studies from the United Arab Emirates, Christine Coombe and Lisa Barlow, Forum Online, http://exchanges.state.gov/forum/vols/vol42/no1/p18.htm

5. Portfolios? more subjective acceptance physical limits

6. Why written tests? Not the only way and maybe not the fairest. But easiest with large numbers of students More objective Accepted by institutions, parents, students

7. Exam Banks A collection of exams for classroom use maintained by the institution itself. Written by the teachers themselves or a special committee Following institutional guidelines Could be various “cycles” covering the same material

8. The benefits of an exam bank Less work for teachers More standardization in large one-campus schools and in multi-campus schools Criteria, instructions and grading Face validity Format Uniform length

9. The basic questions Why test? How often? When? What? How?

10. Student’s need to self-evaluate Institutions require student evaluations Students’ need to be evaluated to see if they are ready for the next level Teacher’s need to evaluate how they have taught Why do YOU test?

11. Once a semester…too little Once a week…too much Depends on students and institution Testing takes time from language exposure in a communicative context How often do YOU test? Why? Do you like it?

12. Why written tests? Not the only way and maybe not the fairest. But easiest with large numbers of students More objective Better? Portfolio: Collection of all the student has done. But…it’s more subjective

13. Exam Banks A collection of exams for classroom use maintained by the institution itself. Written by the teachers themselves or a special committee Following institutional guidelines Could be various “cycles” covering the same material

14. The benefits of an exam bank Less work for teachers More standardization in large one-campus schools and in multi-campus schools Criteria, instructions and grading Face validity Format Uniform length

15. First, the situation Only your students or exam bank? Criteria? Relative weight of grammar, function, vocabulary, reading, listening, writing, etc.? Institutional criteria? Students. Age? Interests? Who will give it? Correct it? What about your situation? How do you decide the relative weight of grammar, function, vocabulary, etc.? Who are your students? Who will give the exam? Correct it?

16. Students Younger students More images Shorter exams Humor Older students Professionalism Humor

17. Teachers Level of English Lack of mathematical skills Time factor Ease of grading Answer key Understand why

18. Factors: Students (younger / shorter) Teachers (mathematical skills) Total points in course Minimum? Maximum? How many points would you like on your exams?

19. How many points for each skill? If institution tells you, just follow through If not, text (the common denominator) General text analysis How much time is spent on each skill Count exercises in a few units, determine percent Keep institutional goals in mind

20. Discussion What are your students like? What are your teachers like? What is the purpose of your English programs?

21. Content Validity Assessment should be based on a content-analysis of the text being used

22. Content Analysis You must test only material students have seen The only common denominator is the textbook Analysis of percent of time spent on each topic (grammar structure, vocabulary item, function, etc.)

23. Content Analysis: Information from the contents Functions (10 points): Talking about imitation products Talking about food and food festivals Discussing the movie industry Making a business plan Grammar (5 points): Nouns in groups Indefinite Pronouns Vocabulary (10 points): Food Business language ///// /// ///// ///// ///// //// ///// /// ///// ///// /// 8 10 9 5 32 3 5 8 5 3 8 8 / 32 =__% 25% 31% 28% 16% 25% X 10 = ___pts 2.5 pts 3 pts 3 pts 1.5 pts 37% 2 pts 63% 3 pts 63% 6 pts 37% 4 pts

25. (Attitude. Macmillan, 2000.)

26. What information can you get out of this overview?

27. Content Analysis: Information from overview Skill Sections in Unit Functions (10 points) Talking about similarites and differences Talking about prefrences Discussing opinions Talking about past events Grammar (10 points) comparative adjectives comparatives and superlatives Superlative adjectives Vocabulary (10 points) sports feelings

28. Categorization of activities Skill Sections in Unit Functions (10 points) Talking about similarities and differences 1 1 Talking about preferences 2 2 2 Discussing opinions 3 3 Talking about past events 4 4 4 4 Grammar (10 points) comparative adjectives 1 1 1 1 w1 w1 comparatives and superlatives 2 2 w2 w2 w3 Superlative adjectives 3 3 w2 Vocabulary (10 points) sports 1 2 2 w1 w1 feelings 2 2

29. Analysis (now you work) Count number of entries in each row (A) Count Functions (10 points) Talking about similarities and differences 1 1 2

30. Get total number of entries per skill aspect (B) Skill Sections in Unit Count Functions (10 points) Talking about similarities and differences 1 1 2 Talking about prefernces 2 2 2 3 Discussing opinions 3 3 2 Talking about past events 4 4 4 4 4 11

31. Divide A by B to get percent of A (C%) Skill Sections in Unit Count % Functions (10 points) Talking about similarities and differences 1 1 2 18% Talking about prefrences 2 2 2 3 Discussing opinions 3 3 2 Talking about past events 4 4 4 4 4 11

32. Multiply C% by the number of points for that skill / aspect (D) = the number of points per topic Skill Sections in Unit Count % Points Functions (10 points) Talking about similarities and differences 1 1 2 18% 1.8 Talking about prefernces 2 2 2 3 Discussing opinions 3 3 2 Talking about past events 4 4 4 4 4 11

33. Skill Sections in Unit Count % Points Functions (10 points) Talking about similarities and differences 1 1 2 18% 1.8 Talking about prefernces 2 2 2 3 27% 2.7 Discussing opinions 3 3 2 18% 1.8 Talking about past events 4 4 4 4 4 36% 3.6 11 Grammar (10 points) comparative adjectives 1 1 1 1 w1 w1 6 43% 4.3 comparatives and superlatives 2 2 w2 w2 w3 5 36% 3.6 Superlative adjectives 3 3 w2 3 21% 2.1 14 Vocabulary (10 points) sports 1 2 2 w1 w1 5 71% 7.1 feelings 2 2 2 29% 2.9 7

34. Communicative testing We teach “communicatively” but we test “traditionally”. What IS communicative testing? Communicative testing means testing in context .

35. Grammar? What will you test?

36. Which version? Why? Circle the correct answer 1. Do you like __________? swimming b. to swum c. swim 2. Where ________ live? does she b. she does c. she 3. I _________ speak French. no speak b. doesn’t c. don’t 4. What __________? a. does he do b. does he c. he does do Write the correct forms of the words in parentheses. Alice: Where (1)______ you _________ (live)? Bart: Acapulco. Alice: My brother (2)___________ (go) there every summer on vacation, but he (3)_________(not speak) Spanish. Bart: Acapulco (4)_________ (attract) tourists from all over the world. Many people there (5)___________(speak) English very well. What about you, (6)______ you ________(speak) Spanish? Alice: A little.

37. Vocabulary? What will you test?

38. Which version? Why? Match the letters (a to e) with the numbers (1 to 5). 1. Your mother’s husband is your___. 2. Your mother’s father is your___. 3. Your mother’s brother is your___. 4. Your uncle’s son is your___. 5. Your father’s sister is your ___. uncle cousin aunt father grandfather Underline the word in each pair that completes the conversation correctly. My (1) [ uncle / aunt ] likes (2) [ playing / going to ] movies. He is my father’s ( 3) [ sister / brother] . He’s (4) [ heavy / average ] and he has (5) [ blue / brown ] hair. His birthday is on October (6) [twelve / twelfth ] .

39. Functions? What will you test?

40. What is a function? The communicative purpose of the users of the language. How language is used. Usually expressed as gerunds: introducing, apologizing, asking directions, requesting

41. Examples of a Functional Cycle Function: Requesting (1) Open the window, please. (2) Would you open the window? (3) Could you please open the window? (4) Would you mind opening the window? (5) I was wondering if you would mind opening the window. (6) I’d be grateful if you opened the window. Each time the difference in register is emphasized.

42. How to test? Complete the conversation. Complete the conversation logically. Use the words in parentheses. Miriam: Tell me about your new apartment. Mary: (1)____________________(big / living room). Miriam: (2)___________________(how / bedrooms)? Mary: There are two, but (3)________(any furniture) in one of them. Or: Miriam: Tell me about your new apartment. Mary: (1)_______________________(living room). Or: Miriam: Tell me about your new apartment. Mary: (1)________________________________.

43. What is an exam section? A certain number of items testing the same skill / aspect To be communicative, they should be written as a conversation, note, letter, or some “real” type of discourse Isolated sentences are difficult to contextualize and don’t represent real communication

44. Determining sections You can combine point values within the same aspect. You can divide point values between two sections You shouldn’t combine point values across aspects / skills…the testing methods are different.

45. Combining point values within the same aspect. Vocabulary (8 points) Family relationships 2.5 Describing people 0.5 say tell ask 1 Phrasal verbs 2 Every day expressions 2 Dividing point values between two sections Vocabulary (8 points) Family relationships 2.5 Describing people 0.5 say tell ask 1 Phrasal verbs 2 Every day expressions 2 1.5 1

46. Accuracy and Fluency Fluency (Communication) The ability to produce written and / or spoken language with ease Communicate ideas effectively Accuracy Ability to produce grammatically correct sentences

47. Production and Recognition Production Student writes more than one word Can be creative / involves more “mental” work More than one answer may be possible Recognition Student recognizes correct answer Not creative Only one correct answer

48. Objective and Subjective Sections Subjective There is more than one possible answer Corrector must be trained and experienced There can be surprises Students can protest grading Objective There is only one answer Anyone can correct the exam No surprises No argument from students

49. So what does this all mean? Accuracy Accuracy Accuracy Accuracy Accuracy / Fluency Fluency Fluency Prod. /Recog. Prod. /Recog. Prod. /Recog. Recognition Recognition Production Production Subj. / Obj Subj. / Obj Objective Objective Objective Subjective Subjective

50. Point Values Give more points to… Production items Give fewer points to… Recognition items Give partial credit in Fluency / Production sections Use fractions only if users are matematical

51. Correcting communicative sections

52. Correcting Grammar, Reading, Vocabulary, Listening In general these sections are all right or all wrong . We rarely give partial credit. These sections test accuracy . Communicative sections You can give partial credit These sections test fluency. Ask yourself if the S’s answer communicates what the S wants to say .

53. Examples of partial credit Correct answer: What’s your name? Student writes: What you name? Correct answer: If you invited me, I’d go. Student writes: If you invite me, I go. Correct answer: I went to the movies yesterday. Student writes: I go to the movies yesterday. I go to the movies.

54. IV. The clerk knows Cleopatra. Caesar asks the clerk about Cleo. Complete the conversation. Use the words in parentheses. ( 4 points, .5 each) Clerk : Yes, I know her. Julius : (1) _______________________________________ (work) ? Clerk : (2) ________________________________ ( palace downtown). Julius : (3) _______________________________________ ( do) ? Clerk : (4) ___________________________________ ( help people). Julius : (5) ___________________________________ ( close friend)? Clerk : Yes, (6) ____________________________________ (funny). Julius : (7) _______________________________________ ( sports)? Clerk : Yes, (8) ___________________________________ ( tennis ). Actual Student Responses on the worksheet

55. Communicative Exam Sections Now you are going to write an exam section. Get out Your content analysis You exam plan You textbook

56. Complete the conversation These sections evaluate a student’s ability to communicate ideas if they are corrected for communication and not for accuracy. They can be written with different degrees of cueing.

57. Testing Reading and Listening Two formats are commonly used: T/ F Multiple Choice

58. True / False Advantages : Can test large amounts of content Students can answer 3-4 questions per minute Disadvantages: They are easy Students have a 50-50 chance of getting the right answer by guessing It is difficult to discriminate between students that know the material and students who don't Need a large number of items for high reliability Designing Test Questions, Grayson H. Walker Teaching Resource Center, The University of Tennessee at Chattanooga, h ttp://www.utc.edu/Administration/WalkerTeachingResourceCenter/FacultyDevelopment/Assessment/test-questions.html

59. Tips for Writing Good True/False items Avoid double negatives. Avoid long/complex sentences. Use specific determinants with caution: never, only, all, none, always, could, might, can, may, sometimes, generally, some, few. Use only one central idea in each item; don't emphasize the trivial. Don't lift items straight from the text. Make more false than true (60/40). (Students are more likely to answer true.)

60. How to “save” a T/F section… Add a third option “ Not mentioned” OR Have student correct F answers But only if students have practiced this version in the textbook

61. Multiple Choice Parts of question Stem Options Distractors XXXXXXXXXXXXXXXXXX? YYYYYYYYY ZZZZZZZZ AAAAAAAA { { Correct answer

62. Stems Don’t include words that do not contribute to the basis for choosing among the options. The American flag has three colors. One of them is (1) red (2) green (3) black vs. One of the colors of the American flag is (1) red (2)green (3) black Or If the pressure of a certain amount of gas is held constant, what will happen if its volume is increased? a. The temperature of the gas will decrease. *b. The temperature of the gas will increase. c. The temperature of the gas will remain the same. Kehoe, Jerard. Writing Multiple-Choice Test Items. ERIC/AE Digest Series EDO-TM-95-3, October 1995. http://www.ericdigests.org/1997-1/test.html

63. Include as much information in the stem and as little in the options as possible. California: a. Contains the tallest mountain in the United States b. Has an eagle on its state flag. c. Is the second largest state in terms of area. *d. Was the location of the Gold Rush of 1849.

64. Avoid irrelevant clues to the correct option. For example, grammatical construction: A word used to describe a noun is called an: *a. Adjective. b. Conjunction. c. Pronoun. d. Verb.

65. Options (Kehoe) 1. Use three or four options. 2. Construct distractors that are comparable in length, complexity, grammatical form Which of the following would do the most to promote the application of nuclear discoveries to medicine? a. Trained radioactive therapy specialists. *b. Developing standardized techniques for treatment of patients. c. Do not place restrictions on the use of radioactive substances. d. If the average doctor is trained to apply radioactive treatments.

66. 8. You have just spent ten minutes trying to teach one of your new employees how to change a typewriter ribbon. The employee is still having a great deal of difficulty changing the ribbon, even though you have always found it simple to do. At this point, you should: a. Tell the employee to ask an experienced employee working nearby to change the ribbon in the future. b. Tell the employee that you never found this difficult, and ask what he or she finds difficult about it. *c. Review each of the steps you have already explained, and determine whether the employee understands them. d. Tell the employee that you will continue teaching him or her later, because you are becoming irritable.

67. 9. Which of the following is the best indication of high morale in a supervisor’s unit? a. The employees are rarely required to work overtime. *b. The employees are willing to give first priority to attaining group objectives, subordinating any personal desires they may have. c. The supervisor enjoys staying late to plan the next day. d. The unit gives expensive birthday presents

68. Ordering Multiple Choice Items Numerical a. 1939 b. 1940 c. 1941 d. 1942 Burton, Steven J. Richard R. Sudweeks, Paul F. Merrill, Bud Wood. How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty , Brigham Young University Testing Services and The Department of Instructional Science. 1991. http://testing.byu.edu/info/handbooks/betteritems.pdf Sequential a. Heating ice from -100°C to 0°C. b. Melting ice at 0°C. c. Heating water from 0°C to 100°C. d. Evaporating water at 100°C. Alphabetical a. C hanging a from .01 to .05. b. D ecreasing the degrees of freedom. c. I ncreasing the spread of the exam scores. d. R educing the size of the treatment effect. After the options are written, vary the location of the answer randomly.

71. Thank you very much JoAnn Miller [email_address] / [email_address] www.efltasks.net