Oral Exams and Rubrics: An introduction


Published on

A talk given at the Universidad Autonoma de Coahuila, August 2013

Published in: Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Oral Exams and Rubrics: An introduction

  1. 1. Oral Exams: Rubrics JoAnn Miller Macmillan Publishers miller@room20.org
  2. 2. A simple way to grade • Situation: – Oral examination of 30 students in 50 minutes. – Examiners had 5 or 6 groups to grade in one or two days. – Goal: Simplicity, fairness, rapidity.
  3. 3. Oral Exam Organization • Students (pairs) given role cards as they enter classroom • Told to prepare—when finish exam, can leave • Can use any reference or ask questions • Come to the front of the class, talk only to teacher • Perform • Either stay together or separate for second role-play
  4. 4. 5 point scale Accuracy (deliberate performance-prepared) 1 ---Little or no language produced. 2 --Poor vocabulary, serious mistakes in grammar, poor pronunciation. 3 --Adequate vocabulary, mistakes in grammar, adequate pronunciation. 4 --Good vocabulary, occasional errors in grammar, good pronunciation. 5 --Wide vocabulary, very few errors in grammar, very good pronunciation. Fluency (effortless interaction-not prepared) 1 ---Little or no communication. 2 ---Very hesitant and brief utterances, sometimes difficult to understand. 3 ---Communicates ideas, but hesitantly and briefly 4 ---Effective communication, but does not elaborate on response. 5 ---Easy and efficient communication. Elaborates on responses.
  5. 5. I. Pair Roleplay. Used to grade accuracy. (5 points) Instructions: • Give each pair of SS a roleplay card. Give them at least 5 minutes to prepare their talk. • Call pairs of Ss to the front. They perform without written notes. • Grade them based on the following scale: [5-point scale presented here.] Note: Please take S's level into account. A Course 1 student cannot produce as much language as a Course 3 student. To get 3 points, the student should be able to use structures and vocabulary taught in the course he/she just finished. However, expect errors since the student has not fully acquired the material. To get 5 points, the student may still make a few isolated errors, but will speak much above a typical student at the same level.
  6. 6. II. Roleplay with the teacher. Use to grade fluency. (5 points) Instructions: • Choose one of the situations given below. • If three people are needed in the situation, keep the pair together. If not, separate Ss and grade individually. • S does not see situations. • Explain the situation to the S and perform your part of the roleplay. • Grade S on the following scale: [5-point scale presented here.] Note: Students should be graded considering their course level.
  7. 7. Student A: Imagine you are at a meeting and not in an exam. All of your classmates are at the meeting too. Your partner doesn’t know anyone. Tell him who the people are. Student B: Imagine you are at a meeting and not in an exam. All of your classmates are at the meeting too. You don’t know anyone. Ask your partner who the people are. Student A: Tell your partner about an accident you or some member of your family had. When he/she tells you, ask some intelligent questions or make relevant comments. Student B: Tell your partner about an accident you or some member of your family had. When he/she tells you, ask some intelligent questions or make relevant comments. Student A: You are making a survey about what people think they will be able to do with telecommunications in twenty years. Ask your partner at least three questions about the topic. Student B: Your partner is making a survey. Answer his/her questions.
  8. 8. II. Roleplay with the teacher. Use to grade fluency. (5 points) (Do not show these questions to the Ss. ) 1. Keep students together. Ask them what they plan to do when they finish school. Then ask them to tell you the pros and cons of that job. 2. Separate students for a moment. You are going to give one student a message for the other student. For example, ask Student A to tell Student B you are going to meet him/her after class. Then have the student pass on the message. Make the messages a little bit complicated. When you finish, give Student B messages for Student A. 3. Separate students. Tell student to imagine his girlfriend / her boyfriend is angry. Ask him / her what he / she will do. Then ask a “what if” question: What if he doesn’t believe you? What if he goes out with someone else?, etc.
  9. 9. II. Oral Exam (10 points) Role Play 1: 5 4 3 2 1 0 Role Play 2: 5 4 3 2 1 0 Total Oral points: ___/10 Part I is the listening comprehension exam. It is on the same page.
  10. 10. But what if you want more….?
  11. 11. Rubrics-Definition(Wikipedia) • an attempt to communicate expectations of quality around a task. • the criteria are public; teachers and students understand the criteria • can also provide a basis for self- evaluation, reflection, and peer review. – accurate and fair assessment – fostering understanding – indicating a way to proceed – ongoing assessment or formative assessment.
  12. 12. Criterion-Referenced a student's aptitude on a task is determined by matching the student's performance against a set of criteria to determine the degree to which the student's performance meets the criteria for the task.
  13. 13. Holistic vs. Analytic Rubrics (Mertler) Holistic • requires the teacher to score the overall process or product as a whole, without judging the component parts separately Analytic • the teacher scores separate, individual parts of the product or performance first, then sums the individual scores to obtain a total score
  14. 14. Holistic (Mertler) When errors in some part of the process can be tolerated provided the overall quality is high
  15. 15. Analytic (Mertler) Performance tasks in which there may be one or two acceptable responses and creativity is not an essential feature of the students' responses.
  16. 16. Why use rubrics (Andrade) • they help define “quality”—Ss know what they are supposed to do • help students become more thoughtful judges of the quality of their own and others’ work • reduce the amount of time teachers spend evaluating student work • their “accordion” nature allows them to accommodate heterogeneous classes • easy to use and to explain
  17. 17. Levels of Performance (1, 2, 3)Criteria Descripters
  18. 18. Descriptors • spell out what is expected of students at each level of performance for each criterion. • tells students more precisely what performance looks like at each level and how their work may be distinguished from the work of others for each criterion. • help the teacher more precisely and consistently distinguish between student work.
  19. 19. Content Validity Assessment should be based on a content-analysis of the text being used
  20. 20. Content Validity • You must test only material students have seen • The only common denominator is the textbook
  21. 21. Goals and Objectives • Goal: A statement of what you expect the student to be able to do on the exam. – Students will be able to order a hamburger successfully. • Objective: A description of the most important components that will lead to a successful completion of the goal. – Vocabulary: demonstrate correct use of functions of asking for something, clarifying, thanking – Pronunciation: demonstrate ability to speak clearly enough to be understood.
  22. 22. Goals and Objectives (Canisius College)
  23. 23. Goals and Objectives, cont.(Moskal) 1.stated goals and objectives for the performance assessment should be clearly aligned with the goals and objectives of instruction. 2.Both goals and objectives should reflect knowledge and information that is worthwhile for students to learn. 3.The relationship between a given goal and the objectives that describe that goal should be apparent. 4.All of the important aspects of the given goal should be reflected through the objectives. 5.Objectives should describe measurable student outcomes. 6.Goals and objectives should be used to guide the selection of an appropriate assessment activity.
  24. 24. Design Goals and Objectives • First: What are you going to grade in an oral exam: pronunciation, grammar, vocabulary, spelling, organization, fluency, listening comprehension, ??????? • Think… what are you evaluating somewhere else? What is important for oral communication and can’t be tested elsewhere.
  25. 25. Example (openMind 2, Unit 1) • Before Unit 1---”In openMind 2 you will….” = oral objectives???? • Plus the Speaking section of the unit..odd numbered units Goal: On the completion of Unit 4, students will be able to describe events in the past describe people’s reactions to past events use follow up questions.
  26. 26. The exams • Have a conversation about a computer disaster. Use follow-up questions. • Have a conversation about last year. Use follow-up questions. • Have a conversation about activities in a specific year. Use follow-up questions. • Have a conversation about an accident. Use follow-up questions. • Have a conversation about a vacation. Use follow-up questions. • Have a conversation about birthday celebration. Use follow-up questions. • Have a conversation about a difficult day Use follow-up questions. • Have a conversation about a trip. Use follow-up questions. • Have a conversation about a funny experience. Use follow-up questions.
  27. 27. Objectives • First reflect on what aspects of the objectives is important and can me measured: – (1) ability to communicate the description of an event in past clearly; (2) distinguish between past tense and past progressive tense use; (3) use of vocabulary related to describing reactions; (4) use logical follow-up questions showing understanding of story. – (1) and (2) could be combined. 1-describe events in the past 2-describe people’s reactions to past events 3-use follow up questions.
  28. 28. • Now, write one or two goals for Unit 5. Begin each one with…”On the completion of Unit 5, students will be able to…..” • When you finish, for each goal, write 2-3 specific objectives. Begin with “Demonstrate (the ability to…).” You can combine related objectives as I did…..
  29. 29. Share your goals and objectives with another group. Feel free to make changes….
  30. 30. Analytic Rubrics Now we will begin writing the rubrics using your goals and objectives. Each goal will be one or more criteria and the objectives will be in the descriptors. Additional criteria could be common to all exams: Pronunciation (as it affects communication); Communication
  31. 31. (Mertler) How many levels of performance will you have? 3, 4?
  32. 32. How can you organize the exams if you use the openMind test builder? The exams are built around units- 9 exams per unit. If you’re going to give ONE exam to students, you could choose the unit at random…. Or….you could give each student 4 exams, but that would take forever….
  33. 33. Review: Steps (Mertler) 1.Re-examine the learning goals and objectives to be addressed by the task. 2.Identify specific observable attributes and behaviors that you want to see (as well as those you don’t want to see) your students demonstrate in their product, process, or performance. For example: the language they should be able to produce (vocabulary), etc.
  34. 34. 3. Brainstorm characteristics that describe each attribute. Identify ways to describe above average, average, and below average performance for each observable attribute identified in Step 2. (Do this for each objective.) 4.Write thorough narrative descriptions for excellent work and poor work for each individual attribute. Describe the highest and lowest levels of performance using the descriptors for each attribute separately. 5.Complete the rubric by describing other levels on the continuum that ranges from excellent to poor work for each attribute. Write descriptions for all intermediate levels of performance for each attribute separately.
  35. 35. And later 7. Collect samples of student work that exemplify each level. These will help you score in the future by serving as benchmarks. 8.Revise the rubric, as necessary. Be prepared to reflect on the effectiveness of the rubric and revise it prior to its next implementation.
  36. 36. Thank you very much… JoAnn Miller miller@room20.org Handouts available at: www.efltasks.net/Presentations Power Point available at: www.slideshare.net/jabbusch
  37. 37. Grading Oral Exams JoAnn Miller, Macmillan Publishers Bibliography Andrade, Heidi Goodrich, Understanding Rubrics, http://learnweb.harvard.edu/alps/thinking/docs/rubricar.htm Carnegie Mellon, Rubrics, http://www.cmu.edu/teaching/designteach/teach/rubrics.html Mertler, Craig A. “Designing Scoring Rubrics for Your Classroom.” Practical Assessment, Research & Evaluation. http://pareonline.net/getvn.asp?v=7&n=25 Moskal, Barbara K. “Recommendations for Developing Classroom Performance Assessments and Scoring Rubrics.” ”. Practical Assessment, Research & Evaluation. http://pareonline.net/getvn.asp?v=8&n=14 Moskal, Barbara K. “Scoring Rubrics: What, When and How?” Practical Assessment, Research & Evaluation. http://pareonline.net/getvn.asp?v=7&n=3 Mueller, Jon. Authentic Assessment Toolbox, http://jfmueller.faculty.noctrl.edu/toolbox/rubrics.htm Rezaei, Ali Reza, Michael Lovorn. “Reliability and validity of rubrics for assessment through writing. Assessing Writing. Vol. 15(1), 2010. Pp. 18*39. http://www.sciencedirect.com/science/article/pii/S1075293510000048 Rubistar, http://rubistar.4teachers.org/ Rubrics: Definition, Tools, Examples, References, The TLT Group, http://www.tltgroup.org/resources/flashlight/rubrics.htm Sample Rubrics, Association for the Assessment of Learning in Higher Education, http://course1.winona.edu/shatfield/air/rubrics.htm Teacher Planet, Rubrics for teachers, http://www.rubrics4teachers.com/ TeacherVision, Creating Rubrics. http://www.teachervision.fen.com/teaching-methods-and-management/rubrics/4521.html teAchnology, Rubrics and Rubric Makers, http://www.teach-nology.com/web_tools/rubrics/