Establishing Speaking Level Benchmarks in an Intensive English Program<br />Sherry Warren<br />University of South Carolin...
Overview<br /><ul><li>The Motivation
The Process
	Preparation
	Structure
	Technology
The Result
Implementing benchmarks in your program</li></li></ul><li>What’s so hard about assessing speaking?<br />
Accommodation<br />“It is a natural feature of authentic conversation that as part of the negotiation of meaning speakers ...
What are your primary concerns when dealing with assessment in an intensive English program?<br />Time.<br />Accuracy.<br ...
How does your program assess speaking ability?<br />Do your students progress through speaking levels?<br />Do teachers de...
English Programs for Internationals<br />The Motivation<br /><ul><li>Under 20 students per class
9 weeks per term
Potential to matriculate into USC subject to scores (TOEFL) and course performance</li></li></ul><li>The Motivation<br />S...
Describing levels of proficiency<br />The Motivation<br />Student Profile<br /> <br />At the beginning of the term, SL6 st...
Assessment Criteria<br />The Motivation<br /><ul><li>For Grammar-Writing class:
Grammar scores on the Michigan Placement Test
Writing Sample
For Reading-Vocabulary class:
Reading and Vocabulary scores on the Michigan Placement Test
For Speaking-Listening class:
Teacher’s score from Oral Interview
Listening and vocabulary scores on the Michigan Placement Test</li></li></ul><li>Assessment Criteria<br />The Motivation<b...
Contributing Factors<br />The Motivation<br />Teachers were not simply being easy graders…<br /><ul><li>Change from having...
Benefits:
more class time for instruction
The teacher knows the student’s strengths and weaknesses
EPI’s recent expansion
New teachers had difficulty knowing whether a student was ready to move up
Is the upper half of the class ready to move up?
Upcoming SlideShare
Loading in …5
×

Creating Speaking Level Benchmarks in an Intensive English Program

1,035 views

Published on

Published in: Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,035
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
55
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • When meaning is negotiated, native speakers accommodate to non-native speakers. Gass and varonis studied phone interviews; teachers get used to non-native speech, but their university teachers colleagues, etc. in the future will not necessarily do so. Proficiency
  • The perfect situation, considering accommodation, would be that oral interviews are conducted anonymously at the end of term by an individual other than the student’s teacher who has had extensive training in oral interview rating.
  • descriptors
  • Creating Speaking Level Benchmarks in an Intensive English Program

    1. 1. Establishing Speaking Level Benchmarks in an Intensive English Program<br />Sherry Warren<br />University of South Carolina<br />sherrytatiana@gmail.com<br />sherry_warren@epi.sc.edu<br />
    2. 2. Overview<br /><ul><li>The Motivation
    3. 3. The Process
    4. 4. Preparation
    5. 5. Structure
    6. 6. Technology
    7. 7. The Result
    8. 8. Implementing benchmarks in your program</li></li></ul><li>What’s so hard about assessing speaking?<br />
    9. 9. Accommodation<br />“It is a natural feature of authentic conversation that as part of the negotiation of meaning speakers adjust their language to features of their interlocutor.”<br />-- Richards and Malvern, 2000<br />Gass and Varonis, 1985<br />
    10. 10. What are your primary concerns when dealing with assessment in an intensive English program?<br />Time.<br />Accuracy.<br />Simplicity.<br />
    11. 11. How does your program assess speaking ability?<br />Do your students progress through speaking levels?<br />Do teachers decide whether to promote students?<br />Are new teachers given adequate training for this?<br />Are teachers expected to be able to holistically assess their students’ speaking proficiency?<br />If so, are they given the tools they need to do so accurately?<br />
    12. 12. English Programs for Internationals<br />The Motivation<br /><ul><li>Under 20 students per class
    13. 13. 9 weeks per term
    14. 14. Potential to matriculate into USC subject to scores (TOEFL) and course performance</li></li></ul><li>The Motivation<br />Students like this<br />
    15. 15. Describing levels of proficiency<br />The Motivation<br />Student Profile<br /> <br />At the beginning of the term, SL6 students are able to<br /> <br />• comprehend, most English spoken at normal speed;<br />• understand main ideas, some details, and important content vocabulary when listening to the radio and watching TV/movies;<br />• speak with pronunciation that is comprehensible without repetition and has acceptable stress, rhythm, and intonation; <br />• speak somewhat confidently but not always accurately;<br />• state and support opinions on abstract topics with some degree of proficiency;<br />• easily participate in everyday conversations with Americans;<br />• sustain conversations (speaking in paragraphs) on a wide range of topics but fluency is uneven.<br />
    16. 16. Assessment Criteria<br />The Motivation<br /><ul><li>For Grammar-Writing class:
    17. 17. Grammar scores on the Michigan Placement Test
    18. 18. Writing Sample
    19. 19. For Reading-Vocabulary class:
    20. 20. Reading and Vocabulary scores on the Michigan Placement Test
    21. 21. For Speaking-Listening class:
    22. 22. Teacher’s score from Oral Interview
    23. 23. Listening and vocabulary scores on the Michigan Placement Test</li></li></ul><li>Assessment Criteria<br />The Motivation<br />Oral Interview: 73%<br />Michigan Placement Test: 27%<br />
    24. 24. Contributing Factors<br />The Motivation<br />Teachers were not simply being easy graders…<br /><ul><li>Change from having formal oral interviews at the end of term to having teacher proctored assessment during class
    25. 25. Benefits:
    26. 26. more class time for instruction
    27. 27. The teacher knows the student’s strengths and weaknesses
    28. 28. EPI’s recent expansion
    29. 29. New teachers had difficulty knowing whether a student was ready to move up
    30. 30. Is the upper half of the class ready to move up?
    31. 31. Is the best student in the class ready to move up?
    32. 32. Pressure from students because of criteria for matriculation</li></li></ul><li>Contributing Factors<br />The Motivation<br />Teachers were not simply being easy graders…<br /><ul><li>Change from having formal oral interviews at the end of term to having teacher proctored assessment during class
    33. 33. Benefits:
    34. 34. more class time for instruction
    35. 35. The teacher knows the student’s strengths and weaknesses very well, so they were able to assess the students the most accurately
    36. 36. EPI’s recent expansion
    37. 37. New teachers had difficulty knowing whether a student was ready to move up
    38. 38. Is the upper half of the class ready to move up?
    39. 39. Is the best student in the class ready to move up
    40. 40. Pressure from students because of criteria for matriculation</li></ul>Teachers needed an objective means of assessing their students at the end of term in order to effectively make recommendations about whether to promote them to the next Speaking-Listening level<br />
    41. 41. Proficiency vs. Achievement<br />Inadequacy of a ‘can do’ checklist<br />Prescriptive<br />Not representative of linguistic competence<br />The need to assess authentic language production<br />Oral Interview<br /><ul><li> ACTFL</li></li></ul><li>General Structure of the Oral Interview<br />The Process<br />Warm up: introductions, explanation of the OI, preliminary assessment<br />Finding the floor (the level at which the student operates)<br />Finding the ceiling (the level at which the student’s language breaks down)<br />Wrap up: closing<br />The interview should be a cycle: Topic A: floor  probe to ceiling<br /> Topic B: floor  probe to ceiling<br /> Topic C: floor  probe to ceiling<br />Breakdown: <br />1. Announced breakdown : “I can’t say that in English.” or “This is very difficult.”<br />2. Using native language or other language<br />Verbal signs of breakdown: stuttering, searching for words, restarting sentences several times, hesitating<br />Nonverbal signs of breakdown: frowning, laughing, rolling eyes<br />Loss of accuracy: grammatical or rhetorical<br />The average interview should be about 15 minutes (a ratable sample is the goal)<br />
    42. 42. Assessment Criteria<br />The Process<br />Displays some features of the level below<br />Displays some features of the level above<br />
    43. 43. Assessment Criteria<br />The Process<br />EXCELLENT—VERY GOOD:<br />Has control of vowels and consonants.<br />GOOD—AVERAGE: Consistently produces most vowels and consonants<br />ADEQUATE—SURVIVAL: Can combines vowels and consonants, but has difficulty producing certain sounds in specific situations<br />
    44. 44. The Process<br />Assessment Criteria<br />
    45. 45. The Process<br />Obtaining the Oral Interviews<br />Teachers suggested a student from their classes who they felt was ready to move to the next level<br />An experienced teacher gave the interview; the class teacher served as a second rater<br />The interview lasted around 15 minutes and was videotaped<br />Inter-rater reliability was high (90%)<br />In the one case where there was a >%10 discrepancy between raters’ scores, a third rater scored the interview <br />
    46. 46. Interviews used for the Benchmarks<br />Interviews used:<br />Students who comfortably placed in the next level at the end of term, based on the oral interview scores from the taped interview and the students’ scores on the MPT<br />A mix of nationalities<br />At least two 15 minute interviews composed each benchmark<br />The students’ full score profile was also presented on the DVD, along with their placement score for SL<br />The Process<br />
    47. 47. Additional Steps taken to Ensure Homogeneity Across Speaking Level Classes<br />The Process<br />Teacher Retreat<br />Discussed and revised goals for each speaking level<br />Viewed oral interviews and discussed criteria for promoting students<br />Decided to change the weight of pronunciation in the assessment criteria (still in process)<br />
    48. 48. Technology used for the Benchmarks<br />The Process<br /><ul><li>Data Collection</li></ul>Higher quality audio equipment was purchased after the first round of data collection because the sound quality was unreliable without the aid of a separate microphone<br />
    49. 49. Technology used for the Benchmarks<br />The Process<br />Mac programs:<br />iMovie<br />iDVD<br />PC program: <br />Roxio Creator 2010<br />DVD-free alternative:<br />Upload files to a simple website for easy distribution<br />
    50. 50. The Finished Product<br /><ul><li>Fits with the communicative language approach at EPI
    51. 51. Interaction between students during a communicative class activity can be used for assessment
    52. 52. New teachers now have a way of setting goals for their own students when teaching a new level
    53. 53. Teachers are provided the means to objectively assess their students and make decisions about whether to promote them</li></ul>The Result<br />
    54. 54. Teacher Feedback<br />The Result<br /><ul><li>New teachers found the benchmarks helpful for assessment
    55. 55. New teachers had a better sense of what their students needed to achieve by the end of term
    56. 56. Students are now better informed about assessment criteria through watching the DVDs during class</li></li></ul><li>Plans for the Future<br />Benchmarks are used every term for oral interview training<br />Teachers have already begun to switch classes and assess one another’s students to deal with the dangers of accommodation<br />Individual interviews may be more appropriate for low levels; these could be implemented during a class in the computer lab<br />The Result<br />
    57. 57. Questions to think about when composing your own benchmarks<br />What is your program’s approach to teaching speaking?<br />What speaking assessment descriptors do you already have?<br />Are your speaking courses geared to helping students achieve communicative competence or is there another goal?<br />
    58. 58. Questions to think about when composing your own benchmarks<br />What do you need to include to adequately deal with diversity in your program?<br /><ul><li> Range of nationalities
    59. 59. Dimensions of proficiency </li></ul> (fluency, grammar, etc.)<br />
    60. 60. Questions to think about when composing your own benchmarks<br />How can teachers use the benchmarks to assess their students without losing class time?<br />
    61. 61. Questions? Comments?<br />

    ×