Facets of TestingL2 Writing Ability Pre-Convention Institute TESOL Conference Philadelphia 2012 Cynthia S. Wiseman, Ed.D. Borough of Manhattan Community College City University of New York
Agenda♦ Introductions♦ General features of language assessment & L2 writing assessment♦ Activity♦ Language Use Argument & L2 Writing♦ Defining the construct: L2 Writing♦ Framework of Task Characteristics♦ Examination of items♦ Review: crossword
Assessment is… Homework Implicit •T & Ss may be unaware -- organic •Continuous – graded, ungraded EXAMS!!! free, pre-writing •Instantaneous – in-class written free wrClassroom •Cyclical – reflective practice observation Explicit •T & Ss are aware of assessment •Clearly distinct from Teaching - ExamQuiz Portfolios Participation Self-Assessment
Purpose of L2 writing assessment: to collectinformation about teaching & learning of L2writing to make decisionsTeacher & Student:–Formative decisions • To correct S errors or not – based on student draft • To change question of inquiry – Revise prompt? description of assignment? • To model a structure – based on student writing, do we need to model a paragraph in class? • To go to next lesson or review? • To go more in-depth in content area – based on thin development, do we need to do more research, discussion, reading? • Use a different strategy to respond to write an essay or explain a genre–Summative decisions • To place, pass, fail or promote a student
Decisions about….♦ Individuals – Selection for admission/employment – screening exams, like CATW – Placement into course of study – Department exams, multiple measures – Certification for profession – LAST (NY Teacher Certification Exam) – Prediction of future performance – TWE, DIALANG♦ Program – Formative, to make changes to improve program – Summative, to continue existing program♦ Research – To decide on new research questions or methodology – To change/modify view/understanding of language phenomenon
Uses of Language Assessments♦ Intended use: – To collect information for making decisions – Beneficial consequences for stakeholders • E.g., ESL writing teacher is teaching lesson on cohesive devices in essay writing – To make decision about instruction – To change/improve instruction so Ss will effectively use cohesive devices to improve writing – Short fill-in-the-blank paragraph to get feedback on Ss’ learning & effectiveness of teaching
Introductory ActivityThink of an L2 writing assessment development situation that you are familiar with. Describe the context, participants, and the test development process. What decisions were to be made based on the results of this assessment?♦ What were some of the intended beneficial consequences of assessment use in this situation?♦ Did any problems came up as a result of this assessment development?
Steps in creation of an assessment♦ Assessment Use Argument (AUA)♦ Set of claims: – Conceptual links between TTs performance on assessment and interpretation about the ability – Decisions to be made – Consequences – What would the AUA of the L2 writing assignment that you just described look like?
Assessment Use Argument --Bachman & Palmer (2010)
Assessment Development Use Bachman & Palmer (2010)
Initial Planning in test development ♦ What beneficial consequences do we want to happen? Who are the stakeholders (i.e., intended TTs, etc.)? Who will be directly affected by the use of the assessment? How? ♦ What are the specific decisions that need to be made to reach the intended consequences? ♦ What do we need to know about the ELL’s language ability as demonstrated in writing to make the intended decision? ♦ What sources could we use to make that decision? Is an existing assessment available? Is it appropriate? ♦ Do assessment tasks correspond to TLU tasks? ♦ Does the developer provide evidence justifying intended uses? ♦ Do we really need to develop our own assessment?
Case scenario♦ Educational Context: CC, US, diverse urban population, 85% ELL/bilingual background, open admissions♦ Decision: To place S in ENG101, 3-credit composition class?♦ Beneficial consequences?♦ What do we need to know about the ELL’s language ability as demonstrated in writing to make the intended decision?♦ What sources could we use to make that decision? Is an existing assessment available? Is it appropriate? – ACT Compass (Writing Sample Test)♦ Do assessment tasks correspond to TLU tasks?♦ Does the developer provide evidence justifying intended uses?♦ Do we really need to develop our own assessment?
Steps in Creating an Assessment♦ Identify the target population/test context♦ Identify the type of assessment♦ Specify the specific purpose of the test♦ Define the construct: Describe the Target Language Use Domain & Target Language Tasks♦ Write specifications for the test♦ Write items/tasks that operationalize the construct & incorporate task characteristics that correspond to TLU tasks♦ Create the test of items/tasks with clear instructions♦ Create an answer key/rubric
Target Language Use (TLU)Domain♦ “…a set of specific language use tasks that the TT is likely to encounter outside of the test itself, and to which we want our inferences about language ability to generalize.” (p. 44) – Distinguishing characteristics of language use tasks to describe language use domain – Inferences that generalize to specific domains in which TT is likely to need to use the language – Inferences about TT’s ability to use language in a target language use domain
Language Use Task♦ Language use task: an activity that involves individuals in using language for the purpose of achieving a particular goal or objective in a particular situation – Specific situations – Goal-oriented – Active participation of language users
TLU Domain Language Use Settings Language Use TasksEnglish for Business Managing & operating •Writing memosCommunication an office •Preparing reports •Taking phone msgs •Writing letters •Writing emails •Texting Negotiating with clients •Writing proposals & customers •Responding to written offers •Writing emails •Texting Promoting products or • Writing advertising services copy • Writing solicitation pitch
Characteristics of Tasks♦ Link between tasks in the domain of test tasks and the domain of non-test tasks –♦ Selection or design of tests that correspond in specific ways to language use tasks♦ Extent and ways TTs’ language ability is engaged♦ Degree of correspondence between characteristics of given test task and a particular language use task: authenticity, validity of inferences, domain to which inferences generalize♦ Control of characteristics of the test task through test design and development
Language Task CharacteristicsCharacteristics of the setting •Physical Characteristics •Participants •Time of taskCharacteristics of the test rubrics •Instructions •Structure •Time allotment •Scoring methodCharacteristics of the input •Format •Language of inputCharacteristics of the expected •Formatresponse •Language of expected responseRelationship between input and •Reactivityresponse •Scope of relationship •Directness of relationship
Framework for Language Task characteristicsCharacteristics of •Physicalthe setting characteristics •Participants •Time of taskCharacteristics of •Instructions •Language (L1, L2)the test rubrics •Channel (aural,visual) •Specification of procedures and tasks Structure •# of parts/tasks •Salience of parts/tasks •Sequence of parts/tasks •Relative importance of parts/tasks •# of tasks/items per part Time allotment Scoring method •Criteria for correctness •Procedures of scoring the response •Explicitness of criteria and procedures
Framework for Language Task characteristicsCharacteristics Format •Channel (aural, visual)of the Input •Form (language, non-language, both) •Language (native, target, both) •Length •Type (item, prompt) •Degree of speededness •Vehicle (live, reproduced, both) Language Language characteristics of input •Organizational Characteristics •Grammatical (vocabulary, syntax, phonology, graphology) •Textual (cohesion, rhetorical, conversational organization) •Pragmatic characteristics •Functional (ideational, manipulative, heuristic, imaginative) •Sociolinguistic (dialect/variety, register, naturalness, cultural references and figurative language) •Topical characteristics
Framework for Language Task characteristicsCharacteristics Format •Channel (aural, visual)of the •Form (language, non-language, both)Expected •Language (native, target, both)Response •Length •Type (item, prompt) •Degree of speededness •Vehicle (live, reproduced, both) Language Language characteristics of input •Organizational Characteristics •Grammatical (vocabulary, syntax, phonology, graphology) •Textual (cohesion, rhetorical, conversational organization) •Pragmatic characteristics •Functional (ideational, manipulative, heuristic, imaginative) •Sociolinguistic (dialect/variety, register, naturalness, cultural references and figurative language) •Topical characteristics
Framework for Language Task characteristicsRelationship Reactivity •Reciprocalbetween •Non-reciprocalInput & •adaptiveResponse Scope of •Broad relationship •Narrow Directness of •Direct relationship •Indirect
Language Use Tests comprised ofTasks♦ Language test: a procedure for eliciting instances of language use from which inferences can be made about an individual’s language ability – Language test should consist of language use tasks – The elemental activities and situations of language use – Performance of a set of interrelated language use tasks♦ Framework of task characteristics
Test Items♦ Is this writing task similar to tasks that the 2nd language learner would have to do in real life?♦ What aspect of second language writing ability is the item testing?♦ Does this task require the TT to demonstrate that aspect of L2 writing ability?
EAP ESL Low Intermediate L2 writing class, 20-25 students/class, 40-minute class, 4x/week,college preparation program w/ 7 levels, level & program exit exam: basic skills writingproficiency test: argumentative essay. Task Characteristics TLU Task Test Task Characteristics of the Setting o Physical characteristics e Participants e Time of task Characteristics of the test rubrics o Instructions o Structure o Time allotment o Scoring method Characteristics of the input o Format o Language of input Characteristics of the expected response o Format o Language of expected response Relationship between input and expected response w Reactivity w Scope of relationship n Directness of relationship
Common Core State StandardsFor ELA & Literacy♦ Council of Chief State School Officers (CCSSO) & National governors Association (NGA) – Aligned with college and work expectations – Standard was included based on the best available evidence that its mastery was essential for college and career readiness in 21st c, globally competitive society
Language use domain: literate ina global 21st century world♦ Close attentive reading to understand and enjoy complex works of literature♦ Critical reading for important points♦ Able to handle large amounts of information♦ Actively seek wide, deep, thoughtful engagement with high-quality literary and information texts that builds knowledge, enlarges experience & broadens worldviews♦ Reflexively demos cogent reasoning and use of evidence essential to private deliberation and responsible citizenship in democracy
Writing: K-5: College & Career Readinessanchor Standards text types and Purposes*♦ Write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence.♦ Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content.♦ Write narratives to develop real or imagined experiences or events using effective technique, well-chosen details, and well-structured event sequences.
Anchor Standards for Writing 6-12Text types and Purposes♦Write arguments to support claims in an analysis ofsubstantive topics or texts, using valid reasoning and relevantand sufficient evidence.♦Write informative/explanatory texts to examine and conveycomplex ideas and information clearly and accurately throughthe effective selection, organization, and analysis of content.♦Write narratives to develop real or imagined experiences orevents using effective technique, well-chosen details, andwell-structured event sequences.
College & Career ReadinessAnchor Standards for LanguageConventions of Standard English♦Demonstrate command of the conventionsof standard English grammar and usage whenwriting or speaking.♦Demonstrate command of the conventionsof standard English capitalization,punctuation, and spelling when writing.
Production & Distribtion ofWriting♦ Produce clear and coherent writing in which the development, organization, and style are appropriate to task, purpose, and audience.♦ Develop and strengthen writing as needed by planning, revising, editing, rewriting, or trying a new approach.♦ Use technology, including the Internet, to produce and publish writing and to interact and collaborate with others.
Agenda♦ Rubrics: facets of L2 writing ability♦ Comparison of rubrics measure L2 writing ability♦ Design of writing task♦ Wrap-up & evaluation
L2 Writing abilityTarget Language Use Domain: Academic writing in acommunity college:Language Use Task: write narrative/persuasive essays Control of content development Rhetorical control Grammatical control Control of register & vocabulary Task fulfillment (McNamara, 1996)
This exceptionally executed essay takes a clear position andexceptionally succeeds in expressing a point of view ortelling a story. The thorough development of ideas includes atleast two outstanding points directly related to the topic, and theexamples used, particularly those from personal experience, are rich,e.g., occasional citation of statistics or reference to personal readings.The essay is clearly and logically organized with nodigressions; the writer demonstrates skillful command ofcohesive devices. Writer demonstrates ability to write in theappropriate academic register and demonstrates extensiverange of vocabulary for academic purposes, with fewproblems in word choice or usage. A few grammatical errors arenoticeable but rarely do the grammar errors interfere with meaning.Sentence variety and complexity reflect a sufficient command ofstandard written English to ensure reasonable clarity of expression.
Holistic Scoring Scale: Criteria for Grading ESL Papers 6This exceptionally executed essay takes a clear position and exceptionally succeeds in expressing a point of view or telling a story. The thoroughdevelopment of ideas includes at least two outstanding points directly related to the topic, and the examples used, particularly those from personalexperience, are rich, e.g., occasional citation of statistics or reference to personal readings. The essay is clearly and logically organized with nodigressions; the writer demonstrates skillful command of cohesive devices. Writer demonstrates ability to write in the appropriate academic register anddemonstrates extensive range of vocabulary for academic purposes, with few problems in word choice or usage. A few grammatical errors are noticeablebut rarely do the grammar errors interfere with meaning. Sentence variety and complexity reflect a sufficient command of standard written English toensure reasonable clarity of expression.5The focus of this competently executed essay is clear but there may be a few digressions. The writer provides substantial support in the development ofthe essay although all examples may not be entirely relevant or appropriate for the topic. The essay is effectively organized, demonstrating systematicallycompetent use of cohesive devices. The writer demonstrates ability to use a variety of patterns of sentence construction but with some errors. Range ofvocabulary for academic purposes is generally competent, and the writer demonstrates accurate and generally appropriate control of word choice, wordforms and idiomatic expressions for academic writing. Some errors in language use, but errors do not generally interfere with meaning. 4In this adequately executed essay the writer’s position is clear despite some possible digressions and contradictions. The writer provides adequatelydetailed support of two or more points that directly relate to the topic. The essay is generally organized, demonstrating generally accurate and appropriateuse of cohesive devices. The writer demonstrates some sentence variety with simple, compound, and some complex sentences though not alwayscorrectly. The essay may contain frequent errors that may occasionally interfere with meaning. Vocabulary is adequate in range, but there are someinappropriate or inaccurate word choices and word forms. 3The essay minimally succeeds in taking a position or relating a narrative with a discernable organizational pattern (introduction, body, conclusion) butmay lack clear focus in development of the central idea. The writer makes an attempt at development although examples are sometimes irrelevant. Thewriter makes minimal use of cohesive devices and he/she demonstrates a minimal range of sentence variety and vocabulary, with some inaccurate and/orinappropriate word choices or inappropriate register. The essay demonstrates minimal control of language, with frequent errors, some of which interferewith meaning. . 2The paper represents limited success in writing a persuasive or narrative essay. The writer provides limited development of the topic with one or morepoints that directly or indirectly relate to the supporting argument or story. The writing shows limited evidence of organization of ideas (paragraphs areoften one sentence) or accurate or appropriate use of cohesive devices. The range of vocabulary and word choice and the use of academic register islimited. The control of language is uneven, with frequent errors, many of which obscure meaning. The writing lacks sentence variety. 1The paper is a failed attempt to write an essay. The writer does not fully develop the topic, lacking related support. There is often no clear organizationalpattern, lacking a clear beginning, middle and end. The writer does not use cohesive devices. The writer demonstrates a narrow range of vocabulary.There is little evidence of appropriate word choice or usage or academic register. The writer demonstrates little control, with frequent errors of all types.The errors generally obscure meaning. The writing lacks basic sentence structure and variety. In some cases, the paper may even be written in thewriter’s first language.
Template for Holistic Rubrics Score Description5 Demonstrates complete understanding of the problem. All requirements of task are included in response.4 Demonstrates considerable understanding of the problem. All requirements of task are included.3 Demonstrates partial understanding of the problem. Most requirements of task are included.2 Demonstrates little understanding of the problem. Many requirements of task are missing.1 Demonstrates no understanding of the problem.0 No response/task not attempted
Criteria Beginning 1 Developing 2 Accomplished 3 Exemplary 4 Score1 Description Description Description Description reflecting reflecting reflecting reflecting highest beginning level movement toward achievement of level of performance of performance mastery level of mastery level of performance performance2 Description Description Description Description reflecting reflecting reflecting reflecting highest beginning level movement toward achievement of level of performance of performance mastery level of mastery level of performance performance3 Description Description Description Description reflecting reflecting reflecting reflecting highest beginning level movement toward achievement of level of performance of performance mastery level of mastery level of performance performance4 Description Description Description Description reflecting reflecting reflecting reflecting beginning beginning level movement toward achievement of level of performance of performance mastery level of mastery level of performance performance
Creating a task to test L2 writing♦ Describe the population you teach♦ Define the purpose for an L2 writing assessment task♦ Define L2 writing ability: Think of a TLU domain task in which that language ability would be demonstrated♦ Using the framework of task characteristics, design a task that would require L2 writing ability to accomplish♦ Share your task with a partner
Self-Assessment♦ I could create an Assessment Use Argument (conceptual link between assessment & intended decisions & consequences).♦ I can define the construct of L2 writing ability.♦ I can articulate a conceptual framework for designing and evaluating L2 writing assessment tasks.♦ I can identify the steps in constructing a language test.♦ I can evaluate strengths and weakness of some specific L2 writing items.♦ I would be able to create a rubric to serve the purposes of an L2 writing assessment in my program.♦ I practiced item/task evaluation.♦ I created an L2 writing task that would be suitable for my program.