IMPLEMENTING AN EMAIL REQUEST TASKFOR ACADEMIC WRITING ASSESSMENT: A STUDY OF RELIABILITY AND TEST DEVELOPMENT Randall Rebman Randall.Rebman@nau.edu Northern Arizona University
LITERATURE REVIEW• Pragmatics is part of most models of communicative competence (Bachman & Palmer, 2010; Canale, 1983; Canale & Swain, 1980).• The testing of pragmatic competence is an underexplored area in second language assessment (Roever, 2011).• Tasks assessing the construct of academic writing can include integrated (reading/writing or listening/writing), independent, and situational-based tasks (Cumming et al., 2000).
TARGET DOMAIN OF ACADEMIC WRITING• It is important to select writing tasks from the Target Language Use (TLU) domain that can be developed for assessment tasks (Bachman & Palmer, 2010). – Select academic writing tasks that: • occur in introductory level academic courses that require writing (Cumming et al., 2000). • occur in academic communication situations between peers and faculty and in general on campus (Youn,S.J., 2009).
TEST PURPOSE• To place L2 students in different levels of writing ability.• To decide if students matriculate into the university from the intensive English program.• To include a representation of writing tasks that second language writers will be required to produce in university contexts.• To explore how a broader coverage of the construct can be attained using a pragmatic task for writing assessment
WHY AN EMAIL TASK?• Needs – EAP students struggle with using the proper conventions of email for communication (Youn,S.J., 2009)• Washback – Potential for positive washback (Crusan, 2010) by encouraging more coverage of pragmatic features of emails (indirect and direct request strategies, genre markers) in course instruction• Adding an email task to a writing test can expand the range of the construct of academic writing that is assessed
LIMITED TEST DOMAIN Independent Task: Prompt-based Argumentative essay Integrated Task: Summary of aSituational-based chart Task: Request to a Professor
RESEARCH QUESTIONS FOR EMAIL TASK (RQ 1&2)1. Can the raters produce consistent ratings of an email writing task using a new rubric?2. Is the email response task testing academic writing ability in a different way than the integrated and independent writing tasks?
PARTICIPANTS• n= 103• International students mainly representing China, Saudi Arabia, Japan, Korea, and Kuwait• Ages ranged from 18-25• All were pre-university students required to take the English placement test to determine level placement into the intensive English program or into Northern Arizona University
METHODS• University IEP students were given the prototype email task as part of a placement/exit test.• A holistic scale for the prototype email task was created by the IEP assessment team using the empirical method (Weigle, 2002) during the piloting of the task.• A 6-point rubric operationalized the construct of writing ability on the email task. This resulted in a score of 0-5 given by a single rater.• The summed score between two raters was used to give a score ranging from 0-10.
METHODS• The Spearman rho is used to measure and inter-rater reliability for RQ1.• A Spearman rho is used to measure the internal consistency between the email task, integrated task and independent task for RQ 2.
METHODSEmail task rubric scoring criteria developed through the empiricalmethod covers the following features:• Language use• Grammatical and lexical features• Register awareness, including appropriate forms of address• Genre markers specific to emails• Topical relevance• Task completion
TEST TASK CHARACTERISTICSTask PromptDdirections: Read the question below. Plan, write, and revise an email.Use the space below to prepare writing your email. You may begin now.Question: You are new at XXX University. You do not know whatclasses to take. Write an email to Professor Smith to do the following:1) introduce yourself2) explain your problem3) ask for advice
RESULTS: RQ1Correlation Coefficient for Inter-rater reliability Ratings Correlation Coefficient N Email Rating 1 -- 103 Email Rating 2 .92 103Note. CI = confidence interval; LL = lower limit; UL = upper limit Descriptive Statistics 95% CI Ratings M SD N LL UL Email Rating 1 3.44 1.13 103 3.22 3.66 Email Rating 2 3.47 1.14 103 3.24 3.69Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;
RESULTS: RQ2 95% CI Writing Task M SD N LL UL Email 6.90 2.22 103 6.46 7.33 Independent 4.91 2.02 103 4.51 5.30 Integrated 4.66 1.87 103 4.23 5.03Note. CI = confidence interval; LL = lower limit; UL = upper limit Correlations Between Writing Tasks Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;
DISCUSSION• Students had higher mean scores on the email task than on the other two task types.• What are the limitations for implementing the new task? – The test scores on the email task did not distinguish students by writing ability. – The task appears to be lacking complexity or the scale made it too easy to get a high score.
IMPLICATIONS FOR ONGOING TEST TASK DEVELOPMENT• The email task could be improved by adding complexity to the task design. – Give more input for learners to respond to • Ex: a sample email from a professor to which they must respond to – involve a higher imposition request to the professor
IMPLICATIONS FOR ONGOING TEST TASK DEVELOPMENT• The scale must be revised to better distinguish criteria expected for different bands of rubric. – A sample of emails to faculty members could be gathered to identify pragmatic features lacking in current task design. – We could take out any criteria in rubric that does not apply to pragmatic features.
REVISED TASK• Writing Task 1: Email a Professor (10 minutes total)• Planning (2 minutes): Read the situation below. Plan and write an email. Use the space below to prepare writing your email.• Situation: You are a student at Northern Arizona University. You have to turn in an essay in five days, but your writing has many problems. Write an email to Professor Smith and do the following: – fill in the subject line – introduce yourself – explain your problem – request an appointment to get help – ask for a reply