This document discusses the essential roles of assessment, measurement, and evaluation in task-based language teaching. It begins by outlining some of the key issues and debates around TBLT, including whether it is effective and for whom. It then discusses the need for empirical measurement and assessment that is aligned with educational purposes to better understand how TBLT works and how to enable task-based learning. Problems of practice are identified, such as lack of construct definition and standardization. The role of evaluation in articulating empirical practice to improve TBLT is also discussed.

1. John M. Norris University of Hawai´i Task-Based Language Teaching 2005

2. This Powerpoint presentation is copyrighted by John M. Norris © September 23, 2005 Citation information: Norris, J. M. (2005, September). The essential roles of assessment, measurement, and evaluation in task-based language teaching . Plenary address presented at the First International Conference on Task-Based Language Teaching, University of Leuven, Belgium (September 23, 2005).

3. 0:00 9:00 3:00 6:00 12:00 21:00 15:00 18:00 What time is it in Leuven ? What time is it in Hawaii ? “ Here and Now” condition “ There and… ” condition Now

5. measurement, The essential roles of assessment, and evaluation in task-based language teaching

6. TBLT 2005: What’s at stake? ??? Language Education ??? Does it work? for whom, when, why? What’s it worth? to whom, when, why? How does it work? for whom, when, why? Can it be improved? by whom, when, why?

7. TBLT 2005: What’s at stake? … another L2 teaching method bandwagon… The critics say… … cognitive-interactionist SLA hegemony… … won’t work with _____ learners… … not possible in _____ program settings… … too much task, not enough language… … not happening in practice… T B L T

8. Setting the record straight: TBLT at work TARGET LANGUAGES English Dutch French Czech Chinese Arabic Ukranian Japanese Russian Spanish Korean German

9. Setting the record straight: TBLT at work PROGRAM TYPES Second Language Foreign Language Vocational Education General Education Local Innovation Small Classes National Ed. Policy Special Purposes Adult Child University Requirement Large Classes Beginning Learners Advanced Learners Face-to- Face Online

10. Setting the record straight: TBLT foundations “ All genuine education terminates in discipline, but it proceeds by engaging the mind in activities worthwhile for their own sake.” Dewey (1933), pp. 86-87

11. Setting the record straight: TBLT foundations “ This discussion rejects the doctrine that students should first learn passively, and then, having learned, should apply knowledge. […] In fact, the applications are part of the knowledge. For the very meaning of things known is wrapped up in their relationships beyond themselves. Thus, unapplied knowledge is knowledge shorn of its meaning.” Whitehead, 1947, pp. 218-219

12. Setting the record straight: TBLT foundations “ For starters, this means having students do tasks, or at least meaningful simulations, that experts do in the various disciplines. Second, it means teaching them to think in ways that experts do when they perform these tasks.” Sternberg (2003), p. 5

13. Setting the record straight: Other task-based domains Medical school Nursing Curriculum by design Successful intelligence Environmental studies Social work Task-Based Practice Music education

14. Setting the record straight: So what is TBLT? Task-Based Language Teaching Rationales and Principles Education Philosophy Psychology SLA Language teaching Sociopolitical values L2 Education Programs Learner needs Assessment Curriculum Instruction Teacher development Evaluation Materials

15. The potential of task-based practice: One example William Reid Reid. W. J. (1992). Task strategies: An empirical approach to social work practice . New York: Columbia University Press. Social work problems : truancy, aging, delinquency, substance abuse, family health

16. Task-Centered Practice Empirical Justification Empirical Implementation Theory Learning, behavior, cognition Research Psychosocial change, Problem solving Evaluation Case work practice, Intervention outcomes Needs Identify, agree, prioritize Action Plan, rehearse, attempt, feedback Assessment Task accomplishment, Problem change Development Practitioner-friendly methods Improvement Field-tested findings The potential of task-based practice: One example

17. The potential of task-based practice: One example “ Assessment, process, and outcome data are systematically collected in each case by means of structured recording guidelines. Continuing research is used to improve the model.” (Reid, 1992, p. 132). Development Improvement Field-testing

18. “ [Reid] was the single most significant figure in social work research” Shaw (2004) p. 113 The potential of task-based practice: One example

19. Task-Based Language Teaching: A(nother) method for language teaching. The potential of task-based practice: TBLT

20. Task-Based Language Teaching: A(nother) method for language education practice. The potential of task-based practice: TBLT

21. Task-Based Language Teaching: An empirical approach to language education practice. The potential of task-based practice: TBLT Assessment Measurement Evaluation

22. The Role of Measurement in TBLT Interpreting research constructs… … to understand task-based learning

23. Measurement in TBLT: Responding to research purposes What is measurement? … trustworthy definition, elicitation, observation, analysis, and interpretation of research constructs…

25. Measurement in TBLT: Responding to research purposes “ Perhaps the best recommendation would be that, with research in this area still developing techniques and understandings, the role of A/L work will be to offer suggestions for relevant variables and the standardization of operationalizations of variables…” Tavakoli & Skehan, 2005, p. 273

28. Task influence on L2 performance (adapted from: Skehan, 2001) Measurement in TBLT: Problems of practice 0 + 0 Transformations (planning) 0 + 0 Outcomes + 0 + Structure - + + Dialogic v. Monologic + 0 + Familiarity Fluency Complexity Accuracy Task characteristic

29. Task influence on L2 performance (adapted from: Skehan, 2001) Measurement in TBLT: Problems of practice 0 + 0 Transformations (planning) 0 + 0 Outcomes + 0 + Structure - + + Dialogic v. Monologic + 0 + Familiarity Fluency Complexity Accuracy Task characteristic

30. “… giving learners the opportunity to plan before a task is done consistently produces greater complexity of language…” (Skehan, 2003, pp. 395-96) Measurement in TBLT: Problems of practice “ To sum up, both modality and group condition affected complexity at the level of syntax…” (Ellis & Yuan, 2005, p. 184)

31. Measurement in TBLT: Problems of practice Task condition Planning L2 performance outcome Complexification Modality? writing/speaking Complexity? Subordination Complexity? Other measures Scope of measure? duration/timing Planning? Focus, structure, pressure Learner proficiency? “ L-H”, “N-I-A”, “450-600” Task features? Type, difficulty, form, mono-dialogic, interlocutors ?

32. Measurement in TBLT: Problems of practice Task condition + strategic planning L2 performance outcome Complexification Complexity? Clauses per C-Unit Learner proficiency? TOEFL: 427-670 Elder & Iwashita (2005) X X X

33. Measurement in TBLT: Problems of practice Task condition + strategic planning L2 performance outcome Complexification Complexity? Words/T-Unit Clauses per T-Unit Total Subordination Learner proficiency? TOEFL: 420-480 TOEFL: 510-580 TOEFL: 560-610 Kawauichi (2005)

34. Measurement in TBLT: An empirical approach to practice In what ways are what kinds of planning associated with what kinds of L2 performance for what kinds of L2 users under what conditions…and why?

35. Measurement in TBLT: An empirical approach to practice “ ... descriptive longitudinal studies which establish norms of performance for particular processes and phenomena in L2 acquisition…” Norris & Ortega, 2003, p. 738 “… attempting to ‘measure’ acquisition without a sound descriptive basis for meaningful differences in particular acquisition-related behaviors would be akin to timing a runner’s performance over a mile without knowing how many times around the track a mile happens to be…”

36. Measurement in TBLT: An empirical approach to practice Georgetown University German Department: “ Syntactic complexity development in task-based writing” Longitudinal (23) + Cross-sectional (329) Extended and extemporaneous writing tasks Multiple measures, including Multiple measures of SC End-of-curricular levels Across 4 years

37. Syntactic complexity in curriculum-embedded writing assessment: LONGITUDINAL Global compexification   Phrasal elaboration Subordination 

38. Syntactic complexity in curriculum-embedded writing assessment: CROSS-SECTIONAL Global compexification  Subordination   Phrasal elaboration

41. The Role of Assessment in TBLT Informing decisions and actions… … to enable task-based learning

42. Assessment in TBLT: Responding to educative purposes What is assessment? … the use of tests and other procedures for gathering trustworthy information about learners and making decisions and taking actions on the basis of that information, in support of teaching and learning…

44. Assessment in TBLT: Responding to educative purposes “ In other words, by focusing on task-based assessment—here writing—faculty faced their own assumptions about long-term development in instructed L2 learning and appropriate learning outcomes at particular stages in that sequence.” Byrnes (2002), p. 428

45. Assessment in TBLT: Responding to educative purposes “… the quality of task completion gave teachers a good understanding of what students did and did not know—often, a better understanding than most tests could provide.” Leaver & Kaplan (2004), p. 52

46. Assessment in TBLT: Responding to educative purposes “ Focusing on what one can do instead of the competencies one has yields a more direct link with language performance that can be observed in the real world.” Gysen & van Avermaet (2005), p. 51

49. Assessment in TBLT: Problems of practice “… the inferences we want to make are about underlying ‘language ability’ or ‘capacity for language use’ or ‘ability for use’…” Bachman (2002), p. 454 “… assessments can have many different functions. What is appropriate for one assessment purpose may be inappropriate for another…” Council of Europe (2001), p. 180

51. Assessment in TBLT: Problems of practice “… is not for the test designer to decide or prescribe. Rather, it should be the test users and stakeholders who inform exactly what needs to be done in the name of language assessment, and it is up to us to enable them to do so.” Norris (2002), p. 344

52. “ 1. The intended use(s) for task-based assessment within the language programme must be specified…” Long & Norris (2000), p. 600 Assessment in TBLT: An empirical approach to practice

53. Specifying intended assessment use From: Norris (2000)

55. Assessment in TBLT: Examples in practice, Specification products Intended assessment use specification Assessment policies Use specs for all tests Priorities for Development Assessment Constraints Quizzes Exams Performance Internal External Time Money Ethics Training Unknowns (Urgent!) Placement Learning outcomes External Accountability Developing Implementing Articulating Communicating Evaluating

57. Task Content L conventions L focus Assignment 1 Assignment 2 Process writing 4-6 tasks per semester Assignment 3 Assignment 4 Prototypical Performance Writing Task: Level I Level II Level III Level IV Curricular level/genre learning expectations Feedback loop Explicit criteria Assessment in TBLT: Examples in practice, Instruments

60. The Role of Evaluation in TBLT Articulating empirical practice… … to improve task-based education … with practical value

61. Evaluation in TBLT: Responding to program purposes What is evaluation? … gathering trustworthy information about program elements in order to understand, demonstrate, and improve program value…

63. Evaluation in TBLT: Responding to program purposes “ Given the checkered history of prescriptions for language teaching, the likelihood that all 10 TBLT MPs will turn out to have been well founded is minimal. Thus, designers … who draw upon these or other sources should build careful evaluation components into their delivery systems.” Doughty & Long (2003), p. 68 “ In contrast to the extensive research into tasks…there have been no more than a handful of evaluations of task-based teaching” Ellis (2003), p. 323

64. Evaluation in TBLT: Problems of practice Nunan (1989) Crookes & Gass (1993) Willis (1996) Skehan (1998) Lee (2000) Bygate, Skehan, & Swain (2001) Ellis (2003) Leaver & Willis (2004) Nunan (2005) Overview books on TBLT… … with chapters on program evaluation… Ellis (2003)

66. Evaluation in TBLT: An empirical approach to practice Patton (1996) Utilization-focused Evaluation Intended uses Intended users Questions + Methods Understand Improve Educate Demonstrate worth Hold accountable Test theory Empower Teachers Administrators Curriculum writers Learners Parents/public Researchers Funders Values clarification Implementation Process-product Experimentation Synthesis

67. Evaluation in TBLT: Examples in practice Norris (2004) Evaluation of a US college German FL assessment program Van den Branden (to appear) Evaluation of task-based teacher development in Flanders

68. Assessment, measurement, and Evaluation in TBLT: Essential roles TBLT in situ Measurement Theory, Research Assessment Imple-mentation Evaluation Society, Value EMPIRICAL PRACTICE

69. http://www2.hawaii.edu/~jnorris [email_address] See link at: www.tblt.org For more information: Contact me

70. See you in Hawaii for TBLT 2007!

71. http://www2.hawaii.edu/~jnorris [email_address] See link at: www.tblt.org

72. REFERENCES Please see the following web site for a complete pdf file of references and resources from this presentation: http://www2.hawaii.edu/~jnorris/TBLT%20references.pdf