Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
An analysis of Japanese EFL learners’ reading-
to-write task completion process:
Triangulation of stimulated recall and
ke...
Acknowledgement
• This work was supported by Grant-in-Aid for
Scientific Research 16K02983.
2
Introduction
• Reading-to-write tasks (e.g., summarization) are
important task types in conducting academic work
in Englis...
Previous Studies on L2 writing
task completion process
• Process features (Baaijen, Galbraith, & de Glopper,
2012; Barkaou...
Research Question
• What task completion process do EFL learners in
Japan engage to respond to a summary task?
Specific fo...
Method (1)
• Participants
• Five undergraduate students majoring in English
language and literature at a private universit...
Method (2): Source texts
A. The War over
Antiquities
(3rd admin. 2014)
B. The Automated
Future
(1st admin. 2014)
Structure...
Method (3)
• Procedure
• Each learner summarized one text in English (Approx. 80
words).
• Random assignment of participan...
Method (4)
Video recording
Two observers took notes on
the observation sheet.
9
Test-taker
Online dictionary (Weblio)
Reco...
Method (5)
• Scoring
• Double scoring on all rating scales by two trained raters
• Rating scales
• Integration (1-4 scale)...
Method (6)
Scoring results summary
Student Text
Total
Number of
words
Integration
(1-4)
Language
(1-4)
String match
(3+ wo...
Method (7)
• WritingMaetriX (Kusanagi, Abe, Fukuta, &
Kawaguchi, 2014)
• A keystroke-logging program that can record, anal...
Results: Overview
• Time required for task completion: approx. 32– 40 min.
• Task completion time divided into two phases ...
Phase 1: Annotations
• Students used various kinds of annotations while
reading the source text.
• Underline: I underlined...
Phase 1: Planning
• Students had different styles of planning
• Student 1, 2, 3: developed the structure of
summary only i...
Phase 2: Descriptive Statistics of learners’
keystroke logging
Student1 student2 Student3 student4 Student5
Task A B B B A...
Phase 2: Time Series Data of Learners'
Writing Process
Revision
First data input
Planning
17
Phase 2: Verbal protocol data
• First data input
• First of all, I wrote industrial, industrial revolution in the
introduc...
Phase 2: Planning and revision
processes
• Planning
• Comprehensive & local planning depending on the situation
(similar t...
Phase 2: Planning example
Stimulated Recall
• I may be thinking about
the paraphrase of “ship”
in this pause. I used the
w...
Phase 2: Revision processes
Stimulated Recall
• I noticed the number of
characters is not
enough and had to
summarize the ...
Phase 2: Reading-writing integration
• Writerly reading (Hirvela, 2004): reading with
consciousness of writing summary
• I...
Phase 2: Reading-writing integration
• Mining: (Hirvela, 2004; Plakans, 2009): detecting
information for text while writin...
Discussion and Conclusion
• Annotation and planning behavior while reading differs as well as how
to use the information h...
References (1/2)
• Baaijen, V. M., Galbraith, D., & de Glopper, K. (2012). Keystroke analysis: Reflections on procedures a...
References (2/2)
• Lindgren, E., & Sullivan, K. P. (2003). Stimulated recall as a trigger for increasing noticing and lang...
Upcoming SlideShare
Loading in …5
×

An analysis of Japanese EFL learners’ reading-to-write task completion process: Triangulation of stimulated recall and keystroke logging data sources.

1,882 views

Published on

Ishii, Y., Sawaki, Y., & Tahara, T. (2017). An analysis of Japanese EFL learners’ reading-to-write task completion process: Triangulation of stimulated recall and keystroke logging data sources. The 21st Annual Conference of the Japan Language Testing Association. University of Aizu.

Published in: Education
  • Be the first to comment

An analysis of Japanese EFL learners’ reading-to-write task completion process: Triangulation of stimulated recall and keystroke logging data sources.

  1. 1. An analysis of Japanese EFL learners’ reading- to-write task completion process: Triangulation of stimulated recall and keystroke logging data sources Yutaka Ishii, Yasuyo Sawaki, & Tatsuro Tahara Waseda University yutakaishii@aoni.waseda.jp The 21st Annual Conference of the Japan Language Testing Association University of Aizu 1
  2. 2. Acknowledgement • This work was supported by Grant-in-Aid for Scientific Research 16K02983. 2
  3. 3. Introduction • Reading-to-write tasks (e.g., summarization) are important task types in conducting academic work in English (Rosenfeld, Leung & Oltman, 2001). • In this Kaken project we examine characteristics of Japanese university students’ performance on summarization tasks and develop materials for teaching summarization in academic writing courses. • This presentation focuses on the process learners engage while completing a summary task • Stimulated recall and keystroke logging are combined for a fine-grained analysis of learners’ writing process from multiple perspectives. 3
  4. 4. Previous Studies on L2 writing task completion process • Process features (Baaijen, Galbraith, & de Glopper, 2012; Barkaoui, 2016; Zhang, & Deane, 2015) • Relationship between English language proficiency and keyboarding skills (Barkaoui, 2015) • Stimulated recall (Lindgren, & Sullivan, 2003) • Time allocation of composing time (De Larios, Manchon, & Murphy, 2006) • Integrated writing task process (Barkaoui, 2015; Chan, 2017) 4
  5. 5. Research Question • What task completion process do EFL learners in Japan engage to respond to a summary task? Specific foci: • Reading the source text • Writing the response • Reading-writing integration 5
  6. 6. Method (1) • Participants • Five undergraduate students majoring in English language and literature at a private university in Tokyo • All participants enrolled in a required academic writing course for second-year students • Materials • Two summarization tasks (each based on a single source text in English) • Source texts were adopted from published Eiken test forms for Grade Pre-1 (Eiken Foundation of Japan) 6
  7. 7. Method (2): Source texts A. The War over Antiquities (3rd admin. 2014) B. The Automated Future (1st admin. 2014) Structure Argumentative Length (words) 508 504 paragraphs/ sentences 4/23 4/24 Vocabulary Level 6000 5000 Readability (F-K Grade Level) 12.4 12.4 Summary length Approx. 80 words 7
  8. 8. Method (3) • Procedure • Each learner summarized one text in English (Approx. 80 words). • Random assignment of participants to texts • 40-minutes per task; dictionary use allowed • The learner’s task completion process videotaped • Keystroke logged via Writing MaetriX • An observation sheet comprising six categories (Read text, Annotate, Plan, Write, Review, & Revise) completed by observers • Upon completion of the summary task, a stimulated recall session was conducted in Japanese. • Prompts: • Video of the task completion process • Replay of Keystroke logging data • Completed summary response • Test booklet with the participant’s notes (annotations and outlines) 8
  9. 9. Method (4) Video recording Two observers took notes on the observation sheet. 9 Test-taker Online dictionary (Weblio) Recording of keystroke logging
  10. 10. Method (5) • Scoring • Double scoring on all rating scales by two trained raters • Rating scales • Integration (1-4 scale): The degree of succinct representation of source-text content by employing appropriate Macrorules (e.g., Brown & Day, 1983; van Dijk & Kintsch, 1983) for integrating information across text (based on Sawaki, 2003; Hijikata-Someya, Ono & Yamanishi, 2015) • Language (1-4 scale): The degree of grammatical control, syntactic variety, and appropriate word choice (adapted from Hijikata-Someya et al., 2015) 10
  11. 11. Method (6) Scoring results summary Student Text Total Number of words Integration (1-4) Language (1-4) String match (3+ words) 1 A 84 2.5 2.0 4 2 B 75 2.5 3.0 0 3 B 80 3.5 3.0 0 4 B 75 2.5 2.0 1 5 A 94 2.0 2.0 0 11
  12. 12. Method (7) • WritingMaetriX (Kusanagi, Abe, Fukuta, & Kawaguchi, 2014) • A keystroke-logging program that can record, analyze, and replay learners’ writing process 12
  13. 13. Results: Overview • Time required for task completion: approx. 32– 40 min. • Task completion time divided into two phases for qualitative analyses: • Phase 1: Reading (text reading, annotating, and planning) • Phase 2: Writing (writing, revising, and reviewing) Task completion process Phase 1 Phase 2 (6-22 min.) (18-33 min.) 13
  14. 14. Phase 1: Annotations • Students used various kinds of annotations while reading the source text. • Underline: I underlined when I found a concrete example (S5: 14) • Mark: I have a habit of writing a reverse triangle when looking at "however" (S4: 48) • parenthesis: I just wrote a parenthesis to highlight the that-clause following “argue" (S1; 9) • Others (slash, notes on blank spaces, etc...) See next slide for examples 14
  15. 15. Phase 1: Planning • Students had different styles of planning • Student 1, 2, 3: developed the structure of summary only in their minds (without writing) • Student 4, 5: took notes on question paper • I thought about the structure while looking at the (question) paper first, before I started writing. (S5: 29) • This is an introduction, and I picked out and took notes of the skeleton only, so that I could use them (for summary) (S4: 80) 15
  16. 16. Phase 2: Descriptive Statistics of learners’ keystroke logging Student1 student2 Student3 student4 Student5 Task A B B B A First data input (sec) 447.0 632.2 721.0 1390.6 893.9 Key input (minute) 28.4 42.0 50.1 34.5 77.3 # Words (minute) 3.72 2.30 2.35 1.90 2.65 Revision (minute) 0.97 1.78 2.61 2.43 4.28 revision 0/7/22 0/24/53 0/45/89 0/16/96 0/79/151 16
  17. 17. Phase 2: Time Series Data of Learners' Writing Process Revision First data input Planning 17
  18. 18. Phase 2: Verbal protocol data • First data input • First of all, I wrote industrial, industrial revolution in the introduction session. (S4: 160) • Planning • After I finished writing about the second paragraph, I was about to write the third paragraph. Then, I was thinking about how to summarize this paragraph. (S5: 76) • Revision • From here, I tried to reduce the word such as UNESCO. (S5: 123) 18
  19. 19. Phase 2: Planning and revision processes • Planning • Comprehensive & local planning depending on the situation (similar to independent writing) • Re-reading and paraphrasing the source text • Revision • Two types of revisions: precontextual and contextual (similar to independent writing). • Another precontextual revision “paraphrase” was frequently observed compared to independent writing task. • Planning & revision: Focus on reducing the number of words frequently reported (meeting the length limit = unique to source-based writing tasks) 19
  20. 20. Phase 2: Planning example Stimulated Recall • I may be thinking about the paraphrase of “ship” in this pause. I used the word “transfer” to report that the antiques finally moved to UK. However, because the meaning was temporailiy obscure, I used weblio at this time. (S1: 49) Keystroke Logging • In recent years, illegally imported antiques have been returned to their hometown. This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon in Athens, Greek. It is originally owned by Greece, but it was transfered to Britain for preservation* 20
  21. 21. Phase 2: Revision processes Stimulated Recall • I noticed the number of characters is not enough and had to summarize the content. So, I was worried about how to edit the content. (S4: 187) Keystroke Logging • Industrial Revolution increased the productivity of that age while it also made difficulties to the labors, which thier working was replaced by machine. However, in this age, information-technology brings new reformation to the way of people. Hanson claims that (considers this reformation possitively,) 21
  22. 22. Phase 2: Reading-writing integration • Writerly reading (Hirvela, 2004): reading with consciousness of writing summary • I underlined a concrete example that I thought was easy to use (for summary) (S3: 39) • I found this the most critical in the paragraph, and this explanation continued, so I thought I need to write it (in my summary) and underlined that part (S5: 17) The parts mentioned above were actually reflected in these students’ summaries. 22
  23. 23. Phase 2: Reading-writing integration • Mining: (Hirvela, 2004; Plakans, 2009): detecting information for text while writing • e.g.) Considering paraphrase, Adding/deleting examples in summary, Comparing text with summary • I underlined the abstract points while reading, (...) While writing a summary, I found it bothersome to look for (these points), I underlined them so that I could go back to the points I used for summary (S3: 144). • I checked if my summary made sense and if there were no points I overlooked (S2: 91) 23
  24. 24. Discussion and Conclusion • Annotation and planning behavior while reading differs as well as how to use the information highlighted in their summaries • The result suggests an important role of reading in integrated tasks, shown as previous studies (e.g. Plakans & Gebril, 2012; Sawaki , Quinlan, & Lee, 2013). • (The most important skill is) reading. Without deep understanding, I could not write or organize (my summary). (S3:197) • Some revision behaviors observed while completing independent tasks were also observed in this study, while other unique ones were also identified. • Revision behavior differs between ESL and EFL learners • Barkaoui (2016) stated that learners revised their writing most frequently in the middle segment of their writing process, while completing an integrated writing task. • In contrast, learners in this study revised their writing in the final segment. • Limitations of this study • Small sample size • Stimulated recall 24
  25. 25. References (1/2) • Baaijen, V. M., Galbraith, D., & de Glopper, K. (2012). Keystroke analysis: Reflections on procedures and measures. Written Communication, 29(3), 246-277. • Barkaoui, K. (2016). What and When Second‐Language Learners Revise When Responding to Timed Writing Tasks on the Computer: The Roles of Task Type, Second Language Proficiency, and Keyboarding Skills. The Modern Language Journal, 100(1), 320-340. • Barkaoui, K. (2015). Test Takers' Writing Activities During the TOEFL iBT® Writing Tasks: A Stimulated Recall Study. ETS Research Report Series, 2015(1), 1-42. • Brown, A. L., & Day, J. D. (1983). Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning and Verbal Behavior 22, 1-14. • Chan, S. (2017). Using keystroke logging to understand writers’ processes on a reading-into-writing test. Language Testing in Asia, 7(10), 1-23. • De Larios, J. R., Manchon, R. M., & Murphy, L. (2006). Generating text in native and foreign language writing: A temporal analysis of problemsolving formulation processes. The Modern Language Journal, 90(1), 100-114. • Hijikata-Someya, Y., Ono, M., & Yamanishi, H. (2015). Evaluation y native and non-native English teacher-raters of Japanese students’ summaries. English Language Teaching, 8(7), 1-12. • Hirvela, A. (2004). Connecting reading and writing in second language writing instruction. Ann Arbor, MI: The University of Michigan Press. • Kusanagi, K., Abe, D, Fukuta, J, & Kawaguchi, Y. (2013). Visualizing writing process using a key-logging system: For construct feedback to enhance autonomous learning. Paper presented at the 81st Spring Conference of the Chubu Chapter, Japan Association for Language Education and Technology (LET), Tokai Gakuen University, Japan. 25
  26. 26. References (2/2) • Lindgren, E., & Sullivan, K. P. (2003). Stimulated recall as a trigger for increasing noticing and language awareness in the L2 writing classroom: A case study of two young female writers. Language Awareness, 12(3-4), 172-186. • Plakans, L. (2015). Integrated Second Language Writing Assessment: Why? What? How?. Language and Linguistics Compass, 9(4), 159-167. • Plakans, L. (2009). The role of reading strategies in integrated L2 writing tasks, Journal of English for Academic Purposes, 8(4), 252-266. • Plakans, L., & Gebril, A. (2012). A close investigation into source use in integrated second language writing tasks. Assessing Writing, 17(1), 18-34. • Rosenfeld, M., Leung, S., & Oltman, P. K. (2001). The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels. Educational Testing Service. • Sawaki, Y., Quinlan, T., & Lee, Y. W. (2013). Understanding learner strengths and weaknesses: Assessing performance on an integrated writing task. Language Assessment Quarterly, 10(1), 73-95. • Sawaki, Y. (2003). A comparison of summarization and free recall as reading comprehension tasks in web-based assessment of Japanese as a foreign language (Unpublished PhD thesis). University of California, Los Angeles (UCLA). • Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press. • Zhang, M., & Deane, P. (2015). Process features in writing: Internal structure and incremental value over product features. ETS Research Report Series, 2015(2), 1-12. • Obunsha (Ed.) (2016). 2016 nendo eiken jun-1 kyu kako 6 kai zenmondaishuu (Test papers for the last six administrations of the Eiken Grade Pre-1 test, 2016). Tokyo: Obunsha. 26

×