Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Designing personalised, automated feedback to develop students' research writing skills

237 views

Published on

Presentation at ASCILITE 2018

Published in: Education
  • Be the first to comment

  • Be the first to like this

Designing personalised, automated feedback to develop students' research writing skills

  1. 1. Designing personalised, automated feedback to develop students’ research writing skills Sophie Abel, Dr. Kirsty Kitto, Dr. Simon Knight & Dr. Simon Buckingham Shum
  2. 2. 2 How do Higher Degree Research (HDR) students learn research writing?
  3. 3. 3 • Writing groups • Writing workshops • Writing retreats • Individual consultations with language experts Students learn research writing through….
  4. 4. 4 Mainly Through Supervisory Feedback
  5. 5. 5 However research shows that…… “…supervisor feedback is often ambiguous, enigmatic, and coded – that is saturated with meaning, but difficult to understand” (Paré, 2010, p.107). “Feedback was not timely enough to be of any use, for example they kept drafts for two months, and feedback was not constructive, just typos circled” (Aitchison et al., 2012, p. 442). “At one meeting I burst into tears and cried for two hours while [my supervisor] went through and said ‘That’s not a sentence’ ‘That’s not a sentence’” (Aitchison et al., 2012, p. 442). “…[supervisor] feedback is not helpful. It is mainly about grammar and sentence structure. Not about how to write in an expanded form, or how to write social research” (Aitchison et al., 2012, p. 445).
  6. 6. 6 Another problem…Writing isn’t linear The Writing Process Approach (Coffin & Hewings, 2003)
  7. 7. 7 So how do we address the challenges of… Constructive feedback Timeliness “Feedback was not timely enough to be of any use, for example they kept drafts for two months and feedback was not constructive, just typos circled.” “At one meeting I burst into tears and cried for two hours while [my supervisor] went through and said ‘That’s not a sentence’ ‘That’s not a sentence’”
  8. 8. 8 So how do we address the challenges of…. Clear, understandable feedback “…supervisor feedback is often ambiguous, enigmatic, and coded – that is saturated with meaning, but difficult to understand” (Paré, 2010, p.107). Holistic feedback vs. just sentence structure and grammar “…[supervisor] feedback is not helpful. It is mainly about grammar and sentence structure. Not about how to write in an expanded form, or how to write social research”
  9. 9. 9 Potential Approach - Writing Analytics What is Writing Analytics? “Writing Analytics is a developing sub-domain of learning analytics with a specific focus on supporting writing practices (Knight et al., 2017)”. Writing Analytics uses Natural Language Processing (NLP) to analyse text and then provide automated feedback about the text.
  10. 10. 10 Some examples include… Grammarly
  11. 11. 11 Turnitin Revision Assistant Some examples include…
  12. 12. 12 Some examples include… Research Writing Tutor
  13. 13. 13 AcaWriter AcaWriter is an web-based tool that provides formative feedback on rhetorical moves in writing. A rhetorical move is a string of text that conveys a communicative function (Swales & Feak, 2012). Developed by the UTS Connected Intelligence Centre ATN project – check out: http://heta.io/technologies/ Demo: http://acawriter demo.utscic.edu.au/
  14. 14. 14 AcaWriter’s Analytical Tags / Moves Emphasis Communicative function: Emphasis on significant, important ideas Stormwater pollution is a significant environmental issue that arises from intense urbanisation. Background Communicative function: Consensus and background knowledge Recent studies indicate that the Earth’s climate is changing rapidly.
  15. 15. 15 Contrast Communicative function: Contrast, disagreement, tension, inconsistency This approach fails to address the issue of bullying outside the classroom and schools. Question Communicative function: Raising a question or missing knowledge Little research exists on how automated feedback impacts students’ writing. AcaWriter’s Analytical Tags / Moves
  16. 16. 16 Novelty Communicative function: Novelty, improvement This framework provides a new approach to tackling childhood obesity. Surprise Communicative function: An unexpected outcome These results are unexpected considering the device was only a prototype. AcaWriter’s Analytical Tags / Moves
  17. 17. 17 Trend Communicative function: A trend, growth, pattern or tendency Futsal was introduced in 1930 and its popularity is growing worldwide. Summary Communicative function: Author’s goals, contribution or conclusion We show how the constructs included in the model link to relevant research. AcaWriter’s Analytical Tags / Moves
  18. 18. 18 AcaWriter’s Analytical (Standard) Parser
  19. 19. 19 Effective Feedback Practices Hattie and Timperley’s (2007) Feedback Model: How am I going? Where am I going? Where to next?
  20. 20. 20 Genre based approach A genre based approach to teaching writing looks at how language is structured in texts to achieve a communicative purpose in particular contexts (Swales, 1990), and involves “being explicit about the way language works to make meaning” (Cope & Kalantzis, 1993, p.1). Focusing on Research Article Introductions and Abstracts
  21. 21. 21 Create A Research Space (CARS) Model (Swales, 1990)
  22. 22. 22 Create A Research Space (CARS) Move 1 – Establishing a research territory Step 1 - Claiming centrality and/or Step 2 - Making topic generalisations(s) and/or Step 3 - Reviewing items of previous research Move 2 – Establishing a niche Step 1A - Counter-claiming or Step 1B - Indicating a gap or Step 1C - Question-raising or Step 1D - Continuing a tradition
  23. 23. 23 Move 3 – Occupying the niche Step 1A – Outlining purposes or Step 1B – Announcing present research or Step 1C – Announcing principal findings or Step 1D – Indicating research article structure Create A Research Space (CARS) (Swales, 1990)
  24. 24. 24 Mapping CARS in AcaWriter Create A Research Space (CARS) (Swales, 1990)
  25. 25. 25 Mapping CARS in AcaWriter Move 1 - Establishing a research territory a. by showing that the general research area is important, central, interesting, problematic, or relevant in some way (optional) b. by introducing and reviewing items of previous research in the area (obligatory) Background Emphasis It is now widely accepted that timely, actionable feedback is essential for effective learning.
  26. 26. 26 Mapping CARS in AcaWriter Move 2 – Establishing a niche: a. by indicating a gap in the previous research, raising a question about it, or extending previous knowledge in some way (obligatory) Contrast Question From a human-centred computing perspective, the end-user’s interpretation of these visualisations is a critical challenge to design for, with empirical evidence already showing that ‘usable’ visualisations are not necessarily effective from a learning perspective.
  27. 27. 27 Mapping CARS in AcaWriter Move 3 - Occupying the niche: a. by outlining purposes or stating the nature of the present research(obligatory) b. by listing research questions and hypotheses (optional) c. by announcing principle findings (optional) d. by stating the value of the present research (optional) e. by indicating the structure of the research paper / thesis (optional) Novelty Summary Summary We present a pilot study that explores the effectiveness of these elements based on educators’ responses to paper prototypes.
  28. 28. 28 Embedding CARS in AcaWriter
  29. 29. 29 Developing Feedback
  30. 30. 30 Developing Feedback
  31. 31. 31 Embedding AcaWriter in HDR Workshop
  32. 32. 32 Positive responses from students
  33. 33. 33 Room for improvement
  34. 34. 34 Next steps… Design: • building an online version of the workshops • building additional modules/feedback for other sections of research writing Implementation: • aligning AcaWriter with Learning Design • investigating how students use the tool outside workshops Evaluation: • how do we measure student impact and learning Watch this space….
  35. 35. Thank you Questions? Don’t forget to check out: • ATN project – different learning contexts : http://heta.io/technologies/ • Demo AcaWriter: http://acawriter demo.utscic.edu.au/
  36. 36. 36 How Rhetorical Parsing Works….(over simplified) Uses a Rhetorical parser called ‘Athanor’ developed by Agnes Sandor from Xerox (now NaverLabs Europe). Athanor identifies rhetorically salient sentences through patterns of concepts - Rules based Natural language processing system.
  37. 37. 37 Questions… • Accuracy: Rules based approach, grounded in literature, lower recall, but higher precision. Machine learning approach has higher recall, but lower precision. • How does it work: It is based on a concept matching framework - where expressions that both convey contextual concepts and are grammatically dependent are classified and tagged as rhetorical moves. E.g: Recent studies indicate that the Earth's climate is changing rapidly. AcaWriter identifies the contextual concept in the words Recent studies as ‘Background Knowledge’ • Can not assess quality of writing

×