This document summarizes a workshop on reflective writing analytics and linking it to learning design. It discusses how the tool AcaWriter provides automatic feedback on reflective writing elements and depth of reflection. It then discusses how AcaWriter was used in business school contexts through co-design of rubrics, prompts and feedback templates to align the tool with learning goals. The workshop involved demonstrations of AcaWriter, discussions on challenges with teaching reflection and how analytics could help, and ideas on redesigning reflection tasks building on writing analytics.

1. UTS CRICOS 00099F
The Fifth Writing Analytics Workshop: Linking
Reflective Writing Analytics to Learning Design
Ming Liu1, Sumati Ahuja2, Rosalie Goldsmith3,Xiaodi Huang4
1. Connected Intelligence Centre, University of Technology Sydney, Australia
2 Business School, University of Technology Sydney, Australia
3 Institute for Interactive Media and Learning, University of Technology Sydney, Australia
4 School of Computing and Mathematics, Charles Sturt University, Australia
@MingLiuResearch

2. Agenda
• Session 1:Hands-on Experience of Reflective Writing Analytics: AcaWriter
• Reflective Writing (5 min)
• Reflective Writing Analytics: AcaWriter (5 min)
• Hands on Experience (30 min)
• Session 2: Linking AcaWriter to Learning Design
• Co-design Methodology (5 min)
• UTS Business School use cases (15 min)
• Group Discussion (30 min)

3. Reflective Writing
• Describe the key learning event, acknowledge thoughts and feelings, identify the
problems/challenges, link academic with experiential knowledge, and propose potential
solutions in the future (Gibson et al.,2017).
• Helps to reflect on practice-based work placements, teamwork and learning (Boud, Keogh
& Walker, 1985; Schön, 1987; Hatton & Smith, 1995, Tsingos et al.,2015).
• Promotes self-reflection and critical thinking (Mezirow, 1990).

4. Reflective writing poses some problems:
1. Challenging to write. Superficial or non-reflective description of experience
mentioned in reflective writing (Dyment & O’Connellb, 2011; Lai is, Lai, & Calandra, 2007; McNeill,
Brown, & Shaw, 2010).
2. Challenging to assess. Manually content analysis is time consuming and
subjective.

5. Reflective Writing Analytics focus on…
• Automatic detection of reflective elements
Reflective rhetorical move detection using rule-based concept matching (Buckingham Shum
et al., 2017; Gibson et al., 2017) and machine learning (Ullmann, 2019).
• Automatic detection of reflection depth
Reflective statement classification using both rule-based and machine learning approaches
(Ullmann, 2015; Ullmann, 2011; Kovanović et al., 2018).
• Generation of formative feedback
Instant 24/7 feedback to students on their drafts (Gibson et al., 2017; Lucas et al. 2018))

6. AcaWriter: A web-based reflective writing feedback
tool.
• Developed based on writing
analytics
• Provides personalized, timely
feedback.
• Provides an assignment
management for a class.
• https://acawriter.uts.edu.au

7. Group Discussion
How do you teach reflective writing at the moment?
What tool do you use?
What reflection model do you use?
What are the challenges when you teach reflective writing?
https://bit.ly/2D0Xofj

8. AcaWriter: Codified theoretical model of reflective writing
distilled from the literature
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing Analytics for Actionable Feedback. Proc. LAK17: 7th Int. Conf. on
Learning Analytics & Knowledge, March 13-17, 2017, Vancouver. (ACM Press). https://doi.org/10.1145/3027385.3027436 Best Paper Award

9. Hands on experience
• AcaWriter is a software tool that helps you develop your academic and reflective writing
by providing you with automatic feedback. For more information about AcaWriter, please
visit the AcaWriter information website. https://www.uts.edu.au/acawriter
• To give it a try and see the feedback, please visit AcaWriter demo site. http://acawriter-
demo.utscic.edu.au/demo
• Now, you can write or copy one paragraph of reflection on the demo site and try to
improve the revision by checking the instant feedback.

10. Group Discussion
How do you find the AcaWriter feedback?
Was it constructive?
Did it help you improve your writing?
Did it help you to understand RW?
Was there something you didn’t like? or had a problem with?
https://bit.ly/2D0Xofj

11. Session 2: Linking AcaWriter to Learning Design
• How to effectively use such analytics tools in pedagogic settings?
• 1. Evidence centred design (ECD): encourages the design of assessments with clear
evidence on the data to be collected (Mislevy, Almond, & Lukas, 2003). E.g. Reflective
Writing Assessment rubrics design based on the reflective elements collected from
AcaWriter.
• 2. Pedagogic approach: augmenting the learning design with AcaWriter analytics.
Learning design: designing for learning. “We can design tasks or tools or other things that
help someone learn” (Goodyear & Carvalho, 2014).
• To aid this LD-L alignment, co-design methodology is used at CIC.

12. 1. CLAD, co-design framework linking reflective writing
analytics to learning design
2. Co-design assessment and task for engineering
students
3. Co-design system intervention and feedback for
pharmacy students
Co-design Methodology for writing analytics

13. CLAD Framework @UTS for educators to co-design
Analytics/AI to augment teaching practice
Student
Task
Design
Feedback
& User
Interface
Features
in the
Data
Educators
Writing
Analytics
designers
Assessment
Shibani, A., Knight, S. and Buckingham Shum, S. (2019). Contextualizable Learning Analytics Design: A Generic Model, and Writing Analytics Evaluations. Proc. 9th International Conference on
Learning Analytics & Knowledge (LAK19). ACM Press, NY, pp. 210-219. DOI: https://doi.org/10.1145/3303772.3303785.
Co-design writing
prompts and rubrics
Co-design feedback
templates

14. An example of co-design rubrics and prompts for business
students
infers
Reflection Model
Reflective Writing Rubric

15. An example of co-design feedback for pharmacy
students

16. Co-design feedback
• The feedback template consists of a set of rules, which will
trigger feedback device to students given the presence or
absence of particular writing analytics features.
• An example of co-designed feedback rules in pharmacy. For a
given feature (col.1), we considered what should be said to
the student if it appeared (col.2) or was missing (col.3). More
complex patterns can be defined.

17. Augmenting reflective writing practices with Acawriter
Footer content here
UTS Discipline Course Level Reflective writing task
Pharmacy postgraduate
Reflective Statements about experiential placements in
clinical environment
Business undergraduate Critical Reflection Essay about team work experiences
Engineering undergraduate
Reflective Learning Journal about engagement with industry
style project.
Data Science postgraduate Reflective Project Review about team work experiences
International Studies undergraduate
Intercultural Reflections Report about critical analysis of
cultural aspects of the host culture and its relevance in the
home cultures

18. • Presenting how AcaWriter was used in business
learning context by Dr. Sumati Ahuja

19. Using AcaWriter for business students

20. Engage students with exemplars and their assessment
LD for Civil Law context
•Antonette Shibani, Simon Knight, Simon
Buckingham Shum and Philippa Ryan (2017). Design
and Implementation of a Pedagogic Intervention
Using Writing Analytics. In Proceedings of the 25th
International Conference on Computers in Education.
New Zealand: Asia-Pacific Society for Computers in
Education

21. Engage students with peer discussion
LD for Civil Law context
Antonette Shibani (2017). Combining automated and peer feedback for effective
learning design in writing practices. In DSC Proceedings of the 25th International
Conference on Computers in Education, New Zealand

22. Engage students with feedback annotation
LD for
Accounting
context
Shibani, Antonette . " Augmenting Pedagogic Writing Practice with Contextualizable Learning Analytics ," PhD dissertation.

23. Learning design for your writing tasks
• How would you redesign the reflective writing task augmented with writing analytics?
• Would you develop new tools with writing analytics?
• Please add your comments in the following shared Gdoc.
https://bit.ly/2D0Xofj

24. Writing analytics is a transdisciplinary undertaking…
• Special thanks to our AcaWriter development team,
including Agnes Sandor, Andrey Inkin, Mahesh Jeshani,
and UTS lecturers including Cherie Lucas, Sumati Ahuja,
Cat Kutay, Eva Cheng, Rosalie Goldsmith, Beate Mueller,
Neil England for providing useful feedback.