Imagine an intelligent entity that helps YOU to reflect - on yourself and your goals, on your experiences, on how to connect newly received knowledge to your existing ways of thinking. This is of course what good human teachers, coaches, and mentors can do. Can computers do that, too? Research is ongoing on various issues of how to enable conversational agents - computer software endowed with some semblance of intelligence - to facilitate just such open-ended, reflective dialogues. This capability of conversational agents to guide through reflection by leading a reflective conversation is a different capability than seeming human, being able to induce human users to feel affect, or teaching factual knowledge (which is important, too!). In this talk, a blueprint dialogue structure for reflecting on a single experience with the goal to learn for the future, will be described. This dialogue structure has been developed based on reflection theory and two use cases in work-related learning. Further, the concept of reflection scripts will be introduced: This is a framework for designers of reflective technology to think in a structured manner about what type of reflection is the goal of design and how to structure the connection between reflection and other human activities (macro scripts), and how to structure guidance through such reflection (micro scripts).

1. 1
www.tugraz.at ◼
WISSEN
TECHNIK
LEIDENSCHAFT
Theory-driven dialogue
design for reflective
conversational agents
Feb 4, 2021 – Freie Universität Berlin
Assoc.-Prof. Dr. Viktoria Pammer-Schindler

2. 2
www.tugraz.at ◼
2
About me
Associate professor at Graz University of Technology
Research area head at the Know-Center
Working, learning and technology
Viktoria.pammer@tugraz.at
Twitter: @ViktoriaPammer
Web: http://staff.tugraz.at/viktoria.pammer-schindler/

3. 3
www.tugraz.at ◼
3
Working as focal context of investigation
Learning as a focal perspective in analysis + design
What is known, learned, how, role of technology, social (organizational) context?
Lifelong learning
▪ Strategy for knowledge workers, teams, organisations, society to be
successful
▪ Foci: Reflective, situated, workplace learning; knowledge construction;
organizational learning, innovation
▪ Computing technology as a tool (interactive, data-driven, artificially intelligent)
Learning w.r.t. ongoing digital transformation
▪ New technologies (+ other global changes) require and enable new practice
▪ Foci: Data analytics, AI
▪ Technology and its integration w. practice as a reflection object
Background – Learning | Research | Conclusion and Outlook

4. 4
www.tugraz.at ◼
4
Team
Senior
Researchers
PhD
Students
Junior Researchers +
Research Developers
+ Co-
Area
Manager
External
Mentor @
NTNU w
Monica
Divitini
External
PhD student
Co-
Supervision
w Eduardo
Veas

5. 5
www.tugraz.at ◼
5
Advance organizer
Introduction
Theory-inspired dialogue design: Two cases
▪ Reflective agent for apprentices
▪ Reflection prompts in a MOOC
Vision: Towards reflection scripts
▪ Background - CSCL scripts
▪ Reflection script: Reflect on single experience

6. 6
www.tugraz.at ◼
6
Advance organizer
Introduction
Theory-inspired dialogue design: Two cases
▪ Reflective agent for apprentices
▪ Reflection prompts in a MOOC
Vision: Towards reflection scripts
▪ Background - CSCL scripts
▪ Reflection script: Reflect on single experience

7. 7
www.tugraz.at ◼
7
Reflection
▪ Critical re-examination of important aspects of own
activities (life, work, …)
▪ With the goal to change – knowledge, behaviour,
values; activity – self, tools, community, object
Ongoing experience (life)
Reflection
Trigger

8. 8
www.tugraz.at ◼
8
Computer support for reflection
▪ Data collection and visualisation (guidance
through creating awareness); facetted case
descriptions; communication and documentation
support; artistic intervention
▪ Prompts: questions or sentence starters, explicit
pointers that aim to trigger reflection
▪ Trigger: something went well, something went
bad, new knowledge, someone/thing asked you –
internal vs external
▪ Mostly manually curated, single prompts, in
educational contexts (cp. Fessl et al., 2017 – a survey)

9. 9
www.tugraz.at ◼
9
Data-driven learning from experience via
reflection + adaptive reflection prompts
Method
▪ User studies; user-centered iterative design
▪ Field studies + hierarchical evaluation: Usage, learning, behaviour,
impact
Conceptual
▪ How can we talk about
reflection across disciplines?
(Pammer et al., 2017)
▪ Reflection in the workplace is
iterative (Krogstie et al., 2013;
Pammer et al., 2017)
▪ … and can be triggered by
tools (Krogstie et al., 2012)
Empirical studies
▪ Adaptive reflection prompts (Fessl et al., 2017)
▪ Automatic activity log data – time
management (Pammer et al., 2015)
▪ Manual mood tracking data – collaborative
work (Fessl et al., 2012; Rivera-Pelayo et al., 2017)
▪ Work-related reflection in a quiz (Fessl et al.,
2014; 2018)

10. 10
www.tugraz.at ◼
10
Reflection prompts in own research
▪ Automatic – automatic choice between manually
designed prompts (adaptivity); automatic formulation
(preliminary work)
▪ Informal and workplace learning contexts
▪ Sequence of prompts – conversational reflection
guidance

11. 11
www.tugraz.at ◼
11
Advance organizer
Introduction
Theory-inspired dialogue design: Two cases
▪ Reflective agent for apprentices
▪ Reflection prompts in a MOOC
Vision: Towards reflection scripts
▪ Background - CSCL scripts
▪ Reflection script: Reflect on single experience

12. 12
www.tugraz.at ◼
WISSEN
TECHNIK
LEIDENSCHAFT
Wolfbauer et al. (2020) Rebo Junior:
Analysis of Dialogue Structure Quality
for a Reflection Guidance Chatbot. EC-
TEL Impact Paper Proceedings 2020

13. 13
www.tugraz.at ◼
13
Apprentices in a training workshop
Training workshop: Train relevant tasks
Learning tasks
▪ Similar to future work tasks, in training workshop
▪ + Learning guidance (documentation + reflection with
Rebo)
Goals of learning guidance
▪ Support learning
▪ Teach reflection as key mechanism
of lifelong professional learning

14. 14
www.tugraz.at ◼
14
Judgement
Emotion
Learning
Planning

15. 15
www.tugraz.at ◼
15
Research goals
1. Do apprentices accept the intervention?
2. Are the resulting conversations coherent and
reflective?
3. Do apprentices learn to reflect?

16. 16
www.tugraz.at ◼
16
RQ1 Reaction - Do the apprentices accept
Rebo?
Interactions with Rebo Junior
Total number: 18
General disposition towards talk with Rebo Junior
neutral: 6% (1)
positive: 94% (17)
They see a benefit in reflecting with Rebo Junior
no: 30% (3)
yes: 70% (7)
Interesting additional feedback
+ It feels like a real conversation.
- Rebo Junior “types” too fast. (→ delay put in)
+ Rebo Junior was compared to traditional forms
of reflection (“a teacher gives you a sheet of
paper”, “an empty textbox”) and found better.
+ Some apprentices started thinking about how
they can benefit from reflection.
Apprentices
accept Rebo
initially.

17. 17
www.tugraz.at ◼
17
Concept Description 153 interactions
Coherence* 0: incoherent, sequence makes no sense 4 (3%)
1: Coherent, given answers and following
questions are fitting
149 (97%)
Stage of
Reflection*
1: Provision and description of experience 153 (100%)
2: Reflection on experiences, including
analysis and potential solutions
114 (75%)
3: Learning or change 133 (87%)
RQ2 Learning - Do the apprentices successfully
reflect with Rebo (process and insights)?
*Inter-coder agreement: 97%
Yes! Almost all dialogues are coherent and reflective.
-> Dialogue structure!

18. 18
www.tugraz.at ◼
18
Advance organizer
Introduction
Theory-inspired dialogue design: Two cases
▪ Reflective agent for apprentices
▪ Reflection prompts in a MOOC
Vision: Towards reflection scripts
▪ Background - CSCL scripts
▪ Reflection script: Reflect on single experience

19. 19
www.tugraz.at ◼
SCIENCE
PASSION
TECHNOLOGY
Cicchinelli & Pammer-Schindler (in
submission). What makes volunteer
mentors tick? A case study in a
preparatory online training course.

20. 20
www.tugraz.at ◼
20
Prompting professionals preparing for
volunteer mentoring
Setting: Training for professionals who have volunteered to
mentor families on AI and entrepreneurship.
Problem: Large drop-out between training and mentoring
Intervention: Prompts that connect between online course,
future practice as mentors and future practice as
professionals, thereby enhancing motivation
▪ Motivation is both intrinsice (what is interesting?) and extrinsic (what is
useful)
▪ Transferring knowledge across contexts is challenging
▪ Reflection guidance through all reflection guidance is maybe too long
in an online course setting

21. 21
www.tugraz.at ◼
21
Prompting professionals preparing for
volunteer mentoring
1) What did you find interesting about the things
that you did in this period (either in this training
module or in volunteering with families)? Why?
2) Having identified interesting things in this
period, can you set goals to apply a particular
knowledge skill or attitude in your life, work, or
volunteering work with families? For example:
− To practice brainstorming techniques with a
problem in my area.
− To test two AI modules.
Revisit +
judge/select
Planning

22. 22
www.tugraz.at ◼
22
Research goals
1. What are different motivations and learning goals of
volunteer mentors to participate in mentoring?
2. Does increased (perceived and actual) relevance of
volunteering increase percentage of volunteers who
actually follow-up training to actual volunteering?

23. 23
www.tugraz.at ◼
23
Different mentor interests
Data: N= 51/90
Bottom-up coding led to developing categories of motivation:
▪ Mentoring
▪ Working with families
▪ Developing communication
skills
▪ Developing problem
solving skills
▪ Learning about AI
▪ Prototyping AI
Answers to reflection
prompts are meaningful
Distinct mentor interest
profiles can be identified

24. 24
www.tugraz.at ◼
24
Advance organizer
Introduction
Theory-inspired dialogue design: Two cases
▪ Reflective agent for apprentices
▪ Reflection prompts in a MOOC
Vision: Towards reflection scripts
▪ Background - CSCL scripts
▪ Reflection script: Reflect on single experience

25. 25
www.tugraz.at ◼
25
CSCL scripts
▪ Formalisable structures for collaborative learning
activities
▪ At different levels of granularity: Integrative, macro,
micro scripts
▪ Used on computational environments for collaborative
learning
▪ Used as a basis for conversational agents
Inspiring to think about „reflection scripts“!

26. 26
www.tugraz.at ◼
26
Example reflection script: Reflect on a single
experience
Integrative level: Reflect on specific type of single (learning) experiences
Macro-Level:
▪ Trigger: Selected type of single
experience (learning task;
learning module, particular type of event)
occurs
▪ Dialogue
▪ Revisiting + Judgement
▪ Emotion / Selection
▪ DONE: Showing empathy (Patrick Steyer)
▪ Learning
▪ In progress: Identify existence of
reasonable answer (Amel Hamidovic)
▪ TODO Micro-level – content-wise
feedback + Reminder to similar past insights
▪ Planning
▪ In progress: Identify existence of reasonable answer (Amel Hamidovic)
▪ TODO Micro-level – content-wise feedback + trigger follow-up session
Developed based on theory and
in parallel to two cases:
apprentices (Irmtraud
Wolfbauer); volunteer mentors
(Analia Cicchinelli)
TODO: Use as a starting point
for design of third dialogue
structure + evaluate

27. 27
www.tugraz.at ◼
27
References
Fessl et al. (2014) Mood Tracking in Virtual Meetings. Proceedings of the 7th European Conference on Technology-Enhanced
Learning (ECTEL 2012), 2012.
Fessl et al. (2017) In-app Reflection Guidance: Lessons Learned across Four Field Trials at the Workplace. IEEE Transactions on
Learning Technologies, Vol 10/4, pp 488-501, 2017. DOI: https://doi.org/10.1109/TLT.2017.2708097
Fessl et al. (2017) The Known Universe of Reflection Guidance: a Literature Review. International Journal of Technology Enhanced
Learning, Vol 9, No 2/3, 2017.
https://graz.pure.elsevier.com/files/30136567/Fessl_et_al._2017_The_Known_Universe_of_Reflection_Guidance_A_Literature_R
eview_AUTHOR_FINAL.pdf
Fessl et al. (2018) Transfer of Theoretical Knowledge into Work Practice: A Reflective Quiz for Stroke Nurses. In: Knowledge
Management in Digital Change, Springer, 2018, p. 291-308. DOI: https://doi.org/10.1007/978-3-319-73546-7_18
Krogstie et al. (2012) Computer Support for Reflective Learning in the Workplace: A Model. 12th IEEE International Conference on
Advanced Learning Technologies, ICALT 2012.
Krogstie et al., (2013) Understanding and Supporting Reflective Learning Processes in the Workplace: The CSRL Model. Scaling up
Learning for Sustained Impact - 8th European Conference, on Technology Enhanced Learning, EC-TEL 2013, Paphos, Cyprus,
September 17-21, 2013.
Pammer et al. (2015) The Value of Self-Tracking and the Added Value of Coaching in the Case of Improving Time Management. In:
Design for Teaching and Learning in a Networked World, Proceedings of the10th European Conference on Technology Enhanced
Learning (ECTEL 2015), pp.467-472, 2015. DOI: https://doi.org/10.1007/978-3-319-24258-3_41
Pammer et al. (2017) Let's Talk About Reflection at Work. International Journal of Technology Enhanced Learning, Vol 9, No 2/3,
2017. https://graz.pure.elsevier.com/en/publications/lets-talk-about-reflection-at-work
Pammer-Schindler (2019) Designing Data-Driven and Adaptive Technologies for Reflective Learning in the Workplace. Habilitation
thesis, Graz University of Technology, 2019
Renner et al. (2019) Computer-supported reflective learning: How apps can foster reflection at work. Behaviour & Information
Technology, Taylor & Francis, 2019. DOI: https://doi.org/10.1080/0144929X.2019.1595726
Rivera-Pelayo et al. (2017) Introducing Mood Self-Tracking at Work: Empirical Insights from Call Centers. ACM Trans. Comput.-
Hum. Interact., ACM, 2017, 24, 3:1-3:28. DOI: https://doi.org/10.1145/3014058
Wolfbauer et al. (2020) Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot. EC-TEL Impact
Paper Proceedings 2020: 15th European Conference on Technology Enhanced Learning, CEUR Workshop Proceedings, 2020.