2021 01-20 - theory-driven dialogue design for conversational agents - public

1
www.tugraz.at ◼
WISSEN
TECHNIK
LEIDENSCHAFT
Working, learning and
technology
Focus topic: Theory-inspired dialogue
design for conversational agents
Jan 18, 2021 – Dialogue design for conversational agents,
online research meet-up between KAIST and TUGraz
Assoc.-Prof. Dr. Viktoria Pammer-Schindler

2
www.tugraz.at ◼
2
About me
Associate professor at Graz University of Technology
Research area head at the Know-Center
Working, learning and technology
Viktoria.pammer@tugraz.at
Twitter: @ViktoriaPammer
Web: http://staff.tugraz.at/viktoria.pammer-schindler/

3
www.tugraz.at ◼
3
Research agenda
Understand working and learning in the ongoing digital
transformation
Design and evaluate socio-technical interventions for it
▪ Reflective, situated, workplace learning; knowledge
construction
▪ Organisational learning and innovation; strategic
decision-making
Research at the intersection of TEL, HCI, IS
Datum wie Fußzeilentext zentral eingeben
Fußzeilentext im Menüpunkt „Kopf- und Fußzeile“ eingeben und für alle Folien übernehmen

4
www.tugraz.at ◼
4
(Informal and professional) Learning as a
focal perspective in analysis and design
Lense of analysis: What is known, learned, how, role of technology, social
(organizational) context?
Lifelong learning
▪ Strategy for knowledge workers, teams, organisations, society to be successful
▪ Computing technology as a tool (interactive, data-driven, artificially intelligent)
Learning w.r.t. ongoing digital transformation
▪ New technologies (+ other global changes) require and enable new practice – technology
and its integration w. practice as a reflection object
▪ Knowledge construction, informal learning
Background – Learning | Research | Conclusion and Outlook

5
www.tugraz.at ◼
5
Socio-technical and design-oriented research
▪ Design-oriented – address relevant challenges using existing scientific knowledge +
contribute to both practice and research
▪ Design: User-centered iterative design, contextual design, experiments, pilot studies
▪ Socio-technical: Understand human activities, technology as tools, and the social
embedding of activity (activity theory)
▪ Understand and evaluate: Field studies – including observations, interviews, contextual inquiry, content
analysis, data analytics
▪ Outcomes: Artefacts, design-oriented knowledge
Background – Research approach | Research | Conclusion and Outlook
Understand Design Evaluate

6
www.tugraz.at ◼
6
Recent publications
Technology-
Enhanced
Learning Socio-
technical
design
perspective
and methods
Learning in
relationship
to ongoing
digitalisation
Wolfbauer, Pammer-Schindler
& Rosé (2020) - Rebo Junior:
Analysis of Dialogue
Structure Quality for a
Reflection Guidance Chatbot.
Dennerlein, Pammer-Schindler et
al. (2020) - Designing a Sandpit-
and Co-Design-informed
Innovation Process for Scaling TEL
Research in Higher Education.
Fruhwirth, Pammer-Schindler
& Thalmann (2021) - A
Network-based Tool for
Identifying Knowledge Risks
in Data-Driven Business
Models. Latinovic & Pammer-Schindler
(2021) - Automation and Artificial
Intelligence in Software
Engineering: Experiences,
Challenges, and Opportunities.

7
www.tugraz.at ◼
7
Ongoing research streams
Technology-
Enhanced
Learning Socio-
technical
design
perspective
and methods
Learning in
relationship
to ongoing
digitalisation
Understanding and
designing for
contextualized learning
w.r.t. AI as a foundation
for informed action.
Conversational
reflection guidance –
intelligent technology
to support lifelong
human learning
Socio-technical design
approaches as a
means to support
innovation and ethical
technology design.
Designing for learning
and knowledge
construction to enable
take-up and innovation
based on Open
Science.

8
www.tugraz.at ◼
8
Team
Senior
Researchers
PhD
Students
Junior Researchers +
Research Developers
+ Co-
Area
Manager
External
Mentor @
NTNU w
Monica
Divitini
External
PhD student
Co-
Supervision
w Eduardo
Veas

9
www.tugraz.at ◼
9
Advance organizer
Reflection in HCI – past and ongoing work
▪ Data-driven and adaptive technologies for reflective learning in the
workplace
▪ Conversational reflection and learning guidance, with theory-inspired
dialogue design
Towards reflection scripts
▪ Background - CSCL scripts, cognitive tutors
▪ Reflection script: Reflect on single experience

10
www.tugraz.at ◼
10
Advance organizer
▪ Data-driven and adaptive technologies for reflective learning in
the workplace
dialogue design
▪ Vision – Intelligent mentoring
▪ Background - CSCL scripts

11
www.tugraz.at ◼
11
Data-driven learning from experience via
reflection, and adaptive reflection prompts
For knowledge workers

12
www.tugraz.at ◼
12
Data-driven learning from experience via
reflection + adaptive reflection prompts
Method
▪ User studies; user-centered iterative design
▪ Field studies + hierarchical evaluation: Usage, learning, behaviour,
impact
Conceptual
▪ Interdisciplinary terminology
(Pammer et al., 2017)
▪ Cyclic model of reflection in
relation to work (Krogstie et al.,
2013; Pammer et al., 2017)
▪ Role of tools (Krogstie et al., 2012)
Empirical studies
▪ Adaptive reflection prompts (Fessl et al.,
2017)
▪ Automatic activity log data – time
management (Pammer et al., 2015)
▪ Manual mood tracking data –
collaborative work (Fessl et al., 2012; Rivera-
Pelayo et al., 2017)
▪ Work-related reflection in a quiz (Fessl et
al., 2014; 2018)

13
www.tugraz.at ◼
13
Meta-reflection (cp. Pammer-Schindler, 2019)
+ move towards conversational reflection guidance
Reflection prompts + adaptive guidance in workplaces:
Interventions can work, BUT much more issues experienced than
expected – cp reporting in learning analytics or quantified self literature
▪ Setting aside time and space for learning often challenging
▪ Responses to reflection prompts are often very short
▪ Conversational reflection guidance can prompt for elaboration
▪ Issues w. data interpretation and identification of strategy
▪ Conversational reflection guidance can lead through different
steps in reflection
▪ Privacy issues in the case of fine-grained activity tracking (esp. outside
learning environment – context switch)
▪ Natural language is easy to understand and review w.r.t. privacy
by learner

14
www.tugraz.at ◼
14
Advance organizer
workplace
▪ Conversational reflection and learning guidance, with theory-
inspired dialogue design
▪ Preliminary reflection script „theory“

15
www.tugraz.at ◼
15
Ongoing research on
conversational reflection guidance
Apprentices in the context of a training workshop
▪ Pilot study 1 – acceptance (“cool”, “like talking to a real
person”), usage via logging, learning via content analysis
(Wolfbauer et la., 2020).
▪ Ongoing: Study 2 w. reflective essay as pre/post test; identifying
fake reflexivity
Prompting online learners to connect learning to their work
▪ Field study – volunteer mentors for AI and entrepreneurship
education (Cicchinelli & Pammer-Schindler, in submission)
Talking about (Artificial) Intelligence
▪ Experimental study 1 – Identifying structure quality in
arguments based on Toulmin’s model of argument (Mirzababaei
& Pammer-Schindler, in submission)

16
www.tugraz.at ◼
16
Conversational agents for apprentices
reflecting on a learning task
Dialogue structure based on conceptualisations of
reflection levels
Two ideas:
▪ Good reflection combines different cognitive and
communicative activities
▪ Conversational reflection guidance can lead through different
levels
▪ Open: Levels vs. different types of thinking
▪ Open: Are levels consecutive?

17
www.tugraz.at ◼
WISSEN
TECHNIK
LEIDENSCHAFT
Wolfbauer et al. (2020) Rebo Junior:
Analysis of Dialogue Structure Quality
for a Reflection Guidance Chatbot. EC-
TEL Impact Paper Proceedings 2020

18
www.tugraz.at ◼
18
Apprentices in a training workshop
Training workshop: Train relevant tasks
Learning tasks
▪ Similar to future work tasks, in training workshop
▪ + Learning guidance (documentation + reflection with
Rebo)
Goals of learning guidance
▪ Support learning
▪ Teach reflection as key mechanism
of lifelong professional learning

19
www.tugraz.at ◼
19
• Background | Research – Conversational Refl. Guidance | Conclusion and Outlook
Judgement
Emotion
Learning
Planning

20
www.tugraz.at ◼
20
RQ1 Reaction - Do the apprentices accept
Rebo?
Interactions with Rebo Junior
Total number: 18
General disposition towards talk with Rebo Junior
neutral: 6% (1)
positive: 94% (17)
They see a benefit in reflecting with Rebo Junior
no: 30% (3)
yes: 70% (7)
Interesting additional feedback
+ It feels like a real conversation.
- Rebo Junior “types” too fast. (→ delay put in)
+ Rebo Junior was compared to traditional forms
of reflection (“a teacher gives you a sheet of
paper”, “an empty textbox”) and found better.
+ Some apprentices started thinking about how
they can benefit from reflection.
Apprentices
accept Rebo
initially.

21
www.tugraz.at ◼
21
Concept Description 153 interactions
Coherence* 0: incoherent, sequence makes no sense 4 (3%)
1: Coherent, given answers and following
questions are fitting
149 (97%)
Stage of
Reflection*
1: Provision and description of experience 153 (100%)
2: Reflection on experiences, including
analysis and potential solutions
114 (75%)
3: Learning or change 133 (87%)
RQ2 Learning - Do the apprentices successfully
reflect with Rebo (process and insights)?
*Inter-coder agreement: 97%
Yes! Almost all dialogues are coherent and reflective.
-> Dialogue structure!

22
www.tugraz.at ◼
Challenges, gaps
Do apprentices learn to reflect? -> Ongoing study w pre/post test
(reflective essay)
Improve reflection quality
▪ Assess reflection quality – reflection analytics, structural vs.
domain-oriented assessment, fake reflexivity
▪ Design implication: Adaptation mechanisms to improve
reflection
Assumption: Engagement decreases over time. Indications there,
but no significant results.
▪ Gap in literature on long-term engagement w. learning
guidance.
▪ Design implication: Adaption mechanisms to combat
disengagement.

23
www.tugraz.at ◼
23
Ongoing research on
fake reflexivity

24
www.tugraz.at ◼
SCIENCE
PASSION
TECHNOLOGY
Cicchinelli & Pammer-Schindler (in
submission). What makes volunteer
mentors tick? A case study in a
preparatory online training course.

25
www.tugraz.at ◼
25
Prompting professionals preparing for
volunteer mentoring
Prompts aim to connect between online course, future
practice as mentors and future practice as
professionals, thereby enhancing motivation
▪ Motivation is both intrinsice (what is interesting?) and extrinsic
(what is useful)
▪ Transferring knowledge across contexts is challenging
▪ Reflection guidance through all reflection guidance is maybe
too long in an online course setting

26
www.tugraz.at ◼
26
Prompting professionals preparing for
volunteer mentoring
1) What did you find interesting about the things
that you did in this period (either in this training
module or in volunteering with families)? Why?
2) Having identified interesting things in this
period, can you set goals to apply a particular
knowledge skill or attitude in your life, work, or
volunteering work with families? For example:
− To practice brainstorming techniques with a
problem in my area.
− To test two AI modules.
Revisit +
judge/select
Planning

27
www.tugraz.at ◼
27
RQ: Based on answers to reflection prompts, which different
motivations and interests do volunteer mentors have?
… as a precursor for identifying different needs w.r.t.
motivation and applicability of what is learned through
mentoring.
Data: N= 51/90
Bottom-up coding led to developing categories
▪ Mentoring
▪ Working with families
▪ Developing communication skills
▪ Developing problem solving skilss
▪ Learning about AI
▪ Prototyping AI

28
www.tugraz.at ◼
28
Ongoing research on
fake reflexivity

29
www.tugraz.at ◼
SCIENCE
PASSION
TECHNOLOGY
Mirzababaei & Pammer-Schindler (in
submission). Developing a conversational
agent’s capability to differentiate
between different types of wrongness in
arguments based on Toulmin’s model of
arguments

30
www.tugraz.at ◼
30
Conversational agent to discuss in what
sense a concrete entity is intelligent
Dialogue structure based on Bloom‘s taxonomy:
Two ideas:
▪ Different types of understanding a concept, with different types
of argumentation expressing them, and different types of
exercises supporting them.
▪ Conversational learning guidance can lead through different
levels
▪ Open: Levels vs. different types of thinking
▪ Open: Are levels consecutive?

31
www.tugraz.at ◼
31
CA: Do you remember the five different
definitions of intelligence from the lecture?
…
CA: Can you explain the definition „thinking
rationally“ in your own words?
…
CA: So, using any of the above definitions, in
what sense is a search engine intelligent?
Remember
Understand
Apply

32
www.tugraz.at ◼
32
Feedback on „apply“ level based on Toulmin‘s model of
arguments
▪ A reasonable argument contains multiple components
(core components: claim, warrant, evidence)
▪ A conversational agent can support constructing a
reasonable argument by identifying the existence of
such components (structural quality)
▪ … and a contentwise correct relationship between
component

33
www.tugraz.at ◼
33
Apply
Warrant is
missing
Evidence is
missing

34
www.tugraz.at ◼
34
RQ: How well can core components of
Toulmin‘s model of arguments be identified?
Data collection
▪ Three datasets (N=100, 1026, 211)
▪ collected via Amazon Mturk
▪ Entities: table, office chair, Statue of Liberty (inanimate objects),
tree, sunflower, Venus flytrap (plants) cat, fish, snake, monkey
(animals) Google search engine, self-driving car (AI-enabled
technologies)
Component Claim Warrant Evidence
Annotation Positive
Negati
ve
Unknow
n
With
warrant
Without
warrant
With
evidence
Without
evidenc
e
Training data
(the dataset 1
and 2)
477 594 55 691 435 835 291
Test data
(the dataset 3)
102 99 10 111 100 159 52

35
www.tugraz.at ◼
35
RQ: How well can core components of
Toulmin‘s model of arguments be identified?
Features
▪ TFIDF with different cut-off lengths; component specific: claim
and warrant – regular expressions on yes/no and definitions;
evidence – specific words/phrases; response length
Model selection based on training datasets 1+2
Accuracy on dataset 3 (accuracy over all classes)
▪ Existence of claim: 0.91
▪ Existence of warrant: 0.89
▪ Existence of evidence: 0.83
▪ Attention: imbalance between classes -> dialogue structure
needs to express this uncertainty!

36
www.tugraz.at ◼
36
Extend from Bloom‘s taxonomy towards reflection
in the sense of planning future action
CA: Do you remember the five different definitions
of intelligence from the lecture?
…
CA: Can you explain the definition „thinking
rationally“ in your own words?
…
CA: So, using any of the above definitions, in what
sense is a search engine intelligent?
…
Application towards case/example of relevance for
learner + reflection on future action
Remember
Understand
Apply

37
www.tugraz.at ◼
37
Advance organizer
workplace
dialogue design

38
www.tugraz.at ◼
Vision
▪ Meta-model of lifelong learning that connect ongoing experience, new
knowledge, and reflection.
▪ Examples: reflection in regular intervals about ongoing progress
towards goals; reflection about relevance of newly received
informatino/knowledge and possible change to future practice
▪ Process models that connect between reflection sessions and
conceptually structure reflection sessions
▪ Example: Reflect on past time interval (revisit + describe,
judge/select, learn) + go back to plan (revisit + judge outcome) from
last reflection session; + formulate new plan
▪ Operational conditional dialogue structures that instantiate such
process models
▪ By providing concrete verbalisations for steps in a reflection session.
▪ By connecting to machine learning (natural language processing
based) functionality required to operationalise conditional dialogue
structures.
How far can technology go towards serving a mentoring function
for adults by facilitating long-term and iterative reflection?

39
www.tugraz.at ◼
39
Advance organizer
workplace
dialogue design

40
www.tugraz.at ◼
40
CSCL scripts
▪ Formalisable structures for collaborative learning
activities
▪ At different levels of granularity: Integrative, macro,
micro scripts
▪ Used on computational environments for collaborative
learning
▪ Used as a basis for conversational agents
Inspiring to think about „reflection scripts“!

41
www.tugraz.at ◼
41
Advance organizer
workplace
dialogue design

42
www.tugraz.at ◼
42
Example reflection script: Reflect on a single
experience
Integrative level: Reflect on specific type of single (learning) experiences
Macro-Level:
▪ Trigger: Selected type of single
experience (learning task;
learning module, particular type of event)
▪ Dialogue
▪ Revisiting + Judgement
▪ Emotion / Selection
▪ DONE: Showing empathy (Patrick Steyer)
▪ Learning
▪ In progress: Identify existence of
reasonable answer (Amel Hamidovic)
▪ TODO Micro-level – content-wise
feedback + Reminder to similar past insights
▪ Planning
▪ In progress: Identify existence of reasonable answer (Amel Hamidovic)
▪ TODO Micro-level – content-wise feedback + trigger follow-up session
Developed based on theory and
in parallel to two cases:
apprentices (Irmtraud
Wolfbauer); volunteer mentors
(Analia Cicchinelli)
TODO: Use as a starting point
for design of third dialogue
structure + evaluate

43
www.tugraz.at ◼
43
References
Fessl et al. (2014) Mood Tracking in Virtual Meetings. Proceedings of the 7th European Conference on Technology-Enhanced
Learning (ECTEL 2012), 2012.
Fessl et al. (2017) In-app Reflection Guidance: Lessons Learned across Four Field Trials at the Workplace. IEEE Transactions on
Learning Technologies, Vol 10/4, pp 488-501, 2017. DOI: https://doi.org/10.1109/TLT.2017.2708097
Fessl et al. (2018) Transfer of Theoretical Knowledge into Work Practice: A Reflective Quiz for Stroke Nurses. In: Knowledge
Management in Digital Change, Springer, 2018, p. 291-308. DOI: https://doi.org/10.1007/978-3-319-73546-7_18
Krogstie et al. (2012) Computer Support for Reflective Learning in the Workplace: A Model. 12th IEEE International Conference on
Advanced Learning Technologies, ICALT 2012.
Krogstie et al., (2013) Understanding and Supporting Reflective Learning Processes in the Workplace: The CSRL Model. Scaling up
Learning for Sustained Impact - 8th European Conference, on Technology Enhanced Learning, EC-TEL 2013, Paphos, Cyprus,
September 17-21, 2013.
Pammer et al. (2015) The Value of Self-Tracking and the Added Value of Coaching in the Case of Improving Time Management. In:
Design for Teaching and Learning in a Networked World, Proceedings of the10th European Conference on Technology Enhanced
Learning (ECTEL 2015), pp.467-472, 2015. DOI: https://doi.org/10.1007/978-3-319-24258-3_41
Pammer et al. (2017) Let's Talk About Reflection at Work. International Journal of Technology Enhanced Learning, Vol 9, No 2/3,
2017. https://graz.pure.elsevier.com/en/publications/lets-talk-about-reflection-at-work
Pammer-Schindler (2019) Designing Data-Driven and Adaptive Technologies for Reflective Learning in the Workplace. Habilitation
thesis, Graz University of Technology, 2019
Renner et al. (2019) Computer-supported reflective learning: How apps can foster reflection at work. Behaviour & Information
Technology, Taylor & Francis, 2019. DOI: https://doi.org/10.1080/0144929X.2019.1595726
Rivera-Pelayo et al. (2017) Introducing Mood Self-Tracking at Work: Empirical Insights from Call Centers. ACM Trans. Comput.-
Hum. Interact., ACM, 2017, 24, 3:1-3:28. DOI: https://doi.org/10.1145/3014058
Wolfbauer et al. (2020) Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot. EC-TEL Impact
Paper Proceedings 2020: 15th European Conference on Technology Enhanced Learning, CEUR Workshop Proceedings, 2020.

2021 01-20 - theory-driven dialogue design for conversational agents - public

Recommended

Recommended

More Related Content

Similar to 2021 01-20 - theory-driven dialogue design for conversational agents - public

Similar to 2021 01-20 - theory-driven dialogue design for conversational agents - public (20)

More from Viktoria Pammer-Schindler

More from Viktoria Pammer-Schindler (6)

Recently uploaded

Recently uploaded (20)

2021 01-20 - theory-driven dialogue design for conversational agents - public