Ongoing design of Rebo, the Reflection Guidance Chatbot. Evaluation in multiple field studies: results, insights, design implications

1. 1
© Know-Center GmbH
Rebo, the Reflection
Guidance Chatbot
A conversational agent for apprentices to reflect on
practical learning experiences
Irmtraud Wolfbauer
iwolfbauer@know-center,at
Research Lab Meeting with CMU- Conversational Agents for Learning and Reflection
11 June, 2021

2. 2
REBO JUNIOR – DIALOGUE STRUCTURE UNDERLYING THE AGENT
Rebo Junior guides apprentices through pre-defined levels of reflection
0. Revisiting
1. Description
 Upload of task
2. Judgement
3. Emotions
4. Learning
5. Planning
 To actively guide learners
from one level of reflection
to the next
 To improve the specific
learning experience by
reflection
 To improve reflection skills
for lifelong learning

3. 3
REBO JUNIOR – DIALOGUE STRUCTURE UNDERLYING THE AGENT
Dialogue structure proves very successful
Evaluation in a field study (18 apprentices, 153 interactions over 3 months)
1. Apprentices like texting with Rebo!
 Workshop feedback: 94% like Rebo, 70% see benefit
2. Conversations with Rebo Junior are coherent and reflective
 97% coherent interactions
 75% reflection stage* 2/3: analysis
 87% reflection stage* 3/3: learning & change
* Prilla & Renner (2014)
Irmtraud Wolfbauer, Viktoria Pammer-Schindler, and Carolyn Rose, “Rebo Junior: Analysis of Dialogue
Structure Quality for a Reflection Guidance Chatbot,” in Proceedings of the Impact Papers at EC-TEL
2020

4. 4
REBO JUNIOR – DIALOGUE STRUCTURE UNDERLYING THE AGENT
Do apprentices learn how to reflect by
repeated interaction with Rebo Junior?
• Field study: 14 apprentices, 108 interactions over 3 months
• 3 reflective essays as measurement of reflection skills
6 weeks RJ 6 weeks RJ
essay 1 essay 2 essay 3
• Coding RJ interactions and reflective essays
• 4 Reflection levels (Judgement, Emotions, Learning, Planning)
• 0 – 1 – 2 for each level (total scores: 0-8)
• RJ interactions:
• Coherence
• Reflective essays:
• Word count, order of reflection levels, textual features mirroring Rebo Junior

5. 5
5
0
1
2
3
4
5
6
7
Essay 1 Essay 2 Essay 3
Average Reflectivity of
Reflective Essays
REBO JUNIOR – DIALOGUE STRUCTURE UNDERLYING THE AGENT
Improvement of participants’ reflection skills
Successive increase of reflectivity
indicating improvement of reflection skills
Slight decrease of reflectivity but
throughout highly reflective interactions
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11
Interaction number
Average Reflectivity of
Rebo Junior Interactions
Scores 6 - 8: highly reflective

6. 6
REBO, THE REFLECTION GUIDANCE CHATBOT
Which adaptive mechanisms improve reflection with Rebo?
• When reflection on given level was not successful
Second question, same level reflection level
• When apprentices “faked” reflection
“Please tell me more. What exactly do you mean?”
• When the answer is too short
“Could you explain this to me? I’m going to need more text ;)”
• Empathic reactions to positive or negative experiences
“I’m glad you’re having fun ”
“Oh no :/”
• Summative comment on interaction
“This talk has been really cool for me…”
“… you could make a bit of an effort next time”

7. 7
REBO, THE REFLECTION GUIDANCE CHATBOT
Successful example Rebo interaction
Hello Rory!
Hello
What task are we talking about today?
My fourth cnc turned part
What did you do for this task? Tell me step by step.
I wrote a program according to the sketch
Are you happy with the result of your work?
Why/Why not?
I’m not sure, there are some problems with the
implementation because new task elements were
added
How did you feel while working on this task?
As if my head was about to explode
What was the problem?
´The problem is not in one specific tasks, it’s the
subject matter. Just when you’re done with one
task you already get the next one, and you really
need to think about every program and this is
confusing for me
Oh no! I hope you’ll have more fun next time!
What did you learn from this task?
How to write a program
Can you describe this a bit more?
I learned how to write a program and enter it into
the cnc lathe
When could you need what you learned here
again?
At cnc la
In order to understand what you mean, I’m going
to need a bit more text ;)
I mean that I will apply what I have learned when
I’m back at my company
Sounds like a plan!
No Problem
Awesome, thanks for the talk!
I think this talk was really cool, I’m glad that you do
your best :D
Take care :-)
Bye .
0. Revisiting
1. Description
2. Judgement
3. Emotions
4. Learning
5. Planning
Follow-up
Empathy
Planning ok
Elaboration
Elaboration
Assessment

8. 8
KNOW-CENTER.AT
KNOW-CENTER GMBH • Research Center for Data-Driven Business and Big Data Analytics • Inffeldgasse 13/6, 8010 Graz, Austria • Firmenbuchgericht Graz • FN 199 685 f • UID: ATU 50367703
gefördert durch das Programm COMET (Competence Centers for Excellent Technologies), wir danken unseren Fördergebern:
© Know-Center GmbH
I’m looking forward to
your comments!
Irmtraud Wolfbauer
Iwolfbauer@know-center.at