Designing Social Interactions in a Teachable Agent

69 views

Published on

Erin Walker, Arizona State University presents "Designing Social Interactions in a Teachable Agent" as part of the Cognitive Systems Institute Speaker Series on 9/22/16.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
69
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Designing Social Interactions in a Teachable Agent

  1. 1. Erin Walker School of Computing, Informatics, and Decision Systems Engineering Arizona State University Designing Social Interactions in Teachable Agents
  2. 2. Spoken Dialogue Systems Spoken dialogue systems are becoming increasingly prevalent (Siri, Alexa, Google Now). These systems have to sustain interactions with people over longer periods of time. It matters what these systems say, but also how they say it.
  3. 3. Teachable agents Betty’s Brain -- Leelawong & Biswas, 2008
  4. 4. Cognitive Factors Novices benefit from helping others  Increased attention  Increased reflection  Elaboration of knowledge  Co-construction of knowledge Roscoe & Chi, 2007; Duran & Monero, 2005; Webb, 2013 These benefits transfer to teachable agents.
  5. 5. Social Factors Less well understood in learning by teaching  Students experience a protégé effect Chase et al., 2009  Students build self-efficacy through learning by teaching Medway & Barron, 1977  Feelings of rapport for one’s collaborating partner may enhance the effects of learning by teaching Ogan et al., 2012
  6. 6. Research Questions 1. How do social factors enhance benefits of learning by teaching? 2. How can we design a teachable agent respond socially to a student in order to enhance the benefits of learning by teaching?
  7. 7. Goal: examine social interactions in teachable agent environments. How do social factors influence learning by teaching? How can social factors be leveraged by an intelligent agent? Implications and future directions
  8. 8. Two examples Prior knowledge, social engagement, & a robotic teachable agent. Rapport & peer tutoring
  9. 9. Two examples Prior knowledge, social engagement, & a robotic teachable agent. Rapport & peer tutoring Kasia Muldner Victor Girotto Cecil Lozano Win Burleson
  10. 10. Students teach a robot about plotting a point on a graph. rTAG Project
  11. 11. 11 Problem: Plot the point (3,1) In the coordinate (3,1), the number 1 corresponds to the y coordinate, right? Move 3 Projected Coordinate System Mobile Interface
  12. 12. 12 We got that right because we tried hard! Problem: Plot the point (3,1) Projected Coordinate System Mobile Interface
  13. 13. 3 Conditions Does the use of the robotic learning environment increase social engagement over the use of a virtual agent? Does the increased social engagement lead to increased learning? rTAG eTAG vTAG
  14. 14. Participants and Procedure 35 participants, 4th-6th grade, 18 male, 17 female. Procedure  5 minute geometry review  15 minute pretest  20 minutes of training  45 minutes of problem-solving  15 minute posttest  Self-report questionnaire
  15. 15. Measures Learning:  Isomorphic, counterbalanced pre and posttest forms, 11 questions Social Perceptions Bartneck et al., 2009  Animacy  Likability  Intelligence  Trustworthiness
  16. 16. Results High prior knowledge students had higher social perceptions of vTAG, while low prior knowledge students had higher social perceptions of rTAG. Effects of prior knowledge on social perceptions vTAG rTAG High prior knowledge + - Low prior knowledge - +
  17. 17. Results: Deeper Dive While not significant, in rTAG and eTAG, better social perceptions were negatively correlated with learning, while in vTAG, social perceptions were positively correlated with learning. Effects of social perceptions on learning vTAG rTAG Better social perceptions + - Worse social perceptions - +
  18. 18. Discussion While low prior knowledge students responded more positively to rTAG, those students that responded more positively learned less.  Low prior knowledge students may have learned less in general  Low prior knowledge students may have been more easily distracted by the additional features of rTAG Limitations: small sample size, short term interaction Important to socially engage students without creating distracting elements.
  19. 19. Two examples Prior knowledge, social engagement, & a robotic teachable agent. Rapport & peer tutoring Amy Ogan Samantha Finkelstein Ryan Carlson Justine Cassell
  20. 20. Peer Tutoring Context 8th-10th grade students, literal equation solving 54 friend dyads, 6 stranger dyads Procedure  20 minute pretest  20 minute individual work  60 minute computer-mediated tutoring (students were randomly assigned to role)  20 minute posttest
  21. 21. Coding Scheme 4 conversational factors  Playfulness to lighten the mood or mitigate negativity  Face-threat remarks directed toward the partner  Attention-getting to draw the partner back on task  Emphasis to add emotive features 2 social functions  Positivity (e.g., politeness, empathy, praise, reassurance)  Impoliteness (e.g., cooperative and uncooperative rudeness)
  22. 22. Results Friends learn more than strangers. In friends, the combination of face threat and positivity improved learning.  When a tutee exhibits face threat, a tutor will focus the conversation on task-related features  Positivity enhances the learning benefits of these face- threatening acts In strangers, face threat was a negative predictor of learning gains.
  23. 23. Discussion rTAG: Students socially engaged more with the robot, but this decreased their learning. Rapport and tutoring: If collaborating partners felt rapport, they appeared to challenge each other more and reflect better on the domain. Social engagement interacts with the context to produce positive or negative effects.
  24. 24. Goal: examine social interactions in teachable agent environments. How do social factors influence learning by teaching? How can social factors be leveraged by an intelligent agent? Implications and future directions
  25. 25. Entrainment Acoustic-Prosodic Entrainment: Two speakers adapt their acoustic-prosodic features including their tone, intensity, and speaking rate to mirror to one another. Levitan et al. 2012 Correlated with  Communicative success Borrie & Liss, 2014  Conversational flow Nenkova, Gravano, & Hirschberg, 2008  Rapport Lubold & Pon-Barry, 2014 Not clear how to produce acoustic-prosodic entrainment within a dialogue system.
  26. 26. Iterative process Prototype different methods of pitch adaptation Evaluate the effects of pitch adaptation in a learning context Nichola Lubold Heather Pon-Barry
  27. 27. Dialogue System Domain: Literal equation solving Students walk the agent, Quinn, verbally through steps displayed on the screen. S: We will divide both sides by negative six Q: Can you explain why we divide? S: On the left hand side, we have negative 6y. We need to have it equal just y so we need to get rid of the negative six. The easiest way is to divide. Q: Thank you for explaining! I get it now. So we divide. Then what?
  28. 28. System architecture
  29. 29. Explore 4 methods of pitch- adaptation Entrainment: low high medium medium Naturalness: high mdium high low Rapport: low high high high
  30. 30. Data Collection Collected 32 dialogues from four college students. Gender of Quinn’s voice chosen to match the gender of the student. Each student encountered all forms of pitch adaptation (order counterbalanced). 5.4 minutes per dialogue (30 turns per dialogue).
  31. 31. AMT Evaluation Selected 10 exchanges per student in each condition (160 possible exchanges). 174 AMT workers listened to each exchange and answer questions regarding the speakers  Naturalness: Quality of the voice from very poor to completely natural (Likert scale)  Rapport: Understanding & closeness between two speakers  E.g., “Alex and Quinn understood each other.”  From Gratch et al., 2007
  32. 32. Results Shift + contour is more natural than other methods, equivalent to control (p < 0.001). Different properties of exchanges:  Quinn speaks first or second  Quinn speaks socially However, when Quinn speaks second for social exchanges, shift+contour produced more rapport than the other three methods (p < 0.001).
  33. 33. Summary Prototyped four different entrainment methods. Findings  Shift+contour appeared to be the best way of mimicking pitch-based entrainment.  A less effective method of entrainment was perceived as worse than doing nothing at all.  Perceived rapport depended on when entrainment was used.
  34. 34. Iterative process Prototype different methods of pitch adaptation Evaluate the effects of pitch adaptation in a learning context Nikki Lubold Heather Pon-Barry
  35. 35. Next step Two factors  Shift+contour pitch-based entrainment  Social turns in the dialogue system How does acoustic-prosodic entrainment and social content influence: 1. Social variables like mutual attention and rapport 2. Learning
  36. 36. Three Conditions Control Social Voice plus social Students explain problem steps to Quinn, and Quinn responds.
  37. 37. Conditions Control Social Voice plus social Quinn responds with social dialogue moves in addition to task-related dialogue moves. Social responses occur 15-20% of the time.
  38. 38. Conditions Control Social content Voice plus social Quinn responds socially to the student, and shifts pitch to match the student’s pitch.
  39. 39. Social Dialogue & Voice Adaptation Social dialogue with voice adaptation “We factor? Ok. Math must be your forte, you are so good at this” Non-social, non- adaptive “Oh, are you saying we factor out the two? Then we can use it to help simplify? That makes more sense”
  40. 40. Method 43 undergraduate students explained 6 problems to Quinn  16 voice plus social, 14 social, 13 control  24 female, 24 male Collected self-report responses relating to rapport and mutual attention. Coded dialogue for rapport-building and rapport- hindering behaviors.
  41. 41. Results: Mutual Attention Condition has a significant effect on mutual attention (p = 0.02). Social condition is significantly lower than the other two conditions (p = 0.02). Effect is driven by the male participants (p < 0.01). 5.5 4.9 5.57 Mutual Attention Control Voice+SocialSocial
  42. 42. Results: Self-Reported Rapport No significant effects of condition on rapport. But, there were gender effects: Females feel more rapport than males (p = .006). 4.7 5.6 Rapport (p < 0.01) Males Females
  43. 43. Rapport-Building Dialogue 0 20 40 60 80 100 120 140 Males Females Males responded best to the voice+social condition (as expected), while females responded best to the social condition (p < 0.05). Some evidence that differences between males and females are due to the expectations regarding entrainment.
  44. 44. Discussion Condition and gender influences social engagement, rapport-building and rapport- hindering behaviors.  Males responded well to the voice+social condition.  Females appeared to perceive it as unnatural. Next step: Entrain more dynamically, adapt based on gender.
  45. 45. Goal: examine social interactions in teachable agent environments. How do social factors influence learning by teaching? How can social factors be leveraged by an intelligent agent? Implications and future directions
  46. 46. Summary Goals: Explore the intersection between agent social responses, human social responses, and learning by teaching interactions. Three related areas of work:  Designing a teachable robotic agent  Study of human-human tutoring dialogues  Research program to build a teachable agent that produces rapport through verbal and non-verbal cues
  47. 47. Summary Social engagement is not always good.  Robotic form improves social perceptions over a virtual agent, but only for those with low prior knowledge.  Improved social perceptions of the robot lead to less learning. In human-human dialogues, higher rapport -> more learning, due to playful challenges in dialogue. Attempts to create rapport using entrainment and social dialogue yielded mixed results.  Dependent on context  Dependent on gender
  48. 48. Discussion Social factors are hard to get right, easy to get wrong.  For some students, learning from the robot may have been distracting.  In the Mechanical Turk study, the least natural form of entrainment was worse than the control.  For men, social condition without pitch adaptation appeared worse than control.  For women, social condition with pitch adaptation was worse than the control.
  49. 49. Discussion Context is important  Individual students may be more or less receptive to these kinds of interventions (gender, prior knowledge)  Social engagement can be influenced by subtle cues We need to think deeply about strategies of adaptation as they interact with context and individual differences.
  50. 50. Conclusions & Future Work Are social adaptations worth it?  We know they influence cognitive factors  May be more critical as users interact with technology across contexts and over time  May be more critical for disengaged students  Choices we make might have unanticipated social impacts Important to understand how the choices we make in technology design interact with individual differences and social perceptions to influence learning.
  51. 51. Thanks! Collaborators: Victor Girotto, Amy Ogan, Samantha Finkelstein, Ryan Carlson, Justine Cassell, Nichola Lubold, Heather Pon-Barry, Kasia Muldner, Cecil Lozano, Win Burleson, Ruth Wylie

×